Newsemble is an API that provides easy access to the current news for programmatic analysis

Rishabh

Last update: Dec 16, 2022

Related tags

Downloader python api flask news newsapi bs4 webscraping

Overview

📰 Newsemble 📰

An API for fetching the current news.

🔖 About 🔖

Blog Post

Newsemble is an API that provides easy access to the current news for programmatic analysis. It has been built using Python, BeautifulSoup and MongoDB.
The data is scraped from these news websites every hour, stored in a database on the cloud and whenever requested, the most recent articles are promptly served.
Developers can make use of this API to fetch current data with each article having the following fields:
Headlines, Content, Source, Link and Time.

💻 Technologies

Newsemble is created with:

Python 3
Flask
PyMongo
BeautifulSoup

📂 File Structure and Description

app.py - Flask code for the API
scraper.py - Collection of scrapers for the various news sites.
db.py - Connecting and Using MongoDB
utils.py - Utility Functions
scheduler.py - Scheduler
Procfile - For Deployment
requirements.txt - Python Requirments

🛠️ Pipeline

🚀 Getting-started

This project can be accessed by using following setup

Links

Links	Description
www.newsemble.ml/news	Link to fetch all the data from all sources
www.newsemble.ml/news/toi	Link to fetch data from Times of India
www.newsemble.ml/news/th	Link to fetch data from The Hindu
www.newsemble.ml/news/tie	Link to fetch data from The Indian Express
www.newsemble.ml/news/ndtv	Link to fetch data from NDTV news
www.newsemble.ml/news/it	Link to fetch data from India Today

Request format

$ import requests
$ url = "http://www.newsemble.ml/news/"
$ requests.get(url).json()

Response format

{   
    ‘link’      :  $source_link$,
    ‘content’   :  $content_text$,    
    ‘source’    :  $news_source$,
    ‘title’     :  $headline$, 
    ‘time       :  $date_time_of_article$  
 }

Sample output

⚙️ Currently Supported Sites

🙏 Thanks!

All contributions are welcome and appreciated. 👍
If you liked this project, or found it useful in any way, please drop a 🌟 !

✍️ Authors ✍️

✒️ Rishabh Gupta
✒️ Vishal Singhania
✒️ Roshan Kumar

You might also like...

music downloader written in python. (Uses jiosaavn API)

35 Jul 20, 2022

Mobile based API for Crunchyroll BETA (and Downloader).

Mobile based API for Crunchyroll BETA (and Downloader). Not restricted on servers and NO CLOUDFLARE

27 Dec 11, 2022

Pypixiv - A fully-typed, asynchronous api wrapper for pixiv

pypixiv this library is a fully-typed, asynchronous api wrapper for pixiv. featu

2 Nov 16, 2022

This project is helps to download contents from Streamtape by utilizing the API

It scrapes Streamtape api and download contents from the site.

5 Dec 28, 2022

A python script that discovers hidden YouTube API clients. Just a research project.

YouTube-Internal-Clients A script that discovers hidden internal clients of the YouTube (Innertube) API using bruteforce methods. The script tries cli

97 Jan 2, 2023

Simple Python script to download images and videos from public subreddits without using Reddit's API 😎

Subreddit Media Downloader Download images and videos from any public subreddit without using Reddit's API Made with ❤ by Nico 💬 About: This script a

106 Jan 7, 2023

This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular intervals.It sends out the most recent news at random!

Nepali-news-notifier This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular in

1 Feb 11, 2022

News-app - This is a news web app for reading news from different sources and topics

1 Feb 2, 2022

Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and other financial information. This repository provides an SDK for developing applications to access the NCDS.

Nasdaq Cloud Data Service (NCDS) Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and ot

8 Dec 1, 2022

The windML framework provides an easy-to-use access to wind data sources within the Python world, building upon numpy, scipy, sklearn, and matplotlib. Renewable Wind Energy, Forecasting, Prediction

windml Build status : The importance of wind in smart grids with a large number of renewable energy resources is increasing. With the growing infrastr

125 Dec 24, 2022

Repo Home WPDrawBot - (Repo, Home, WP) A powerful programmatic 2D drawing application for MacOS X which generates graphics from Python scripts. (graphics, dev, mac)

DrawBot DrawBot is a powerful, free application for macOS that invites you to write Python scripts to generate two-dimensional graphics. The built-in

342 Dec 27, 2022

A curated list of programmatic weak supervision papers and resources

118 Jan 2, 2023

Learning trajectory representations using self-supervision and programmatic supervision.

Trajectory Embedding for Behavior Analysis (TREBA) Implementation from the paper: Jennifer J. Sun, Ann Kennedy, Eric Zhan, David J. Anderson, Yisong Y

58 Jan 6, 2023

Programmatic interface to Synapse services for Python

A Python client for Sage Bionetworks' Synapse, a collaborative, open-source research platform that allows teams to share data, track analyses, and collaborate

54 Dec 23, 2022

Django API that scrapes and provides the last news of the city of Carlos Casares by semantic way (RDF format).

"Casares News" API Api that scrapes and provides the last news of the city of Carlos Casares by semantic way (RDF format). Usage Consume the articles

6 May 12, 2022

This python module can analyse cryptocurrency news for any number of coins given and return a sentiment. Can be easily integrated with a Trading bot to keep an eye on the news.

Python script that analyses news headline or body sentiment and returns the overall media sentiment of any given coin. It can take multiple coins an

185 Dec 22, 2022

NLP project that works with news (NER, context generation, news trend analytics)

СоАвтор СоАвтор – платформа и открытый набор инструментов для редакций и журналистов-фрилансеров, который призван сделать процесс создания контента ма

38 Jan 4, 2023

Your own movie streaming service. Easy to install, easy to use. Download, manage and watch your favorite movies conveniently from your browser or phone. Install it on your server, access it anywhere and enjoy.

Vigilio Your own movie streaming service. Easy to install, easy to use. Download, manage and watch your favorite movies conveniently from your browser

141 Jan 6, 2023

💰 An Alfred Workflow that provides current price of cryptocurrency

Coin Ticker for Alfred Workflow An Alfred Workflow that provides current price and status about cryptocurrency from cryptocompare.com. Supports Alfred

14 Nov 17, 2022

Comments

Bump lxml from 4.5.0 to 4.6.5
Bumps lxml from 4.5.0 to 4.6.5.

Changelog

Sourced from lxml's changelog.

4.6.5 (2021-12-12)

Bugs fixed

A vulnerability (GHSL-2021-1038) in the HTML cleaner allowed sneaking script content through SVG images (CVE-2021-43818).

A vulnerability (GHSL-2021-1037) in the HTML cleaner allowed sneaking script content through CSS imports and other crafted constructs (CVE-2021-43818).

4.6.4 (2021-11-01)

Features added

GH#317: A new property system_url was added to DTD entities. Patch by Thirdegree.

GH#314: The STATIC_* variables in setup.py can now be passed via env vars. Patch by Isaac Jurado.

4.6.3 (2021-03-21)

Bugs fixed

A vulnerability (CVE-2021-28957) was discovered in the HTML Cleaner by Kevin Chung, which allowed JavaScript to pass through. The cleaner now removes the HTML5 formaction attribute.

4.6.2 (2020-11-26)

Bugs fixed

A vulnerability (CVE-2020-27783) was discovered in the HTML Cleaner by Yaniv Nizry, which allowed JavaScript to pass through. The cleaner now removes more sneaky "style" content.

4.6.1 (2020-10-18)

... (truncated)

Commits

a9611ba Fix a test in Py2.

a3eacbc Prepare release of 4.6.5.

b7ea687 Update changelog.

69a7473 Cleaner: cover some more cases where scripts could sneak through in specially...

54d2985 Fix condition in test decorator.

4b220b5 Use the non-depcrecated TextTestResult instead of _TextTestResult (GH-333)

d85c6de Exclude a test when using the macOS system libraries because it fails with li...

cd4bec9 Add macOS-M1 as wheel build platform.

fd0d471 Install automake and libtool in macOS build to be able to install the latest ...

f233023 Cleaner: Remove SVG image data URLs since they can embed script content.

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0

Releases(v1.0)

v1.0(Jul 17, 2021)

Source code(tar.gz)
Source code(zip)
v0.1(Jul 16, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Rishabh

CSE Undergrad at IIIT-D.

GitHub http://www.newsemble.ml/news

Newsemble is an API that provides easy access to the current news for programmatic analysis

Related tags

Overview

📰 Newsemble 📰

🔖 About 🔖

🗒️ Table of contents

💻 Technologies

📂 File Structure and Description

🛠️ Pipeline

🚀 Getting-started

⚙️ Currently Supported Sites

🙏 Thanks!

✍️ Authors ✍️

You might also like...

music downloader written in python. (Uses jiosaavn API)

Mobile based API for Crunchyroll BETA (and Downloader).

Pypixiv - A fully-typed, asynchronous api wrapper for pixiv

This project is helps to download contents from Streamtape by utilizing the API

A python script that discovers hidden YouTube API clients. Just a research project.

Simple Python script to download images and videos from public subreddits without using Reddit's API 😎

This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular intervals.It sends out the most recent news at random!

News-app - This is a news web app for reading news from different sources and topics

Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and other financial information. This repository provides an SDK for developing applications to access the NCDS.

The windML framework provides an easy-to-use access to wind data sources within the Python world, building upon numpy, scipy, sklearn, and matplotlib. Renewable Wind Energy, Forecasting, Prediction

Repo Home WPDrawBot - (Repo, Home, WP) A powerful programmatic 2D drawing application for MacOS X which generates graphics from Python scripts. (graphics, dev, mac)

A curated list of programmatic weak supervision papers and resources

Learning trajectory representations using self-supervision and programmatic supervision.

Programmatic interface to Synapse services for Python

Django API that scrapes and provides the last news of the city of Carlos Casares by semantic way (RDF format).

This python module can analyse cryptocurrency news for any number of coins given and return a sentiment. Can be easily integrated with a Trading bot to keep an eye on the news.

NLP project that works with news (NER, context generation, news trend analytics)

Your own movie streaming service. Easy to install, easy to use. Download, manage and watch your favorite movies conveniently from your browser or phone. Install it on your server, access it anywhere and enjoy.

💰 An Alfred Workflow that provides current price of cryptocurrency

Comments

Bump lxml from 4.5.0 to 4.6.5

4.6.5 (2021-12-12)

Bugs fixed

4.6.4 (2021-11-01)

Features added

4.6.3 (2021-03-21)

Bugs fixed

4.6.2 (2020-11-26)

Bugs fixed

4.6.1 (2020-10-18)

Releases(v1.0)

v1.0(Jul 17, 2021)

v0.1(Jul 16, 2021)

Owner

Rishabh

S3 file download with Python and access with VBA

this is udemy course downloader, before a start you know how to get access token.

DYA ( Ditch YouTube API ) is a package created to power the user with YouTube Data API functionality without any API Key

Using Youtube downloader is the fast and easy way to download and save any YouTube video.

Easy automated ebook downloader using openbooks as the backend

Download Youtube videos in mp4 format in a fast, easy, convenient way made with Python!

You Can download any video/image in all social medias very easy and High Speed.

Storing, versioning, and downloading files from S3 made as easy as using open() in Python. Caching included.

Twayback: Downloading deleted Tweets from the Wayback Machine, made easy

A tool to make easy to search for directories in the URL.