Real-time Web Scraping for a list of Large News Website (need scalability)




Ca Ankit India


OpenMay 5, 2018
The task is straight forward: design an script (python preferred) to scrape all article information from a News website in real-time.

There are a few requirements though:
1. Real-time: able to keep the program running and detect any new posted articles.
2. Web application compatible: the script should be capable of being integrated into a web application so that the article information can be pushed onto the web app realtime.
3. Production ready: the script should be able to handle most the exception and detect any anomaly and be production-ready
4. For those who are also web app developer, it would be a plus if you can also handle the integration with the web app.

Skills & Expertise Required

Python Scrapy Selenium Web Scraping 

Offer to work on this project closes in 79 days!

Submit A Proposal

Share this project with your friends

Similar Projects

python block chain

create blocks from strings from a queue and perform merle tree operation, result while be stored in a block along with the previous block hash

Hourly, $18.00

Android app with a web-based dashbo...

Deliverable: Android App and web dashboard.

Our product is similar to Yelp! and Zomato (with 1 major difference) where our clients (restaurants) need to manage their info, pictures and more

Fixed, $1,500.00

Odoo - Print Multiple PDF Reports o...

Hello, I am looking for an Odoo developer who can solve our challenge

How to Print Multiple PDF Reports on Single Button Click

Looking forwards

Hourly, $20.00

Company Directory Search + Scraping...


We are compiling a directory of U.S minority-owned businesses to identify potential award winners. This project is to:

1.) Efficiently search for website directories and more

Hourly, $28.00

Here are our top professional picks for you to hire

Customer Support HTML5 CSS3 
Automation tester
Core Java Python Selenium WebDriver 
$2 /hr
Data Entry Graphic Design Web Search 
$2 /hr
Software Developer
Python Android Machine Learning 
$5 /hr
Manual Testing Selenium Selenium WebDriver 
Web Scraper
Web Scraping Data Entry Data Mining 
$5 /hr
System Architecture
Odoo Python AngularJS 
$19 /hr
Security Researcher| bug hunter | penetration testing
Python Developers Python Linux 
$10 /hr
Software development expert engineer
Python Golang C Programming 
$100 /hr
To view more profile join Toogit

Get Started