Real-time Web Scraping for a list of Large News Website (need scalability)




Ca Ankit India


OpenMay 5, 2018
The task is straight forward: design an script (python preferred) to scrape all article information from a News website in real-time.

There are a few requirements though:
1. Real-time: able to keep the program running and detect any new posted articles.
2. Web application compatible: the script should be capable of being integrated into a web application so that the article information can be pushed onto the web app realtime.
3. Production ready: the script should be able to handle most the exception and detect any anomaly and be production-ready
4. For those who are also web app developer, it would be a plus if you can also handle the integration with the web app.

Skills & Expertise Required

Python Scrapy Selenium Web Scraping 

Offer to work on this project closes in 133 days!

Submit A Proposal

Share this project with your friends

Similar Projects

ETL Expert with experience in Googl...

We are looking for a ETL Expert that can help us automate data acquisition. We have plans and connections and principles, need someone that can replicate and implement. The tools you HAVE to be more

Hourly, $18.00

Bayes Rule and Information Cascades

Bayes' Rule and Information Cascades
This examples demonstrates how Bayes' Rule can be used to model information cascades.

You need to simulate how a set of rational players would more

Fixed, $100.00

Web scrapping for Real Estate

Create a database with data from individual property entries - each apartment, detached, land, building, farm, garage, office - up to a max of 34 fields - scrapping 8 real estate websites.
The more

Fixed, $2,200.00

Here are our top professional picks for you to hire

Web developer
PHP javascript jquery 
$35 /hr
Backend Engineer JAVA
Java Apache Solr Spring Framework 
$20 /hr
Mathematics Expert; Machine Learning and Finance Enthusiast and loves to code in Python
Mathematics Business Mathematics Calculus 
$10 /hr
I Have The Ability To Assist You......
Lead Generation Data Entry Web Scraping 
$3 /hr
Automation Engineer
Software Testing Automated Testing HP QuickTest Professional (HPQTP) 
$16 /hr
Data consultant/ Solution Architect
Hadoop Oracle Database Microsoft SQL Server 
$8 /hr
Data Scientist
Data Science Python Machine Learning 
$2 /hr
Technology loving software developer, interested in all things computer science.
Web Scraping Java Java Developers 
$2 /hr
Cloud Engineer
AWS Linux System Administration Git 
$8 /hr
To view more profile join Toogit

Get Started