Real-time Web Scraping for a list of Large News Website (need scalability)

The task is straight forward: design an script (python preferred) to scrape all article information from a News website in real-time.

There are a few requirements though:
1. Real-time: able to keep the program running and detect any new posted articles.
2. Web application compatible: the script should be capable of being integrated into a web application so that the article information can be pushed onto the web app realtime.
3. Production ready: the script should be able to handle most the exception and detect any anomaly and be production-ready
4. For those who are also web app developer, it would be a plus if you can also handle the integration with the web app.
About the recuiter
Member since Mar 14, 2020
Dilip Babu
from Sao Paulo, Brazil

---

---

Looking for help? Checkout our video tutorial
How to search and apply for jobs

How to apply? Do you have more questions about the Job?
See frequently asked questions