I am running existing code on linux VM using node to scrape websites in which I need to transition to goLang. I prefer using a framework go-colly.
We are using a reverse proxy service and will provide a list of custom user agents to keep identities unique and prevent us from getting banned.
For new sites it will be a unique case but have an overall strategy on how to scrap, currently, we are going to scrap network requests and map all the data from product single pages related to a certain category (groceries). This will be case by case basis on what categories to scrap.
Currently, just one VM. Using VSTS dev-ops service for CI/CD deploy to Linux VM
Scrapping will be done frequently (daily). If you can give me an idea of staples.com estimation.
1. Food Category EP for product single.
2. Scroll infinite scroll category pages for all food sub categories to get to product single.
About the recuiterMember since May 20, 2018 Peter J Krenz
from Bolivar, Venezuela