Scrapybot on AWS [ec2, IAM, docker], Firebase, Python, Scrapy, Github

Main developer had a family emergency, project 70% complete


> Scrapybot built-in Python to scrape 170 links. The bot will enter a first(given) + last(surname) and a birthday, if there is a return (a record), the information will be displayed in a dashboard to the user performing the search.

PROJECT

> The project is a scrapy bot built-in Python that searches 170 links (search a database for a persons name) and returns the result to a user dashboard. Some of the (170) links have Captcha, ReCaptcha, click to agree, click Yes or similar (70% of the code is complete). The code is commented well. All the links in the code are numbered (e.g., #12--- CAPTCHA, #14- CAPTCHA WITH PICTURES) and the code. If there is no bypass mechanism, then one needs to be implemented. It is important that the main links.csv be checked to make sure every link is implemented in the code.) Roughly 100 more links need to be coded.

USER DASHBOARD (My front-end dev is designing the front end)

> There should be a user dashboard that will allow a user (the one performing the search) to log-in and get an update on their search. An email should be sent to the user performing the search. Once their search is complete, the user will get a second email indicating they should check their dashboard for the results. There is a front-end dev that you will collaborate with via Toogit chat and Github

CAPTCHA BYPASS

I have an account with purchased 10 residential IPs. We can get more or less; I just need you to tell me after we get all of the sites integrated in the code. As I said in the `PROJECT` section, the code is commented for each link, there are about 100 more links that need to be coded.

LIBRARIES & SERVICES

> The previous dev used URLLIB and pytesseract. The plan was to use our service to bypass the captchas that could not be fooled with a bot. The dev never implemented 2captcha service.

AUTHENTICATION & AUTHORIZATION: (not implemented)

Firebase should be used as the authentication method; (you will be added as a collaborator on the project in Firebase) It should have email, Facebook and Google signups.

Firebase Authorization should allow access to a subscription service for $150usd a year.

RESOURCES & ACCESS: (Amazon Web Services) (not implemented)



The site is to be built on Docker AWS; EC2,S3, etc. )



IAM (AWS) privileges to be granted to you upon a successful Toogit contract.



Coding best practices should be used, (commenting, grouping, naming, DRY, using try, etc.)



You must commit to GitHub weekly (no exceptions, we dont care if its one line of code)



We may use Trello for team collaboration



Do not send me a generic response. Tell me how you will comply with the project requirements. We expect the incumbent to be engaged in the project, communicate, work with the other developer via Toogit chat and GitHub.
About the recuiter
Member since May 20, 2018
Thankamsyachtpa
from Val-d'Oise, France

---

---

Looking for help? Checkout our video tutorial
How to search and apply for jobs

How to apply? Do you have more questions about the Job?
See frequently asked questions