Find more Web Development Remote Jobs posted recently Worldwide

Required Automation,Browser Automation,Chrome Extension,Data Scraping,Node.js freelancer for WhatsApp Web Scrape Dynamic/Updating/Changing Page: Download & sync messages + media per chat/group job

Posted at - Jun 6, 2020

Toogit Instant Connect Enabled

We need a script/executable/process which automates browsing to WhatsApp Web, go to selected chats & groups, and sync down & download the full ordered sequential data-consistent message history out of the live/changing page to a saved format (database / HTML structured data files) and media files downloaded to a related file storage directory.
We need the source code of this working project clean readable and well designed code-base, properly indented & spaced, good coding practices and up to date libraries & dependencies etc.

Target Browser: Google Chrome. Optional: Also other/cross browser.

Possible technologies:
- Selenium WebDriver with Java, Python, Node.js JavaScript etc.
- Chrome Extension using HTML/CSS/JavaScript (lacking database/file-system facilities)
- Tampermonkey Userscript using JavaScript (lacking database/file-system facilities)
- Other - Any appropriate technology stack you might recommend

Data Wanted: Any & all available data that can be obtained out of the webpage.
Some data points examples: Group/chat name, message date & time, all author information; phone number and/or name when available etc., full message content with emojis & Unicode content & links etc., all media (audio, image, video, documents etc.), in reply to message references, ordinal/position in reference to previous and next message, author/group/status & quote content from referenced message, is message deleted, is message forwarded (and forwarded details), and ANY other obtainable data from the webpage.

Some requirements descriptions and known complications/challenges:
- There needs to be communication between the already downloaded persisted database & file store, and the script processing the webpage, in order to identify messages and find what has already been downloaded what is already synced down & up to date, and what still needs to be fetched from the webpage.
- Data consistency; distinctly identifying messages correctly (perhaps by hashing a digest based on message group + date & time + message content + ordinal position relative to previous/next messages + other identifying factors etc.)
- Progressively & continuously syncing history from the web whatsapp interface down to database/file store, keeping track of where the process is holding, which messages are already synced, which need updating/re-downloading etc.
- Indexing & searching/seeking into message history
- Will probably need a lot of Deferred & Asynchronous processing; waiting for media to download into browser, detect timeouts/failures, retry, saving state about what needs to be retried again in future processing
- Possibly using the group info/chat info Media panel to access & load media files
- Apparently the only way to get full message text with emojis etc. in whole is by selecting the full text content and copying to clipboard.
- Dealing with long messages collapsed 'Read more' content
- Scrolling up/down the message history roll and waiting for messages to load in infini-scroll loading batches.
- Downloading media, waiting for media to load, retry failures, store in organized folder structure with relations saved to database/HTML data files.
- How to download video clips
- Other complications that might be discovered in the effort to sync down whatsapp per-chat history

About the recuiterMember since Mar 14, 2020 Atoz Directory
from Osun, Nigeria

Skills & Expertise Required

Automation Browser Automation Chrome Extension Data Scraping Node.js 

Candidate shortlisted and hiredHiring open till - Apr 2, 2021

Work from Anywhere
40 hrs / week
Fixed Type
Remote Job

Looking for help? Checkout our video tutorial
How to search and apply for jobs

How to apply? Do you have more questions about the Job?
See frequently asked questions

Apply on more work from home jobs posted in Web Development category.

Related Jobs

Latest In Automation Jobs

Latest In Browser Automation Jobs

Latest In Chrome Extension Jobs

Latest In Data Scraping Jobs

Latest In Node.js Jobs