I am looking for an individual who would help me find / come up with / specify the best automated solution for data processing. The goal is to come up with such a solution and its specification that could be then used as another job posting on Toogit.
The current situation is:
- at the moment we are processing large amounts of data, mostly points of interest (POI)
- some are gathered via automated scrapers and verified manually, some are collected entirely manually
- we have no problem gathering the data but we struggle with updating them - we can do it, it just is very time consuming.
I am seeking an automated solution, a database solution of sort, which would keep our data up to date. That means:
- get data, either by API or by some manual import (CSV, JSON,...);
- compare those data with what it has in storage;
- update stored data accordingly;
- allow distribution of stored data, either as whole or changes only.
I would prefer the possible solution to use the G Suite infrastructure, but it's not a must-have.
For your cover letter please consider the following scenario and suggest a solution:
- we are processing an ATM database
- this database is delivered to our clients on monthly basis
- clients don't care whether the database is delivered as a whole (however in this case they require each item to have a unique identifier) or in form of changes only
- data get scraped and a human sorts them to a particular structure so all information are comparable
- we need a solution that wouldn't neccessarily eliminate the human-sorting labour but that would ensure that data processed in this way are processed with already existing data resulting in an up-to-date data set.
About the recuiterMember since Sep 12, 2017 Marina Kuznetso
from Moscow, Russia