We are looking for an engineer with experience in setting up ETL processes in AWS allowing simple integrations with multiple entities with different policies.
The project will consist in setting up production-ready MVP infrastructure using existing AWS tools (i.e. Glue, etc) within a 1-2 week period working alongside our team.
If this initial project is successful there we are also looking for an opportunity to develop a long term collaboration.
Project Details:
(i) Different entities will be sharing two types of structured files (.csv) so we can load them in S3 buckets periodically (most likely on a daily basis).
(ii) These files would need to be aggregated and processed into a unique well-formatted file living in AWS (preference for Amazon Redshift).
(iii) An additional processing step will take place on this Redshift instance to create an output file to be send to the same SFTP connection.
(iv) Policies on access would need to be managed for all existing partners.
We have the the logic for all the aggregation and processing stages but mostly need help to create a stable MVP version on AWS separating Staging and Production environments.
About the recuiterMember since Mar 14, 2020 Farooq Ahmad
from Tamil Nadu, India