Automation of ETL process based in AWS based on S3 and Redshift

We are looking for an engineer with experience in setting up ETL processes in AWS allowing simple integrations with multiple entities with different policies.

The project will consist in setting up production-ready MVP infrastructure using existing AWS tools (i.e. Glue, etc) within a 1-2 week period working alongside our team.

If this initial project is successful there we are also looking for an opportunity to develop a long term collaboration.

Project Details:

(i) Different entities will be sharing two types of structured files (.csv) so we can load them in S3 buckets periodically (most likely on a daily basis).
(ii) These files would need to be aggregated and processed into a unique well-formatted file living in AWS (preference for Amazon Redshift).
(iii) An additional processing step will take place on this Redshift instance to create an output file to be send to the same SFTP connection.
(iv) Policies on access would need to be managed for all existing partners.

We have the the logic for all the aggregation and processing stages but mostly need help to create a stable MVP version on AWS separating Staging and Production environments.
About the recuiter
Member since Mar 14, 2020
Farooq Ahmad
from Tamil Nadu, India

---

---

Looking for help? Checkout our video tutorial
How to search and apply for jobs

How to apply? Do you have more questions about the Job?
See frequently asked questions