Looking for an experienced python developer to help with a data ingestion process. Needs:
- Data Engineering with Python 3
- Both Functional and OOP Algorithms
- Packaged with modules
- Advanced data structures
- Incorporate JSON "control" file
o how to validate/QA/cleanse erroneous or poor formatted incoming CSV files
o process & pipeline movement of data with compression, formats, local or remote destination, etc.
o error reporting (multiple file logging, email)
- Batchable (non-interactive) & interactive command-line driven
- CSV data cleaning/correcting
o Tracking all the cleaning operations performed on the data
o Tracking/reporting errors
- In-line/by-record Data transformations
- Familiarity of dev environments in Linux, git, and Linux/bash scripting
- AWS experience, such as S3 is an advantage
- AWS Lambda, Glue/Athena (Apache Hive) is an advantage
- multiprocessing in Python is an advantage
About the recuiterMember since Sep 14, 2017 Frankie Puckett
from California, United States