I need to convert JSON, Avro or other row-based format files in S3 into Parquet columnar store formats using an AWS service like EMR or Glue.
I already have code that converts JSON to parquet using Python but the process is very manual, accounting for NULL values in the JSON elements by looking at each and every field/column and putting in default values if there's a NULL.
I am looking for an easier, less manual way of doing this using something like Spark or other similar methods.
Since I am working exclusively on AWS, I am only looking for solutions using AWS services such as EMR, Glue or similar AWS service.
I am thus looking for someone with experience using AWS EMR, Glue, Python, Pyspark etc.
Please note: Since this is going to be a learning experience for me, this is going to be a live session on Skype, Zoom, Google Hangouts etc where you code and I watch and you answer any questions I have in the process.
Thus, I will pay in one-hour increments. The initial contract is going to be for one hour and if we need more time we can have another one hour contract and so on and so forth.
Please only apply if you're ok with all these conditions and have the required experience.
About the recuiterMember since May 20, 2018 Dibyashakti Moh
from Karnataka, India