Duties:

Responsible for solution design and development of Various Functionality in AWS for the project flow
Develop and maintain automated ETL pipelines (with monitoring) using scripting languages such as Python, SQL and AWS services such as S3, Glue, Lambda, SNS, Redshift, SQS, KMS
Develop Glue Jobs for batch data processing and created Glue Catalog for metadata sync up
Develop Data Pipelines using AWS lambda, Step Functions for data processing
Manage data coming from different sources
Experienced with AWS services to smoothly manage application in the cloud and creating or modifying instances.
Implement with Scaled Agile Framework (SAFe)
Be involved with the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark
Be involved in creating Hive Tables, loading with data and writing Hive queries

Â
Requirements: