Apache Airflow - South Africa
A1L
Our client in the Telecommunications sector is looking for an experienced Apache Airflow. This is a contract role for Three months. Description: This is a 3-month remote engagement working for a large telco. They will be working with a team supporting a Big Data Platform consisting of an ever-growing list of Airflow DAGs and associated HQL jobs. Apache Airflow is part of a much larger Cloudera implementation. They will be expected to help stabilise existing Apache Airflow DAGs, to ensure predictable and stable execution of the existing jobs. They will be joining the core platform team as a lead Airflow administrator, and developer and assisting Operations to effectively monitor Apache Airflow. Additionally they will be expected to be ensure that act as a mentor to at least two more junior consultants in order to advance their understanding and competence with Apache Airflow. Requirements: Design, develop, and maintain data pipelines using Apache Airflow to ensure data reliability and availability Proficiency in Python, and PySpark to support optimal DAG job design and development Experience with Apache Airflow setup and configuration, including Installation and Configuration, Database Management, Experience with monitoring, resource management and integration with third-party APM tools. In-depth understanding of troubleshooting, stabilising and fine-tuning Apache Airflow Experience with integrating Apache Airflow with the Application Performance Monitoring (APM) software. Experience with Apache Hive databases and SQL (Oracle and MySQL) Nice-to-haves In-depth experience and understanding of the Cloudera Data Platform.
Apply Now
Facebook
WhatsappRelated Jobs
Data Base Administrator Midrand - Midrand
A1L Digital
...
Sql Administrator Stellenbosch - South Africa
Oxbridge Academy
...
Database Administrator - Midrand
...
Senior Devops Engineer Johannesburg - Johannesburg
E-Merge IT Recruitment
...
Sap Consultant Centurion - Centurion
Liyema Consulting
...