and maintaining their data infrastructure and pipelines. You will collaborate closely with the data scientists develop, and maintain scalable and efficient data pipelines and ETL processes to ingest, transform, and load integrity of data. ▪ Monitor and troubleshoot data pipelines to identify and resolve issues in a timely manner g., Pandas, PySpark). ▪ Familiarity with data pipeline orchestration tools (e.g., Apache Airflow, Luigi
/>- Experience building data pipelines using AWS Glue or Data Pipeline, or similar platforms.
-
Store
- Secrets Manager
- CodeBuild/Pipeline
- CloudFormation
- Business Intelligence
responsible for building and maintaining Big Data Pipelines using Cloud Data Platforms. Data Engineers are
Store
- Secrets Manager
- Code Build/Pipeline
- CloudFormation
- Business Intelligence
/>- Experience building data pipelines using AWS Glue or Data Pipeline, or similar platforms.
-
responsible for building and maintaining Big Data Pipelines using Data Platforms. Data Engineers are custodians
work by provisioning automated build and deploy pipelines. Key Responsibilities and/or Outputs Design and and maintain automated build pipelines and automated deployment pipelines Manage internal and external build, maintain and optimize automated build pipeline (TFS, GitHub Actions, Azure DevOps, Bamboo, Jenkins
work by provisioning automated build and deploy pipelines. Key Responsibilities and/or Outputs Design and and maintain automated build pipelines and automated deployment pipelines Manage internal and external build, maintain and optimize automated build pipeline (TFS, GitHub Actions, Azure DevOps, Bamboo, Jenkins
usable data to stakeholders. They also create data pipelines, Ingestion, provisioning, streaming, self-service ta Pipeline Build (Ingestion, Provisioning, Streaming and API): Build and maintain data pipelines to:
non-functional, and business aspects of the data pipeline and analytics solutions. Identify areas suitable with CI/CD pipelines to enable continuous testing and ensure the reliability of the data pipeline. Data Validation scalability, responsiveness, and efficiency of the data pipeline and analytics solutions. Perform load testing visibility into the quality and stability of the data pipeline and analytics solutions. Maintain traceability
Programming languages such as Python and Big Data pipelines. This position is based on a Hybrid working model Programming languages such as Python and Big Data pipelines such as ETL, SQL, Spark etc. Ability to develop branching strategies and development approaches CI/CD pipeline development using Code Build and Github Actions Coverage Terraform and IAC deployments PySpark pipeline development Knowledge of multiple database types when to create alerts and setting up alerting pipelines. Be comfortable with Data Structures and Algorithms
South Africa You'll build and maintain Big Data Pipelines, ensuring data integrity and compliance. Leverage Function - Param Store - Secrets Manager - Code Build/Pipeline - CloudFormation - Business Intelligence (BI) multi-task. - Experience building data pipelines using AWS Glue or Data Pipeline, or similar platforms. - Familiar responsible for building and maintaining Big Data Pipelines using Data Platforms. Data Engineers are custodians
- Experience building data pipeline using AWS Glue or Data Pipeline, or similar platforms.
-
Function