Data Test Engineer As a data test engineer working in an agile project methodology to automate functional, non-functional, and business test cases in an Azure Databricks environment with Power BI and Plotly, the key responsibilities would include: Test Planning and Design -Collaborate with the development team, product owners, and stakeholders to understand project requirements and define test strategies -Create and maintain test plans, test cases, and test scenarios covering functional, non-functional, and business aspects of the data pipeline and analytics solutions. -Identify areas suitable for test automation and prioritize test cases based on their impact and criticality. · Test Automation Development: -Develop and implement automated test scripts using Python to validate the functionality, performance, and data integrity of the Azure Databricks environment. -Utilize Python libraries and frameworks such as pytest or unittest to create robust and maintainable test automation scripts. - -Integrate test automation with CI/CD pipelines to enable continuous testing and ensure the reliability of the data pipeline. · Data Validation and Quality Assurance: Data Validation and Quality Assurance: -Develop automated data validation scripts to ensure the accuracy, completeness, and consistency of data processed in the Azure Databricks environment. -Perform data profiling, data quality checks, and data reconciliation to identify and report data anomalies or discrepancies. -Collaborate with data engineers and analysts to establish data quality standards and implement data validation rules. Power BI and Plotly Testing: -Create and execute test cases to validate the functionality, usability, and performance of Power BI reports and dashboards. -Ensure the accuracy and consistency of data visualizations created using Plotly. -Test the integration between Azure Databricks, Power BI, and Plotly to ensure seamless data flow and visualization. Non-Functional Testing: -Conduct performance testing to evaluate the scalability, responsiveness, and efficiency of the data pipeline and analytics solutions. -Perform load testing to assess the system's behaviour under various workload conditions. -Execute security testing to identify potential vulnerabilities and ensure data privacy and compliance. Agile Collaboration and Communication: -Actively participate in agile ceremonies such as sprint planning, daily stand-ups, and retrospectives. -Collaborate closely with data engineers, data analysts, and other stakeholders to align testing efforts with project goals and timelines. -Provide regular updates on test progress, identified issues, and test metrics to the agile team and stakeholders. Test Reporting and Documentation: -Document test cases, test scenarios, and test results in a clear and comprehensive manner. -Generate test reports and dashboards to provide visibility into the quality and stability of the data pipeline and analytics solutions. -Maintain traceability between requirements, test cases, and defects to ensure comprehensive test coverage. Continuous Improvement: -Continuously review and optimize the test automation framework and scripts to improve efficiency and maintainability. -Stay up-to-date with the latest testing methodologies, tools, and best practices in the data engineering and analytics domain. -Identify opportunities for process improvements and implement automation initiatives to enhance the overall quality assurance process. 1. Hands-on experience with Azure Databricks, including data ingestion, processing, and transformation using Apache Spark and Delta Lake. 2. Familiarity with data testing frameworks and tools for data quality assurance and validation. 3. Experience in developing and maintaining test automation frameworks using Python and its testing libraries. 4. Proficiency in writing complex SQL queries and working with relational databases. 5. Experience with data visualization tools like Power BI and / or Plotly for creating interactive dashboards and reports. 6. Exposure to agile development methodologies and collaboration with cross-functional teams. 7. Familiarity with version control systems (e.g., Git) and CI/CD pipelines for automated testing and deployment. 8. Experience in performance testing, load testing, and security testing of data pipelines and analytics solutions. 9. Knowledge of data governance, data quality management, and data compliance regulations (e.g., POPIA, GDPR) relevant to the industry. Market related Apply Now

Share this job with someone you think should apply!

Facebook

Data Engineer Sandton, Johannesburg - South Africa

Related Jobs

Data Engineer Sandton - Sandton

Senior Data Engineer Sandton - South Africa

Senior Azure Data Engineer – Johannesburg – R1.2mil to R1.4mil per annum - Four Ways North

Intermediate Full Stack BI Consultant - Cape Town City Centre

Powerbi Developer Johannesburg - Johannesburg

Want to do another search?