Lead Data QA Engineer

Job Description

As a QA Lead Engineer, you will play a crucial role in providing technical leadership and hands-on programming expertise to drive the testing strategy and quality assurance for data and analytics initiatives. You will work closely with Data Engineers and business stakeholders to ensure the highest quality of data pipelines, ETL processes, and analytics deliverables.**Key Responsibilities:**- Act as the QA Lead Engineer, designing and optimizing testing processes for data and analytics functions.- Code and maintain automated test scripts, frameworks, and utilities in Python, PySpark, and Spark SQL to validate data pipelines and transformations.- Support the data engineering team by contributing to data transformation tasks and ensuring smooth execution of data pipelines.- Take ownership of the testing strategy, driving the design, execution, and reporting of test cases using various testing frameworks and tools.- Lead and actively develop/customize testing frameworks and methodologies to meet business objectives and testing requirements.- Provide end-to-end technical leadership for implementing testing processes and maintaining existing test frameworks and applications.- Implement and promote best practices in test automation, manual testing, performance testing, and integration testing.- Write efficient, maintainable, and scalable test scripts to maximize testing coverage, accuracy, and issue detection.- Create and maintain detailed documentation for testing processes, test cases, and results.- Collaborate with business users and stakeholders to understand testing needs and recommend solutions aligned with business goals.- Partner with Data Engineers to validate data pipelines, ETL processes, and data integrity during testing phases.- Build simple Power BI reports and dashboards to track and visualize test results and quality metrics.**Technical Skills & Qualifications:**- Hands-on expertise in Python for test automation, scripting, and data validation.- Strong skills in PySpark and Spark SQL for validating and testing large-scale data transformations.- Experience with Azure Data Factory to orchestrate and validate ETL workflows.- Proficiency in Azure Databricks for data processing and distributed testing of data pipelines.- Ability to create simple Power BI reports for test result visualization or reporting.- Proven experience implementing, customizing, and maintaining testing frameworks in data and analytics environments.- Deep understanding of test automation, performance testing, integration testing, and QA best practices for data-centric systems.- Excellent communication and stakeholder engagement skills. As a QA Lead Engineer, you will play a crucial role in providing technical leadership and hands-on programming expertise to drive the testing strategy and quality assurance for data and analytics initiatives. You will work closely with Data Engineers and business stakeholders to ensure the highest quality of data pipelines, ETL processes, and analytics deliverables.**Key Responsibilities:**- Act as the QA Lead Engineer, designing and optimizing testing processes for data and analytics functions.- Code and maintain automated test scripts, frameworks, and utilities in Python, PySpark, and Spark SQL to validate data pipelines and transformations.- Support the data engineering team by contributing to data transformation tasks and ensuring smooth execution of data pipelines.- Take ownership of the testing strategy, driving the design, execution, and reporting of test cases using various testing frameworks and tools.- Lead and actively develop/customize testing frameworks and methodologies to meet business objectives and testing requirements.- Provide end-to-end technical leadership for implementing testing processes and maintaining existing test frameworks and applications.- Implement and promote best practices in test automation, manual testing, performance testing, and integration testing.- Write efficient, maintainable, and scalable test scripts to maximize testing coverage, accuracy, and issue detection.- Create and maintain detailed documentation for testing processes, test cases, and results.- Collaborate with business users and stakeholders to understand testing needs and recommend solutions aligned with business goals.- Partner with Data Engineers to validate data pipelines, ETL processes, and data integrity during testing phases.- Build simple Power BI reports and dashboards to track and visualize test results and quality metrics.**Technical Skills & Qualifications:**- Hands-on expertise in Python for test automation, scripting, and data validation.- Strong skills in PySpark and Spark SQL for validating and testing large-scale data transformations.- Experience with Azure Data Factory to orchestrate and validate ETL workflows.- Proficiency in Azure Databricks for data processing and distributed testing of data pipelines.- Ability to create simple Power B

Keyskills :
PythonPower BIPySparkSpark SQLAzure Data Factory