QA

Job Summary

We are seeking a QA Engineer with experience in testing cloud-based data platforms, ETL pipelines, and Databricks solutions. The ideal candidate will be responsible for validating data quality, data transformations, API integrations, and end-to-end data workflows within the Databricks Lakehouse environment. The role requires strong SQL skills, data validation expertise, and a thorough understanding of data engineering concepts.

Key Responsibilities

Data Validation & Testing

• Validate data ingestion, transformation, and loading processes in Databricks.
• Develop and execute test cases for ETL/ELT pipelines.
• Perform source-to-target data validation and reconciliation.
• Validate data quality, completeness, accuracy, and consistency across datasets.
• Perform record count, aggregation, null check, duplicate check, and business rule validations.

Functional & Integration Testing

• Test Databricks notebooks, workflows, and jobs.
• Validate API integrations and downstream system consumption.
• Perform end-to-end testing of data pipelines across Bronze, Silver, and Gold layers.
• Verify data model implementation and business logic against requirements.

Automation & Performance Testing

• Develop automated test scripts for data validation.
• Execute regression testing for data pipelines and APIs.
• Support performance and volume testing of large-scale datasets.
• Identify bottlenecks and work with development teams to improve performance.

Defect Management & Reporting

• Log, track, and manage defects through closure.
• Work closely with Data Engineers, Data Modelers, Architects, and Business Analysts.
• Prepare test reports, test evidence, and sign-off documentation.
• Participate in requirement reviews and quality assurance activities.

Required Skills

• 3-6 years of experience in Data Testing / ETL Testing.
• Hands-on experience testing Databricks-based solutions.
• Strong SQL skills for complex data validation.
• Experience with PySpark and understanding of Spark transformations.
• Experience in source-to-target validation and data reconciliation.
• Knowledge of ETL/ELT processes and Data Warehousing concepts.
• Experience in API testing using Postman or similar tools.
• Understanding of Delta Lake and Lakehouse Architecture.
• Familiarity with defect tracking tools such as JIRA or Azure DevOps.

Preferred Skills

• Experience with Databricks Workflows and Jobs.
• Knowledge of Azure Data Factory (ADF).
• Experience with test automation frameworks.
• Exposure to CI/CD pipelines and DevOps practices.
• Understanding of Data Governance and Data Quality frameworks.
• Experience with cloud platforms such as Azure.

Educational Qualification

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

Key Competencies

• Strong analytical and troubleshooting skills.
• Attention to detail and data accuracy.
• Good communication and stakeholder management skills.
• Ability to work independently and within Agile teams.
• Strong documentation and reporting capabilities.

Mandatory Skills

• Databricks Testing
• SQL
• Data Validation
• ETL/ELT Testing
• Source-to-Target Mapping Validation
• API Testing
• Azure Databricks
• Defect Management

Nice to Have

• PySpark
• Azure Data Factory (ADF)
• Test Automation
• CI/CD
• Delta Lake
• Power BI Validation
• Performance Testing