Jobs
Interviews
DataArt logo

Senior Data QE Engineer, Hyderabad-based

DataArt
Bengaluru

Posted:10 hours ago

Job Description

Client: Our client is a global organization focused on professional education and certification.

  • Project overview: The client runs a large application landscape and the analytical datasets behind it. Data moves from PostgreSQL sources on AWS into Snowflake through orchestrated pipelines. Quality for that data is part of delivery: test design, checks on movement and transformation, quality metrics, and practical help when data has to be corrected.
  • Data QE engineers own that hands-on work for data moving from source systems into analytical datasets, from test design through execution, investigation, and release-readiness reporting.

  • Position overview: We are looking for a Senior Data QE Engineer with solid experience in ETL testing and data validation, and strong SQL skills. You will design and run data quality tests, validate pipelines end to end, reconcile source and analytical data, and give engineering and business teams evidence for release decisions.
  • You will also automate comparisons in Python, work with Airflow pipeline runs, and use AWS and Grafana to observe pipeline health.
  • You will use an AI coding assistant to analyze data and author SQL, Python, and quality checks, and you will review generated output before it is used.
  • This position is intended for Hyderabad-based candidates who are available to work from the office.

  • Technology stack: SQL, Snowflake, PostgreSQL, AWS Aurora, Python, pytest, Airflow, AWS, CloudWatch, Secrets Manager, EKS, Grafana, GitHub, GitHub Actions, TestRail, Jira, AI coding assistants, Cursor, GitHub Copilot, dbt

  • Responsibilities: Own data quality for data moving from source systems into analytical datasets, from test design through execution, investigation, and reporting.
  • Design, implement, and execute data quality tests, and publish quality metrics and dashboards that producing and consuming groups can use.
  • Validate pipelines end to end, covering movement, transformation, completeness, history, freshness, and compliance.
  • Write reconciliation, drift, and quality SQL on Snowflake layered models (raw, history, silver, and gold), including slowly changing dimensions and merge behavior.
  • Validate PostgreSQL sources with consistent snapshot counts and checks that logical replication kept the expected data.
  • Use logical and physical data models and data contracts to assess downstream impact and set quality coverage.
  • Automate data comparison and reconciliation in Python with pytest, and keep the checks in GitHub.
  • Trigger pipeline runs in Airflow and read task results into the quality verdict.
  • Investigate quality issues and support remediation with the groups that produce and consume the data.
  • Design failover, outage, and volume tests and judge the evidence, including recovery time and recovery point. The platform team executes destructive and chaos tests.
  • Run read-only checks in pre-production and production.
  • Integrate quality checks with CI/CD on GitHub Actions or an equivalent tool.
  • Use AWS and Grafana to observe pipeline health, including CloudWatch metrics, secrets metadata, and container platform state.
  • Check classification, masking, and retention against the stated data rules.
  • Track defects to closure in Jira and keep test cases and results in TestRail.
  • Provide quality evidence, risks, and readiness input for releases.
  • Use an AI coding assistant to analyze data and author SQL, Python, and quality checks, and review generated output before it is used.

  • Requirements: Senior-level data quality engineering experience in ETL testing and data validation.
  • Strong SQL and Snowflake data-modelling experience, including layered raw, history, silver, and gold models, reconciliation and drift queries, and slowly changing dimensions and merge behavior.
  • Experience with logical and physical data models and with assessing the downstream impact of schema and data-contract changes.
  • Experience validating PostgreSQL sources, including consistent snapshot counts and logical replication.
  • Hands-on Python automation for data comparison and reconciliation, using pytest or an equivalent framework.
  • Experience running and reading orchestrated pipelines in Airflow or an equivalent scheduler.
  • Experience validating data movement, transformation, completeness, history, freshness, and compliance from source systems into analytical datasets.
  • Experience diagnosing data quality issues and supporting remediation.
  • Experience integrating data quality checks with CI/CD using GitHub Actions or equivalent tools.
  • Experience with test management and defect tracking in TestRail and Jira.
  • Working knowledge of AWS and Grafana for pipeline observation, including CloudWatch, secrets management, and container platforms.
  • Experience designing resilience tests and judging evidence for failover, outage, volume, and recovery objectives.
  • Working knowledge of data classification, masking, and retention checks.
  • Experience producing data quality metrics, reports, and dashboards.
  • Experience with risk-based testing in Agile or Kanban teams.
  • Hands-on use of AI coding assistants (Cursor, GitHub Copilot, Claude Code, Codex, or equivalent) to author SQL, Python, and quality checks, with human review of generated output.
  • Ability to communicate clearly with engineering and business stakeholders.
  • Experience providing quality evidence, risks, and readiness input for releases.

  • Nice to have: Experience observing Snowflake Openflow or NiFi connector state well enough to recognize a stalled load.
  • Experience with custom cross-table data quality checks that publish results for dashboarding.
  • Familiarity with GenAI, LLMs, and agentic workflows for maintaining data quality checks.
  • Familiarity with RAG and MCP concepts for QE tooling and for generating test data.
  • Knowledge of AI-driven test optimization and coverage analytics (SeaLights or equivalent).
  • Experience with service virtualization using WireMock or an equivalent tool, so checks can run when a dependency is unavailable.
  • Experience specifying and reviewing dbt tests. Developers author those tests.

Skills Required

Postgresql AWS Test Design ETL Testing Data Validation SQL Data Quality Python GitHub GitHub Actions Jira Reconciliation

Learn the skills this job asks for

Free courses, interview questions and practice for each skill.

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
DataArt logo
DataArt

RecommendedJobs for You

DataArt logo

Senior Data QE Engineer, Hyderabad-based

DataArt

Bengaluru

DataArt logo

Senior Data QE Engineer, Hyderabad-based

DataArt

Bengaluru