ETL Automation Tester

Job Title: ETL Automation Test Engineer

Job Summary:

Key Responsibilities:

• Design, develop, and execute test plans for data pipelines using Databricks, PySpark, and Delta Lake.
• Validate data accuracy, completeness, and integrity at every stage of the pipeline.
• Develop and maintain automated test frameworks and scripts using Python and SQL.
• Collaborate with data engineers and analysts to identify testing requirements and ensure data quality.
• Troubleshoot and debug issues with data pipelines and testing frameworks.
• Stay up-to-date with new technologies and features in Databricks, PySpark, and Delta Lake.
  

Required Skills:

• Strong SQL skills (window functions, joins, aggregates).
• Proficiency in PySpark (DataFrame APIs, UDFs, Structured Streaming).
• Delta Lake fundamentals: ACID, time travel, schema enforcement/evolution, Change Data Capture (CDC).
• Databricks Lakehouse concepts: medallion architecture (Bronze/Silver/Gold), Photon, serverless vs classic clusters.
• Hands-on experience with Databricks platform, including notebooks, Jobs, and DLT pipelines.
• Extensive experience with ETL (Extract, Transform, Load) and ELT processes.
• Test automation experience with Python and SQL.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills.
  

Prefer to Have:

• Experience with data warehousing and big data technologies.
• Familiarity with Agile development methodologies.
• Knowledge of data governance and data quality frameworks.
  

What We Offer:

• Competitive salary and benefits package.
• Opportunity to work with cutting-edge technologies and innovative teams.
• Collaborative and dynamic work environment.
• Professional growth and development opportunities.