Python Pyspark SDET

Job Title:

Location:

Experience Required:

Role Overview:

Seeking a skilled Software Development Engineer in Test (SDET) with expertise in PySpark and Python for data engineering and data pipeline validation on Azure Cloud. You will play a critical role in building automation frameworks, developing and executing test cases to ensure the quality and reliability of large-scale data solutions.

Key Responsibilities:

• Develop, maintain, and execute automated test scripts for data pipelines and engineering workflows using Python and PySpark.
• Validate data transformations, data quality, and pipeline performance on Azure Cloud environments (preferably using Databricks).
• Collaborate with data engineers and developers to understand data workflows and test scenarios.
• Build and optimize test frameworks that integrate with CI/CD pipelines.
• Perform thorough regression, performance, and API testing for data applications.
• Report and track defects, and work closely with engineering teams for resolution.

Skills & Qualifications:

• Strong programming skills in Python and PySpark.
• Experience in automated testing for data pipelines and big data platforms.
• Working knowledge of Azure Cloud services, including Azure Data Factory and Databricks.
• Familiarity with CI/CD tools and test automation frameworks.
• Understanding of data validation, ETL testing, and SQL query optimization.
• Excellent problem-solving and communication skills.

Preferred:

• Hands-on experience with Databricks and related Azure data technologies.
• Background in software development and test engineering for data-intensive applications.