Python Pyspark SDET

Role Overview: As a skilled Software Development Engineer in Test (SDET) specializing in PySpark and Python for data engineering and validation on Azure Cloud, you will be instrumental in constructing automation frameworks, creating and implementing test cases to guarantee the quality and dependability of extensive data solutions. Key Responsibilities: - Develop, maintain, and execute automated test scripts utilizing Python and PySpark for data pipelines and engineering workflows. - Validate data transformations, data quality, and pipeline performance within Azure Cloud setups (particularly leveraging Databricks). - Collaborate closely with data engineers and developers to comprehend data workflows and test scenarios. - Construct and enhance test frameworks that seamlessly integrate with CI/CD pipelines. - Conduct comprehensive regression, performance, and API testing for data applications. - Identify, report, and monitor defects, collaborating effectively with engineering teams to ensure resolution. Skills & Qualifications: - Proficient programming skills in Python and PySpark. - Hands-on experience in automated testing for data pipelines and big data platforms. - Working knowledge of Azure Cloud services, specifically Azure Data Factory and Databricks. - Familiarity with CI/CD tools and test automation frameworks. - Understanding of data validation, ETL testing, and optimization of SQL queries. - Excellent problem-solving abilities and communication skills. Preferred: - Practical experience with Databricks and related Azure data technologies. - Background in software development and test engineering for data-intensive applications.,