Big Data Engineer

As a Big Data QA Engineer, you will be responsible for validating and testing large-scale data processing systems. Your main role will involve designing, executing, and maintaining automated and manual test suites focused on batch workflows, data integrity, and transformation validation. Working closely with data engineers, developers, and product teams, you will ensure high-quality, reliable, and scalable data pipelines. - Design, develop, and execute automated and manual batch test cases for big data pipelines. - Validate data integrity, schema accuracy, and transformation logic across multiple stages. - Monitor and troubleshoot batch job failures, performance issues, and data mismatches. - Maintain and enhance a Python-based test automation framework. - Reproduce data issues, analyze system performance, and collaborate with developers to implement fixes. - Analyze logs and metrics to determine the root cause of failures. - Work closely with cross-functional teams to improve testing processes and ensure high-quality deliveries. Qualifications Required: - Minimum 4 years of experience in Big Data testing, batch processing, and data validation. - Strong proficiency in Python for automation and scripting. - Hands-on experience with AWS services such as S3, Lambda, EMR, and Athena. - Experience working with big data formats: JSON, Parquet, CSV. - Practical knowledge of Pandas and PySpark for data analysis and validation. - Experience with version control tools (Git) and CI/CD pipelines (Jenkins, GitHub Actions). - Strong understanding of testing methodologies: regression, integration, system testing. - Ability to analyze logs/metrics and solve data quality issues. - Experience working in Agile/Scrum environments. - Excellent communication and collaboration skills. This role is ideal for individuals highly skilled in Python automation, AWS big data services, and batch data validation, with a passion for ensuring the accuracy and reliability of data pipelines.,