Big Data QA

Role Overview: As a Big Data QA Engineer with 4+ years of experience in batch testing and data validation, your primary responsibility will be to ensure the quality and reliability of large-scale data pipelines. You will be involved in designing and executing automated and manual tests, validating data transformations, and maintaining Python-based test automation frameworks. Key Responsibilities: - Design and execute automated and manual tests to validate data transformations - Ensure the quality and reliability of large-scale data pipelines - Maintain Python-based test automation frameworks - Collaborate closely with data engineers in an Agile environment - Analyze logs and troubleshoot batch job failures Qualifications Required: - Strong Python skills - Hands-on experience with AWS (S3, EMR, Athena, Lambda) - Expertise in big data formats such as Parquet, JSON, and CSV using Pandas and PySpark - Knowledge of SQL - Experience in CI/CD - Familiarity with Agile methodology Additional Details: N/A Role Overview: As a Big Data QA Engineer with 4+ years of experience in batch testing and data validation, your primary responsibility will be to ensure the quality and reliability of large-scale data pipelines. You will be involved in designing and executing automated and manual tests, validating data transformations, and maintaining Python-based test automation frameworks. Key Responsibilities: - Design and execute automated and manual tests to validate data transformations - Ensure the quality and reliability of large-scale data pipelines - Maintain Python-based test automation frameworks - Collaborate closely with data engineers in an Agile environment - Analyze logs and troubleshoot batch job failures Qualifications Required: - Strong Python skills - Hands-on experience with AWS (S3, EMR, Athena, Lambda) - Expertise in big data formats such as Parquet, JSON, and CSV using Pandas and PySpark - Knowledge of SQL - Experience in CI/CD - Familiarity with Agile methodology Additional Details: N/A