Big Data QA Engineer

As a Senior QA Data Engineer at the company, you will play a crucial role in ensuring the quality and reliability of data solutions. Your primary responsibilities will include: - Building and maintaining automated test scripts for data workflows and ETL pipelines using Python and PySpark - Performing data validations using SQL and working with NoSQL databases for comprehensive test coverage - Collaborating with DevOps teams to validate and enhance CI/CD pipelines, preferably using Jenkins - Supporting testing of data integrations across various AWS services like Glue, Lambda, S3, RDS, and Iceberg - Designing data validation frameworks to uphold data quality and integrity throughout the pipeline - Monitoring data pipeline performance and proactively flagging issues or anomalies - Contributing to the documentation of test plans, strategies, and best practices To excel in this role, you should possess: - Strong hands-on experience with Python, PySpark, and SQL - Background in data pipeline/ETL testing - Experience with data testing automation tools and frameworks - Familiarity with working on CI/CD tools like Jenkins - Understanding of AWS cloud services, or a willingness to quickly ramp up - Experience in validating both relational and NoSQL data stores Additionally, exposure to tools like Airflow or Spark, as well as an AWS certification (e.g. Developer Associate) would be beneficial.,