Job
Description
:
Job TitleQA Engineer
LocationPune, India
Corporate TitleAssistant Vice President
Role Description
Technology serves as the foundation of our entire organization. Our Technology, Data, and Innovation (TDI) strategy aims to enhance engineering capabilities, implement an agile delivery framework, and modernize the bank's IT infrastructure.We are committed to investing in and developing a team of forward-thinking technology professionals, offering them the training, autonomy, and opportunities necessary to engage in groundbreaking work. As a QA lead Engineer, you will oversee the comprehensive delivery of engineering solutions to meet business objectives.You will have deep expertise in Hadoop ecosystem administration, scripting, and managing complex, large-scale migration projects. We are seeking a highly skilled Senior Test/QA Engineer to lead the quality assurance and testing efforts for a large-scale migration from Cloudera Distribution Hadoop (CDH) or Cloudera Data Platform (CDP) to Apache Hadoop.The ideal candidate will have extensive experience in end-to-end testing, automation, and developing comprehensive test suites for the Hadoop ecosystem.This role requires hands-on expertise in validating complex migrations, ensuring data integrity, performance, and system reliability while collaborating with cross-functional teams.You will have the opportunity to collaborate closely with clients while being part of a larger, creative, and innovative team dedicated to making a significant impact.In RiskFinder, you will join a local team, collaborating with other teams both in your area and in different geographical locations.You will have the chance to architect, design, and implement an open-source big data platform, enabling clients to produce, access, and analyze extensive datasets through our custom components and applications.
What well offer you
100% reimbursement under childcare assistance benefit (gender neutral)Sponsorship for Industry relevant certifications and educationAccident and Term life Insurance
Your key responsibilities
Develop robust architectures and designs for big data platform and applications within the Apache Hadoop ecosystemImplement and deploy big data platform and solutions on-premises and in hybrid cloud environments.Develop and execute migration scripts, workflows and automation tools to streamline the transition process.Read, understand, and modify open-source code to implement bug fixes and perform upgrades.Security ArchitectureEnsure all solutions adhere to security best practices and compliance requirements.Design, develop, and execute E2E test plans to validate the migration of data, applications, and workflows from Cloudera to Apache Hadoop.Test Hadoop ecosystem components (HDFS, YARN, Hive, Spark, Kafka, Oozie, etc.) to ensure functionality, performance, and compatibility post-migration.Validate data integrity, consistency, and accuracy across source and target systems during migration.Develop and maintain automated test scripts using tools like Python, Ansible, Shell, Java, or Scala to validate Hadoop cluster configurations, data pipelines, and workflows.Implement automation frameworks for regression, performance, and scalability testing of the Apache Hadoop environment.Integrate automated tests into CI/CD pipelines using tools like Jenkins, GitLab CI, or similar.Build comprehensive test suites to cover functional, non-functional, security, and performance aspects of the Hadoop ecosystem.Create test cases for edge cases, failure scenarios, and high-availability configurations (e.g., Kerberos, Ranger, failover)
Your skills and experience
Proven experience in architecting, designing, building, and deploying big data platforms and applications using the Apache Hadoop ecosystem in hybrid cloud and private cloud scenarios.Experience with hybrid cloud big data platform designs and deployments, especially in AWS, Azure, or GCP.QA/testing, with at least 10 years focused on big data platforms and the Hadoop ecosystem. Proven experience testing large-scale Hadoop migrations (e.g., Cloudera to Apache Hadoop or similar). Hands-on experience validating petabyte-scale data systems in production environmentsExperience in large-scale data platform builds and application migrations. Expert knowledge of Apache Hadoop ecosystem and associated Apache projects (e.g., HDFS, Hive, HBase, Spark, Kafka, Yarn etc.).Strong programming/scripting skills in Python, Java, Shell, Ansible, Scala, or Bash for test automation.Proficiency in test automation frameworks (e.g., Selenium, TestNG, JUnit, or custom frameworks for big data). Experience with performance testing tools (e.g., JMeter, Gatling) and monitoring tools (e.g., Grafana, Prometheus). Familiarity with security testing (e.g., Kerberos, Ranger, LDAP) and cluster management tools (e.g., Ambari).Knowledge of CI/CD pipelines and tools like Jenkins, Git, or GitLab CI.Experience with version upgrades of technology stacks.Experience testing cloud-based Hadoop deployments (e.g., AWS EMR, Azure HDInsight, GCP Dataproc). Familiarity with containerized environments (e.g., Docker, Kubernetes) for testing Hadoop deployments.Excellent problem-solving and analytical skills.Strong communication and collaboration skills to work effectively with global teams.Ability to work independently and take initiative.Contributions to open-source projects.
How well support you
