Lead Apache Hadoop Engineer

Job Description

: Job TitleLead Apache Hadoop Engineer LocationPune, India Corporate TitleVice President Role Description Technology serves as the foundation of our entire organization. Our Technology, Data, and Innovation (TDI) strategy aims to enhance engineering capabilities, implement an agile delivery framework, and modernize the bank's IT infrastructure. We are committed to investing in and developing a team of forward-thinking technology professionals, offering them the training, autonomy, and opportunities necessary to engage in groundbreaking work. As a Lead Engineer, you will oversee the comprehensive delivery of engineering solutions to meet business objectives. You will have deep expertise in Hadoop ecosystem administration, scripting, and managing complex, large-scale migration projects. You will provide engineering leadership across various teams, mentor junior engineers, and promote ongoing enhancements in delivery methodologies. You will have the opportunity to collaborate closely with clients while being part of a larger, creative, and innovative team dedicated to making a significant impact. In RiskFinder, you will join a local team, collaborating with other teams both in your area and in different geographical locations. You will have the chance to architect, design, and implement an open-source big data platform, enabling clients to produce, access, and analyze extensive datasets through our custom components and applications. What well offer you 100% reimbursement under childcare assistance benefit (gender neutral) Sponsorship for Industry relevant certifications and education Accident and Term life Insurance Your key responsibilities Develop robust architectures and designs for big data platform and applications within the Apache Hadoop ecosystem Implement and deploy big data platform and solutions on-premises and in hybrid cloud environments. Develop and execute migration scripts, workflows and automation tools to streamline the transition process. Read, understand, and modify open-source code to implement bug fixes and perform upgrades. Security ArchitectureEnsure all solutions adhere to security best practices and compliance requirements. Mitigate risks by identifying potential challenges (e.g., data loss, compatibility issues) and implementing contingency plans. Work directly with the Lead Architect and support cross-functional teams globally. Strategy DevelopmentContribute to defining the end-to-end platform engineering and application migration strategy. Your skills and experience Proven experience in architecting, designing, building, and deploying big data platforms and applications using the Apache Hadoop ecosystem in hybrid cloud and private cloud scenarios. Experience with hybrid cloud big data platform designs and deployments, especially in AWS, Azure, or GCP. Experience in large-scale data platform builds and application migrations. Expert knowledge of Apache Hadoop ecosystem and associated Apache projects (e.g., HDFS, Hive, HBase, Spark, Kafka, Yarn etc.). Proficiency in Kubernetes for container orchestration. Ability to read and modify open-source code. Experience with version upgrades of technology stacks. Excellent problem-solving and analytical skills. Strong communication and collaboration skills to work effectively with global teams. Ability to work independently and take initiative. Contributions to open-source projects. How well support you