Lead Big Data Engineer - Python & Spark

Job Description

Job Title: Lead Big Data Engineer - Python & Spark Location: Gurgaon, India Experience: 7+ Years Employment Type: Full-Time | Onsite Department: Data Engineering About Hubnex Labs: Hubnex Labs is a forward-looking IT consulting and software services company, building next-generation data platforms, AI systems, and enterprise-grade applications. We re looking for a Lead Big Data Engineer to drive the development and deployment of high-performance data processing solutions. Role Overview: As a Lead Big Data Engineer , you will be responsible for architecting and implementing scalable big data solutions using Spark, Python, Hive, and related technologies. You will mentor a team of developers and work closely with cross-functional stakeholders to ensure on-time, error-free software delivery. Key Responsibilities: Lead and mentor a team of Python and big data developers to deliver robust data-driven applications Design, develop, and maintain scalable data processing pipelines using Spark (Scala/PySpark), Hive, and Hadoop ecosystems Write efficient, reusable, and well-documented code in Python, SQL, and shell scripting Optimize Spark applications for performance and scalability; tune existing Hadoop-based systems Collaborate with QA, DevOps, and business teams to ensure high-quality software delivery Perform code reviews , enforce coding standards, and contribute to overall architectural decisions Actively participate in daily Scrum meetings , sprint planning, retrospectives, and release cycles Troubleshoot complex issues across the full data pipeline Required Qualifications: 7+ years of hands-on experience in software development, with a strong background in big data frameworks Deep expertise in Hadoop ecosystem including HDFS, Hive, HQL, Spark (Scala/PySpark), Sqoop 4+ years of programming experience using Python , SQL , and Unix shell scripting Proven experience in leading development teams and delivering enterprise-level solutions Strong grasp of Spark architecture , performance tuning, and data frame APIs Excellent understanding of database concepts , data structures , and distributed systems Bachelors degree in Computer Science , Engineering , or a related field Exceptional communication, problem-solving, and leadership skills Preferred Skills: Experience with CI/CD pipelines for data projects Familiarity with cloud-based data platforms (AWS EMR, GCP DataProc, or Azure HDInsight) Working knowledge of Kafka , Airflow , or other data orchestration tools Why Join Hubnex Labs? Work on mission-critical big data solutions that impact businesses globally Be part of an innovative, agile, and supportive team Leadership opportunities and exposure to latest technologies in AI and cloud computing Competitive salary, performance incentives, and professional growth support