Big Data Developer

Job Title: Big Data Developer Location: Pune, India Experience: 4 to 5 Years Notice Period: Immediate to 15 Days Job Description: We are looking for a highly skilled and motivated Big Data Developer to join our team in Pune. The ideal candidate will have hands-on experience in building scalable data pipelines, optimizing big data jobs, and working with modern data tools and frameworks. This role involves end-to-end ownership of data workflows, performance tuning, and working across a range of technologies including Spark, Airflow, and DBT. Key Responsibilities: Develop, optimize, and maintain complex SQL queries using joins, aggregations, group by, and window functions . Build and manage scalable Apache Spark applications, focusing on performance optimization techniques like task tuning, partitionBy, broadcast joins , adaptive query execution (AQE) , and data pruning . Design and schedule data workflows using Apache Airflow , handling task dependencies, scheduling, and error/debug handling. Implement data transformation pipelines using DBT , following modular data modeling best practices: staging , intermediate , and mart layers . Perform SQL-based testing to ensure data quality and pipeline reliability. Apply cost and performance optimization techniques across data pipelines, including repartitioning , caching , and join strategy tuning . Use standard tooling such as Git for version control and handle common tasks like resolving merge conflicts . Work within Linux environments and use basic scripting knowledge in Python or Java to support automation and integration tasks. Use Docker for containerized development and deployment workflows. Required Skills & Qualifications: 45 years of experience in Big Data Development or Data Engineering Proficiency in advanced SQL and big data query optimization Strong hands-on experience with Apache Spark and distributed data processing Practical knowledge of Airflow for workflow orchestration Experience with DBT and modular data modeling Sound understanding of performance tuning techniques for big data environments Familiarity with Git, Linux, Python/Java, and Docker