Technology Lead-Java Spark

Key Responsibilities : Big Data Development : Design and implement big data solutions using Spark and Scala . Develop and manage data pipelines and workflows within the big data ecosystem . Data Processing & Analysis : Write optimized SQL queries for data processing and analysis. Apply data transformations and business rules using Spark , Scala , and Hadoop . Hands-on Coding : Develop and maintain code using Java , Python , and optionally Go Lang , ensuring the efficient processing of large datasets and integrating with other platforms and services. Data Optimization : Tune and optimize Spark jobs for better performance, reducing computation time and enhancing resource utilization in a big data environment. Agile Development : Collaborate with cross-functional teams using the Agile methodology to ensure timely delivery of high-quality data solutions. Collaboration & Reporting : Work with business analysts and other stakeholders to gather requirements, provide technical input, and ensure alignment with business needs. Regularly update stakeholders on the progress of ongoing projects. Testing & Troubleshooting : Conduct unit tests, troubleshoot issues, and ensure the overall quality of the data pipelines and solutions. Requirements : Experience : 7+ years of hands-on experience as a Big Data Engineer , specifically with Spark , Scala , Python , and Hadoop . Strong Coding Skills : Proven expertise in coding with Java , Python , and Scala for building scalable and efficient data processing pipelines. SQL Expertise : Strong knowledge of SQL for data querying and transformations. Big Data Ecosystem Knowledge : Deep understanding of the big data ecosystem , including Hadoop and related tools like Hive , Pig , and HBase . Analytical & Problem-Solving : Strong analytical and troubleshooting skills, with the ability to find solutions to complex big data challenges. Agile Development : Familiarity with Agile development methodology and experience working in Agile teams. Nice to Have : Experience with Go Lang and familiarity with cloud-based big data platforms (AWS, GCP, Azure). Key Technologies : Big Data Tools : Spark , Hadoop , Hive , Pig . Languages : Scala , Java , Python , SQL , Go Lang . Methodology : Agile . Operating Systems : Unix/Linux for executing commands and scripts. Work Environment : Work Mode : [Specify WFO/WFH/Hybrid]. Location : [Specify Location]. Vendor Rate : [Insert vendor rate in local currency].