Spark, Java, ADF, ADB

Job Description

The candidate will leverage their strong expertise in designing and developing data analytics and data warehouse solutions. This includes writing efficient Java and Python scripts on Spark for processing large-scale data. The role involves working with complex data distribution architectures, integrating service-oriented components, and implementing Data Lake solutions using Azure Databricks. A key responsibility is translating conceptual solutions into executable packages using the Azure platform, with a strong emphasis on Azure Storage Gen2 and other Azure analytics services. Key Responsibilities: Design and develop scalable data analytics and data warehouse solutions using Spark on Azure Databricks Write Spark code primarily in Java, with additional scripting in Scala or Python as needed Develop and maintain ETL pipelines using Azure Data Factory (ADF) Work extensively with Azure Storage Gen2 and other Azure data services including Azure SQL, Function Apps, and Logic Apps Integrate multi-format data into data lake architectures ensuring performance and scalability Translate business requirements and working solutions into implementable working packages on Azure platform Manage multiple projects concurrently in a fast-paced environment, ensuring quality and attention to detail Collaborate with cross-functional teams to deliver robust and optimized solutions Continuously learn and adopt new tools and technologies to enhance the data engineering capabilities Mandatory Skills: 8+ years of hands-on experience with Spark (Java preferred, also Scala/Python) Extensive experience with Azure Databricks and Azure Data Factory Proficiency in Java programming and Spark coding Strong understanding of Azure Storage Gen2 and Azure Analytics services Experience writing T-SQL queries and familiarity with Azure SQL Knowledge of Azure function apps and logic apps Demonstrated ability to handle pressure and deliver high-quality results Preferred Skills: Experience with service-oriented architecture and large-scale data integration Strong problem-solving, communication skills, and ability to work collaboratively Agile and organized approach to managing multiple projects