Data Bricks Developer

Responsibilities:

• Design, develop, and maintain data processing pipelines using Databricks on the Azure platform.
• Collaborate with data engineers, data scientists, and other stakeholders to optimize workflows and ensure data quality.
• Implement ETL processes to extract, transform, and load data from various sources.
• Work with Spark SQL and Databricks notebooks to analyze and transform data.
• Ensure efficient data flow across diverse systems.
• Provide technical guidance to teams and anchor proof-of-concept projects.
• Responsible to assemble large, complex sets of data that meet non-functional and functional business requirements.
• Responsible to identify, design and implement internal process improvements including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes.
• Building required infrastructure for optimal extraction, transformation and loading of data from various data sources using Azure, Databricks and SQL technologies
• Working with stakeholders including data, design, product and executive teams and assisting them with data-related technical issues
• Understanding of Cognos DB and Synapse.

Requirements:

• Bachelors degree in computer science, Engineering, or related field.
• 3 to 5 years of hands-on experience as a Databricks
• Strong proficiency in Apache Spark and Databricks.
• Strong knowledge on PySpark
• Experience with Scala and/or Python programming languages.
• Solid understanding of data warehousing concepts and ETL processes.
• Familiarity with cloud platforms - Azure
• Version control tools knowledge (e.g., Git).