Azure Databricks Lead

As an Azure Databricks Lead with over 6 years of experience, located in Chennai, your primary role will revolve around spearheading data engineering initiatives and offering technical leadership in Azure-based data platform projects. Your expertise in designing and creating scalable big data pipelines using Azure Databricks, Spark, and related Azure services will be crucial for the success of these initiatives. Your key responsibilities will include leading the design, development, and deployment of scalable data pipelines utilizing Azure Databricks and Spark. Collaboration with stakeholders to comprehend business requirements and transforming them into technical specifications will be essential. You will architect comprehensive data solutions by leveraging Azure services like Data Lake, Data Factory, Synapse, Key Vault, and Event Hub. Optimizing existing data pipelines to ensure adherence to security, performance, and cost best practices will be part of your routine tasks. Guiding and mentoring a team of data engineers and developers will also be a significant aspect of your role. Implementation of data governance, quality checks, and CI/CD practices within the Databricks environment will be necessary for maintaining data integrity and efficiency. Your ability to work closely with cross-functional teams, including data scientists, analysts, and BI developers, will be pivotal in ensuring seamless collaboration and project success.,