Azure Data Engineer

Job Description

Azure Databricks Overview We are seeking a highly skilled and motivated Senior Technical Lead to join our team and drive the development and deployment of Azure Databricksbased solutions This role will require expertise in Python programming Databricks administration CICD pipelines for Databricks artifacts and handson experience with Delta Live Tables Auto Loader Unity Catalog and Databricks Asset Bundles The ideal candidate will have a strong technical background proven experience in leading teams and a passion for building highquality scalable solutions Key Responsibilities Python Programming Lead the development of robust scalable and efficient Python code for data processing transformation and analysis within the Databricks environment Ensure adherence to coding standards quality and performance best practices Azure Databricks Administration Oversee Databricks platform configuration resource management cluster optimization and monitoring to ensure high availability and performance Implement best practices for managing Databricks workspaces libraries and notebooks CICD for Databricks Artifacts Design and implement CICD pipelines for automating the deployment of Databricks artifacts such as notebooks libraries jobs and Delta tables Use tools such as Azure DevOps GitHub Actions or Jenkins to streamline the deployment process Delta Live Tables Design and manage Delta Live Tables pipelines for realtime data processing ensuring data quality reliability and optimal performance in the processing layers Auto Loader Implement and optimize Auto Loader for scalable faulttolerant ingestion of data from cloud storage sources into Databricks Unity Catalog Manage and configure Unity Catalog for centralized governance of data across workspaces ensuring proper access controls data lineage and compliance Databricks Asset Bundles Leverage Databricks Asset Bundles to manage and share reusable components including notebooks libraries and ML models across teams or projects Team Leadership Mentor and guide junior team members fostering a collaborative environment that encourages knowledge sharing innovation and personal growth Provide technical leadership and support to ensure the success of the teams initiatives Collaboration and Communication Work closely with stakeholders including data engineers data scientists and business analysts to understand requirements and deliver endtoend solutions Communicate technical challenges and solutions clearly to both technical and nontechnical stakeholders Required Qualifications Experience in Python Programming o Minimum of 5 years of experience in Python programming with a focus on data engineering and data science workflows o Expertise in designing and building scalable Pythonbased ETL pipelines using libraries such as Pandas NumPy PySpark and requests o Proficiency in debugging optimizing and maintaining Python code to ensure high performance and reliability in a distributed environment Extensive Experience with Azure Databricks o Minimum of 3 years of handson experience working with Azure Databricks in a production environment o Strong knowledge of Databricks clusters workspaces and the Databricks runtime o Ability to optimize and troubleshoot Sparkbased jobs and notebooks ensuring performance and costefficiency in cloud environments CICD for Databricks Artifacts o Proven experience in setting up and managing Continuous IntegrationContinuous Deployment CICD pipelines for Databricks artifacts notebooks libraries and jobs o Familiarity with version control systems Git and experience with tools such as Azure DevOps GitHub Actions or Jenkins for automated deployment o Experience with integrating Databricks with external version control repositories eg GitHub Azure Repos for collaboration and artifact management Delta Live Tables Experience: 5 to 12 Years Location: Bangalore, Chennai, Noida, Kolkata Show more Show less