Job
Description
Responsibilities: Design, develop, and maintain scalable ETL pipelines using Azure Databricks, Spark, and other Azure services. Optimize and troubleshoot data pipelines for performance, scalability, and reliability. Implement data governance policies to ensure accuracy, consistency, and security. Conduct data analysis, profiling, and validation to maintain high data quality. Integrate Azure Databricks with Azure Data Lake, Azure SQL Database, and Azure Synapse Analytics. Develop and manage ETL pipelines using Azure Data Factory and PySpark. Build reusable frameworks to automate and streamline data processing tasks. Create and maintain technical documentation for data pipelines and processes. Collaborate with cross-functional teams to align solutions with business and technical objectives. Identify opportunities to expand data coverage and enhance analytics capabilities, including search and visualization tools. Monitor and optimize system performance, proactively addressing issues and stakeholder requests. Provide insights on data pipeline performance and recommend optimizations. Stay updated with emerging Azure technologies and best practices to enhance data engineering capabilities. Train and onboard team members and stakeholders on the data platforms capabilities. Skills and Experience: 3+ years of hands-on experience with Azure Databricks and data engineering in cloud environments. Extensive experience with Azure Data Factory, Azure SQL Database, Azure Synapse Analytics, Data Lake, Cosmos DB, EventHub, ServiceBus, Topics, Blob Storage, Azure Functions, AKS, and Key Vault. Strong understanding of data warehousing concepts (star schema, snowflake schema, data lakes) and data modeling for scalable architecture. Proficiency in Python and SQL (R is a plus), with hands-on experience in writing complex SQL queries. Hands-on experience with distributed computing frameworks such as PySpark (Spark experience required; Hadoop or Dask is a plus). Working knowledge of Azure Pipelines, GitHub Actions & Workflows, and version control (Git). Experience with data integration, ETL workflows, and pipeline automation using Azure Data Factory and PySpark. Familiarity with ML workflows and integrating ML models into Databricks pipelines. Exposure to Docker and Kubernetes (AKS is a plus) for scalable deployments. Strong analytical skills to troubleshoot complex data issues, with excellent communication and stakeholder collaboration. Experience with Agile methodologies for project tracking (JIRA) and collaboration. Azure certifications such as Azure Data Engineer, Azure Solutions Architect, or Azure Developer ID are good to have Excellent communication and interpersonal skills to collaborate with cross-functional teams and stakeholders Education: Bachelors or Masters from Premier Institutes preferred. Experience 4-12 years
