Job
Description
Job Title: Azure Data Engineer - Manager (5-9 Years of Experience) Location: [Bangalore, Coimbatore, Pune] Employment Type: Full-Time Job Description: We are looking for an experienced Azure Data Engineer with 9-13 years of experience to take on a strong Technical and Managerial role within our organization. The ideal candidate will have a strong technical background in Databricks , Azure Data Services , and data engineering, coupled with proven leadership and team management skills. This role requires a balance of hands-on technical expertise and the ability to lead, mentor, and manage a team of data engineers to deliver high-quality data solutions. Key Responsibilities: Technical Responsibilities: Data Pipeline Development: Design, develop, and optimize scalable data pipelines using Databricks , Azure Data Factory , and other Azure data services. Implement advanced ETL/ELT processes to handle large volumes of data from diverse sources. Ensure data pipelines are efficient, reliable, and scalable to meet business needs. Data Processing and Analytics: Write and optimize complex SQL queries for data extraction, transformation, and analysis. Use PySpark for large-scale data processing, transformation, and analytics. Implement data partitioning, indexing, and caching strategies for optimal performance. Data Integration and Governance: Integrate data from multiple sources, including structured, semi-structured, and unstructured data. Implement data governance practices to ensure data quality, consistency, and security. Monitor and troubleshoot data pipelines to ensure data accuracy and availability. Architecture and Design: Define and implement data architecture best practices, including data lake and data warehouse design. Collaborate with cross-functional teams to design and deliver end-to-end data solutions. Evaluate and recommend new tools and technologies to enhance data engineering capabilities. Managerial Responsibilities: Team Leadership: Lead, mentor, and manage a team of data engineers, ensuring high performance and professional growth. Assign tasks, set priorities, and ensure timely delivery of projects. Conduct regular team meetings, performance reviews, and one-on-one sessions. Project Management: Oversee the end-to-end delivery of data engineering projects, ensuring alignment with business goals. Collaborate with stakeholders to define project scope, timelines, and deliverables. Manage project risks, issues, and dependencies to ensure successful project execution. Stakeholder Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand requirements and deliver solutions. Act as a bridge between technical teams and business stakeholders, ensuring clear communication and alignment. Process Improvement: Identify opportunities for process improvement and implement best practices in data engineering. Drive innovation and continuous improvement within the team. Must-Have Skills: Databricks: Extensive hands-on experience with Databricks for data processing, analytics, and machine learning. Azure Data Services: Proficiency in Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Azure SQL Database. SQL: Expertise in writing and optimizing complex SQL queries. PySpark: Advanced knowledge of PySpark for large-scale data processing and transformation. ETL/ELT: Strong understanding of ETL/ELT processes and tools. Data Modeling: Deep knowledge of data modeling techniques and best practices. Leadership: Proven experience in leading and managing teams of data engineers. Project Management: Strong project management skills with the ability to manage multiple projects simultaneously. Good-to-Have Skills: Experience with Azure DevOps for CI/CD pipelines. Knowledge of Delta Lake for building reliable data lakes. Familiarity with streaming data technologies like Azure Event Hubs or Kafka. Qualifications: Bachelors or Masters degree in Computer Science, Information Technology, or a related field. 9-13 years of experience in data engineering, with a focus on Azure and Databricks. Proven experience in a leadership or managerial role, leading teams of 5+ members. Relevant certifications such as Microsoft Certified: Azure Data Engineer Associate or Databricks Certified Associate Developer are a plus. Soft Skills: Strong leadership and team management skills. Excellent communication and interpersonal skills. Ability to work in a fast-paced, dynamic environment. Strong problem-solving and analytical skills. Self-motivated with a strong sense of ownership and accountability.
