Senior Consultant – Cloud Data Engineer – Azure

Key Responsibilities:

• Design, develop, and manage scalable, secure, and efficient data pipelines on Azure to process and transform large datasets from multiple sources.
• Implement and optimize Azure data services such as Azure Synapse Analytics, Azure Data Lake Storage (ADLS), Azure SQL Database, Azure Data Factory (ADF), Azure Databricks, and Azure Functions.
• Architect and maintain ETL/ELT processes to ensure efficient data ingestion, transformation, and storage.
• Architect and implement data models on Azure platforms to support efficient data storage and retrieval.
• Collaborate with data architects to design and implement data lakes, data warehouses, and data marts on Azure.
• Build and maintain data integration workflows using tools like Azure Data Factory (ADF) or Azure Synapse Pipelines.
• Utilize Azure DevOps for source control, build automation, release management, and Infrastructure as Code (IaC) using ARM templates, Terraform, or Bicep.
• Implement CI/CD pipelines within Azure DevOps to automate the build, test, and deployment of data pipelines and infrastructure.
• Leverage big data technologies and frameworks such as Azure Databricks (Apache Spark) for large-scale data processing and analytics.
• Implement real-time/streaming data solutions using Azure Event Hubs, Azure Stream Analytics, or Kafka on Azure.
• Implement data governance policies and security controls within Azure environments to ensure compliance with GDPR, HIPAA, PCI DSS, and organizational security requirements.
• Optimize data pipelines and workflows for performance, scalability, and cost-effectiveness, leveraging Azure Monitor, Cost Management, and Advisor.
• Monitor, troubleshoot, and optimize performance of Azure-based data pipelines and systems to ensure high availability and scalability.
• Work closely with cross-functional teams to translate business requirements into robust Azure data solutions.
• Conduct regular code reviews, provide feedback, and ensure adherence to best practices and standards.
• Keep up to date with the latest Azure cloud services and data technologies to drive innovation within the team.

Job Qualifications:

• Bachelor’s degree in computer science engineering, or a related field.
• 4-6 years of experience as a Cloud Data Engineer or similar role, with a strong focus on Azure cloud services.
• Proficiency in designing and building scalable data architectures using Azure data services such as ADLS, Synapse, Data Factory, Azure SQL, Azure Functions, Databricks.
• Strong experience with ETL/ELT frameworks such as ADF, Synapse Pipelines, or Databricks workflows.
• Expertise in SQL, Python, or other data engineering languages.
• Hands-on experience with data lakes, data warehouses, and data orchestration on Azure.
• Familiarity with CI/CD pipelines and IaC tools such as ARM templates, Terraform, or Bicep.
• Understanding of data governance, security, and compliance standards (encryption, Azure RBAC, Key Vault, VNETs).
• Experience in data modeling, performance tuning, and optimization for Azure-based solutions.
• Azure certifications such as Azure Data Engineer Associate, Azure Solutions Architect Expert, or Azure Developer Associate are preferred.
• Experience with Azure Databricks, Apache Spark, or ML integrations is desirable.
• Familiarity with containerization and DevOps practices using Docker, Kubernetes (AKS).
• Excellent communication and collaboration skills to work effectively with stakeholders.
• Strong problem-solving and troubleshooting abilities in Azure ecosystems.
• Ability to thrive in a fast-paced, agile environment and manage multiple priorities.