Data Engineer (Azure Expertise)

We are seeking a highly skilled Data Engineer with a strong background in Azure technologies and expertise in data pipeline development, transformation, and analytics. The ideal candidate will have hands-on experience with Azure Data Factory (ADF), Azure Functions, Synapse Analytics, SQL Server, Python, and PySpark, along with a solid understanding of CI/CD pipelines and Power BI. Your role will involve designing, building, and maintaining scalable and reliable data solutions that empower the organization to make data driven decisions.

Key Responsibilities:

Design, Build, and Maintain Data Pipelines:

• Develop and optimize end-to-end ETL/ELT pipelines using Azure Data Factory (ADF).
• Integrate, transform, and load data from diverse sources into Azure Synapse Analytics or SQL-based data stores.
• Implement data workflows and automation using Azure Functions.

Database Development and Optimization:

• Design and manage data models and schemas in SQL Server and Synapse Analytics.
• Write optimized SQL queries and stored procedures for efficient data retrieval and transformation.
• Ensure database integrity, performance tuning, and scalability.

Big Data Processing:

• Use PySpark to process and analyze large-scale data in a distributed environment.
• Implement and manage data lakes and data lake houses for batch and real-time processing.

Data Governance and Security:

• Implement data governance policies, including data quality, lineage, and compliance.
• Secure data pipelines using role-based access control and encryption standards.

CI/CD and Automation:

• Build and manage CI/CD pipelines for data integration and deployment using tools like Azure DevOps, Git, or GitHub Actions.
• Automate testing, version control, and deployment of data engineering solutions.

Data Visualization and Reporting:

• Collaborate with business stakeholders to design and deliver interactive dashboards using Power BI.
• Enable data exploration and self-service reporting through semantic layers.

Collaboration and Communication:

• Work closely with cross-functional teams including analysts, and business users to gather requirements and implement data solutions.
• Document workflows, data models, and architecture to ensure knowledge sharing across teams.

Required Skills and Qualifications:Azure Expertise:

• Hands-on experience with Azure Data Factory (ADF) for building ETL/ELT workflows.
• Proficiency with Azure Synapse Analytics for data warehousing and analytics.
• Strong knowledge of Azure Functions for building event-driven automation.

Programming:

• Proficiency in Python, with experience in libraries such as Pandas, NumPy, and PySpark.
• Experience with distributed data processing frameworks like Apache Spark.

Database Management:

• Expertise in SQL Server.
• Knowledge of designing and implementing data lakes and lake houses.

CI/CD:

• Experience building and deploying pipelines with Azure DevOps, Jenkins, or similar tools.

Data Visualization:

• Proficiency in Power BI for creating dashboards and reports.
• Experience with DAX and Power Query for advanced analytics.

Data Integration and APIs:

• Experience with REST and SOAP APIs for data integration.
• Familiarity with messaging systems like Azure Event Hubs or Azure Service Bus.

General Skills:

• Strong problem-solving and analytical skills.
• Excellent communication and documentation skills. Ability to work in Agile/Scrum teams.

Preferred Skills:

• Experience with Azure Databricks for advanced analytics.
• Knowledge of machine learning workflows and integration with data engineering pipelines.
• Familiarity with streaming technologies like Azure Stream Analytics or Kafka.
• Understanding of data governance frameworks (eg, Azure Purview).
• Hands on with C#, MVC .Net.

Education and Experience:

• bachelors or masters degree in computer science, Information Systems, or a related field.
• 3+ years of experience as a Data Engineer or in a similar role.