Position: Data Engineer (Azure Expertise) (CB30FT RM 3695)
Work mode : Hybrid / work from office
Position Overview
We are seeking a highly skilled Data Engineer with a strong background in Azure technologies and expertise in data pipeline development, transformation, and analytics. The ideal candidate will have hands-on experience with Azure Data Factory (ADF), Azure Functions, Synapse Analytics, SQL Server, Python, and PySpark, along with a solid understanding of CI/CD pipelines and Power BI. Your role will involve designing, building, and maintaining scalable and reliable data solutions that empower the organization to make data driven decisions.
Key Responsibilities
Design, Build, and Maintain Data Pipelines:
- Develop and optimize end-to-end ETL/ELT pipelines using Azure Data Factory (ADF).
- Integrate, transform, and load data from diverse sources into Azure Synapse Analytics or SQL-based data stores.
- Implement data workflows and automation using Azure Functions. Database Development and Optimization:
- Design and manage data models and schemas in SQL Server and Synapse Analytics.
- Write optimized SQL queries and stored procedures for efficient data retrieval and transformation.
- Ensure database integrity, performance tuning, and scalability. Big Data Processing:
- Use PySpark to process and analyze large-scale data in a distributed environment.
- Implement and manage data lakes and data lake houses for batch and real-time processing. Data Governance and Security:
- Implement data governance policies, including data quality, lineage, and compliance.
- Secure data pipelines using role-based access control and encryption standards. CI/CD and Automation:
- Build and manage CI/CD pipelines for data integration and deployment using tools like Azure DevOps, Git, or GitHub Actions.
- Automate testing, version control, and deployment of data engineering solutions. Data Visualization and Reporting:
- Collaborate with business stakeholders to design and deliver interactive dashboards using Power BI.
- Enable data exploration and self-service reporting through semantic layers. Collaboration and Communication:
- Work closely with cross-functional teams including analysts, and business users to gather requirements and implement data solutions.
- Document workflows, data models, and architecture to ensure knowledge sharing across teams.
Azure Expertise
Required Skills and Qualifications:
- Hands-on experience with Azure Data Factory (ADF) for building ETL/ELT workflows.
- Proficiency with Azure Synapse Analytics for data warehousing and analytics.
- Strong knowledge of Azure Functions for building event-driven automation. Programming:
- Proficiency in Python, with experience in libraries such as Pandas, NumPy, and PySpark.
- Experience with distributed data processing frameworks like Apache Spark. Database Management:
- Expertise in SQL Server.
- Knowledge of designing and implementing data lakes and lake houses. CI/CD:
- Experience building and deploying pipelines with Azure DevOps, Jenkins, or similar tools. Data Visualization:
- Proficiency in Power BI for creating dashboards and reports.
- Experience with DAX and Power Query for advanced analytics. Data Integration and APIs:
- Experience with REST and SOAP APIs for data integration.
- Familiarity with messaging systems like Azure Event Hubs or Azure Service Bus.
General Skills
- Strong problem-solving and analytical skills.
- Excellent communication and documentation skills. Ability to work in Agile/Scrum teams.
Preferred Skills
- Experience with Azure Databricks for advanced analytics.
- Knowledge of machine learning workflows and integration with data engineering pipelines.
- Familiarity with streaming technologies like Azure Stream Analytics or Kafka.
- Understanding of data governance frameworks (e.g., Azure Purview).
- Hands on with C#, MVC .Net.
Education And Experience
- Bachelor’s or master’s degree in computer science, Information Systems, or a related field.
- 3+ years of experience as a Data Engineer or in a similar role.
