Data Engineer(Azure)

Job Summary

We are seeking a Data Engineer to work on a Generative AI initiative to join our team. The ideal candidate will have a deep understanding of data modeling, data schemas and has developed ETLs from various sources eensuring high data availability, fault tolerance and security and governance.

Responsibilities:

• Collaborate with stakeholders to understand data requirements and design scalable and efficient data models, schemas, and architectures on the Azure platform.
• Develop and implement data integration and ETL (Extract, Transform, Load) processes to ingest, transform, and load data from various sources into Azure data storage systems.
• Build and maintain scalable and reliable data pipelines using Azure Data Factory, Azure Databricks, or similar tools to automate the movement and transformation of data.
• Optimize data storage and retrieval mechanisms, ensuring high data availability, fault tolerance, and efficient query performance.
• Hands-on experience with cloud platforms (MS Azure, AWS or Google Cloud Services)
• Hands-on experience with distributed computing (preferably using Apache Spark or Databricks)
• Monitor and troubleshoot data pipelines, identifying and resolving performance issues, data quality problems, and data integration challenges.
• Collaborate with data scientists and analysts to provide them with access to reliable and well-structured data sets for analysis and reporting purposes.
• Implement data governance and security measures to protect sensitive data, ensure compliance with regulatory standards, and adhere to data privacy policies.
• Work closely with the DevOps team to automate deployment processes, ensure continuous integration and delivery of data solutions, and maintain infrastructure as code.
• Stay updated with the latest advancements in Azure data services, evaluate and implement new tools and features to enhance data engineering processes.

Requirements

• Bachelor''s or masters degree in computer science, Engineering, or a related field.
• Proven experience as a Data Engineer, with a strong focus on Azure cloud platform.
• In-depth knowledge of Azure data services such as Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure SQL Database, Azure Data Lake Storage, and Azure Cosmos DB.
• Proficiency in designing and implementing scalable data models, data warehouses, and data lakes.
• Strong understanding of ETL processes, data integration patterns, and data transformation techniques.
• Experience with programming languages such as Python, SQL, and PowerShell for data manipulation, scripting, and automation.
• Familiarity with big data technologies and frameworks like Hadoop, Spark, and NoSQL databases is a plus.
• Strong analytical and problem-solving skills, with the ability to troubleshoot and optimize data pipelines.
• Excellent communication and collaboration skills to work effectively in a team environment and interact with stakeholders from various disciplines.

Preferred Qualifications

• Bachelor''s or master''s degree in computer science, Data Science, Statistics, Math, Physics, or other Science related discipline with course work in AI/ML.
• Cloud certifications preferred.
• Contribute to the design and implementation of new data-driven solutions, using cutting-edge machine learning and artificial intelligence techniques.
• Self-motivated with a strong desire to learn and stay up to date with new technologies in the field.
• Ability to work collaboratively in a team environment and communicate complex technical concepts to non-technical stakeholders.