Data Engineer

Job Summary:

We are seeking a highly skilled Data Engineer with expertise in leveraging Data Lake architecture and the Azure cloud platform to develop, deploy, and optimise data-driven solutions. . You will play a pivotal role in transforming raw data into actionable insights, supporting strategic decision-making across the organisation.

Responsibilities

• Design and implement scalable data science solutions using

  Azure Data Lake, Azure Data Bricks, Azure Data Factory and related Azure services.

• Develop, train, and deploy machine learning models to address business challenges.
• Collaborate with data engineering teams to optimise data pipelines and ensure seamless data integration within Azure cloud infrastructure.
• Conduct exploratory data analysis (EDA) to identify trends, patterns, and insights.
• Build predictive and prescriptive models to support decision-making processes.
• Expertise in developing end-to-end Machine learning lifecycle utilizing crisp-DM which includes of data collection, cleansing, visualization, preprocessing, model development, model validation and model retraining
• Proficient in building and implementing RAG systems that enhance the accuracy and relevance of model outputs by integrating retrieval mechanisms with generative models.
• Ensure data security, compliance, and governance within the Azure cloud ecosystem.
• Monitor and optimise model performance and scalability in production environments.
• Prepare clear and concise documentation for developed models and workflows.

Skills Required:

• Good experience in using Pyspark, Python, MLops (Optional), ML flow (Optional), Azure Data Lake Storage. Unity Catalog
• Worked and utilized data from various RDBMS like MYSQL, SQL Server, Postgres and NoSQL databases like MongoDB, Cassandra, Redis and graph DB like Neo4j, Grakn.
• Proven experience as a Data Engineer with a strong focus on Azure cloud platform and Data Lake architecture.
• Proficiency in Python, Pyspark,
• Hands-on experience with Azure services such as Azure Data Lake, Azure Synapse Analytics, Azure Machine Learning, Azure Databricks, and Azure Functions.
• Strong knowledge of SQL and experience in querying large datasets from Data Lakes.
• Familiarity with data engineering tools and frameworks for data ingestion and transformation in Azure.
• Experience with version control systems (e.g., Git) and CI/CD pipelines for machine learning projects.
• Excellent problem-solving skills and the ability to work collaboratively in a team environment.