Data Engineer

As a Data Engineer with 3-5 years of experience, you will be responsible for creating data ingestion flows for consumption in Gen AI solutions. Your role will involve designing, developing, and optimizing ETL processes and data pipelines using Azure Data Factory and Azure Databricks. Additionally, you will implement AI-driven data solutions leveraging Azure OpenAI, Prompt Engineering, and Retrieval-Augmented Generation (RAG). You will be expected to develop and maintain Azure Functions for seamless integration and automation, as well as utilize Python and C# for data transformation, integration, and AI model deployment. Managing CI/CD pipelines using Azure DevOps for efficient deployment and monitoring will also be a key aspect of your responsibilities. Experience with Langchain and LlamaIndex to enhance AI-driven data applications would be preferred. Your expertise in ETL & Data Pipelines using Azure Data Factory and Azure Databricks, along with proficiency in C# and Python for data processing and integrations, will be crucial for this role. Familiarity with Azure Functions for cloud-based automation, Azure OpenAI, Prompt Engineering, and RAG for AI-enhanced solutions, as well as Azure DevOps for CI/CD and deployment automation, will also be required. Ensuring data quality, security, and scalability in cloud-based environments will be an essential part of your responsibilities.,