Data Engineer

Job Description

Design, develop, and maintain robust data pipelines to support machine learning and AI applications, ensuring data integrity and efficiency. Collaborate with cross-functional teams to collect, process, and analyze data from various sources for AI model training, validation, and deployment. Work with large language models (LLM) and generative AI technologies (e.g., OpenAI GPT) to implement intelligent systems and applications. Optimize and scale AI/ML pipelines to handle vast amounts of structured and unstructured data efficiently. Support the deployment and monitoring of AI models into production environments, ensuring high performance and low-latency operations. Implement best practices for data management, data quality, and data security in AI-driven systems. Troubleshoot and resolve data-related issues and bottlenecks across the AI infrastructure. Contribute to the design of data architecture to support the integration of AI models and machine learning frameworks. Key Requirements : 2-5 years of hands-on experience as a Data Engineer or in a similar role. Proven experience working with large language models (LLM), OpenAI technologies, and Generative AI (GenAI) frameworks. Strong proficiency in Python, SQL, and other programming languages commonly used in data engineering. Experience with data processing frameworks (e.g., Apache Spark, Apache Kafka, Hadoop) and cloud platforms (e.g., AWS, GCP, Azure). Familiarity with machine learning models, model deployment, and API integration. Solid understanding of data warehousing, ETL processes, and data architecture. Knowledge of data security, privacy standards, and compliance regulations. Strong problem-solving skills and ability to work in a fast-paced, collaborative environment. Excellent communication skills to work with cross-functional teams and stakeholders. Preferred Skills : Experience with OpenAI s GPT models and API integrations. Familiarity with advanced AI techniques like reinforcement learning, unsupervised learning, etc. Knowledge of containerization (e.g., Docker, Kubernetes) and CI/CD pipelines. Experience working with both structured and unstructured data (e.g., text, images, audio). Exposure to advanced data analytics and visualization tools (e.g., Tableau, Power BI, etc.)