Job
Description
Key Responsibilities : Design, construct, and maintain efficient and reliable data pipelines to support both traditional data warehousing and real-time data processing needs. Implement ETL (Extract, Transform, Load) processes and data workflows that are scalable and optimized for performance. Manage data lakes, data warehouses, and databases, ensuring data quality, consistency, and security. Index and catalog unstructured data to enable advanced analytics and machine learning capabilities. Administer vector databases for AI/ML applications, optimizing for query performance and scalability. Securely integrate REST APIs and external data sources into our systems, maintaining compliance with data governance standards. Work with AI/ML teams to leverage OpenAI, Azure, and other LLM endpoints, ensuring seamless data integration and pipeline compatibility. Contribute to the creation of synthetic data sets for robust model training, employing data anonymization and augmentation techniques where necessary. Utilize prompt engineering to refine the inputs and outputs of generative AI models, enhancing their utility and accuracy. Monitor, troubleshoot, and optimize data systems to ensure high availability and minimal downtime. Research and implement best practices in data engineering, staying abreast of new technologies and methodologies in the field. Skills & Qualifications Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related field. Minimum of 3 years of experience in data engineering, with a proven track record of successful data pipeline development. Expertise in SQL and NoSQL databases, data modeling, and data warehousing solutions. Proficiency in programming languages such as Python and scripting languages for automation. Experience with big data technologies (e.g., Hadoop, Spark, Kafka) and understanding of distributed computing principles. Familiarity with vector databases and their role in supporting AI/ML workloads. Demonstrated experience with REST API development and consumption, with a focus on security and data protection. Practical knowledge of cloud services and infrastructure as it relates to data engineering (e.g., AWS, Azure, GCP). Understanding of machine learning concepts and experience with LLMs is a plus. Excellent analytical and problem-solving abilities, with meticulous attention to detail. Strong communication and collaboration skills, with the ability to work effectively in a team environment. Additional Desirable Skills Certifications in cloud platforms, big data technologies, or related fields. Experience with data governance, compliance, and privacy regulations. Familiarity with DevOps practices and tools for data operations. Active engagement with the data engineering community through blogs, talks, or open-source contributions. Qualifications : Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field. Strong experience with Python and Node.js in a production environment. Proficiency in working with vector embeddings and understanding their role in AI applications. Experience with Elastic or similar databases for storing and querying large datasets. Familiarity with AI frameworks and libraries, particularly those related to NLP and chatbot development. Solid understanding of RESTful API design and development. Knowledge of cloud services and deployment (AWS, Azure, GCP). Proficient in using version control systems like Git. Prior experience building conversational AI or chatbots. Understanding of machine learning operations (MLOps) and deployment of AI models. Experience with containerization and orchestration tools (Docker, Kubernetes). Familiarity with CI/CD pipelines and automated testing frameworks.
