Data Engineer

Experience- 3 to 12 yrs

Responsibilities:

• Develop, deploy, and operate data extraction and automation pipelines in production

• Integrate and deploy machine learning models into those pipelines (e.g., inference services,

batch scoring)

• Lead critical stages of the data engineering lifecycle, including:

• End-to-end delivery of complex extraction, transformation, and ML deployment

projects

• Scaling and replicating pipelines on AWS (EKS, ECS, Lambda, S3, RDS)
• Designing and managing DataOps processes, including Celery/Redis task queues and

Airflow orchestration

• Implementing robust CI/CD pipelines on Azure DevOps (build, test, deployment,

rollback)

• Writing and maintaining comprehensive unit, integration, and end-to-end tests

(pytest, coverage)

• Strengthen data quality, reliability, and observability through logging, metrics, and

automated alerts

• Define and evolve platform standards and best practices for code, testing, and deployment

• Document architecture, processes, and runbooks to ensure reproducibility and smooth

hand-offs

• Partner closely with data scientists, ML engineers, and product teams to align on

requirements, SLAs, and delivery timelines

Technical Requirements:

• Expert proficiency in Python, including building extraction libraries and RESTful APIs

• Hands-on experience with task queues and orchestration: Celery, Redis, Airflow

• Strong AWS expertise: EKS/ECS, Lambda, S3, RDS/DynamoDB, IAM, CloudWatch

• Containerization and orchestration: Docker (mandatory), basic Kubernetes (preferred)

• Proven experience deploying ML models to production (e.g., SageMaker, ECS, Lambda

endpoints)

• Proficient in writing tests (unit, integration, load) and enforcing high coverage

• Solid understanding of CI/CD practices and hands-on experience with Azure DevOps

pipelines

• Familiarity with SQL and NoSQL stores for extracted data (e.g., PostgreSQL, MongoDB)

• Strong debugging, performance tuning, and automation skills

• Openness to evaluate and adopt emerging tools and languages as needed

Good to have:

• Master's or Bachelor's degree in Computer Science, Engineering, or related field

• 2-6 years of relevant experience in data engineering, automation, or ML deployment

• Prior contributions on GitHub, technical blogs, or open-source projects

• Basic familiarity with GenAI model integration (calling LLM or embedding APIs)