ML pipeline engineer

About the Role

Pipeline Engineer

Key Responsibilities

• Design, implement, and maintain

  CI/CD pipelines

  for software applications or automated data workflows.
• Develop, test, and deploy

  ETL/data processing pipelines

  using tools like Apache Airflow, NiFi, or similar.
• Automate build, deployment, and monitoring processes to improve efficiency and reliability.
  
• Collaborate with

  DevOps, data engineering, and software development teams

  to ensure seamless integration and delivery.
• Monitor pipeline performance, troubleshoot failures, and optimize workflows for speed and reliability.
  
• Maintain version control, configuration management, and release documentation.
  
• Implement

  security, compliance, and quality checks

  within pipelines.
• Stay updated with industry trends and best practices in CI/CD, DevOps, and data pipeline automation.
  

Required Skills & Qualifications

• Bachelors degree in

  Computer Science, Information Technology, or a related field

  .
• 2–6 years of experience in

  pipeline development, DevOps, or data engineering

  .
• Strong experience with

  CI/CD tools

  (Jenkins, GitLab CI, CircleCI, Azure DevOps, etc.).
• Proficiency in

  scripting languages

  (Python, Bash, Shell, Groovy, etc.).
• Knowledge of

  version control systems

  (Git, SVN).
• Hands-on experience with

  cloud platforms

  (AWS, Azure, GCP) and cloud services for compute, storage, and orchestration.
• Familiarity with

  containerization

  (Docker) and

  orchestration

  (Kubernetes).
• Understanding of

  monitoring, logging, and alerting tools

  (Prometheus, Grafana, ELK stack).
• Strong analytical, problem-solving, and collaboration skills.
  

Nice-to-Have

• Experience with

  data pipeline tools

  like Apache Airflow, Luigi, NiFi, or Spark.
• Knowledge of

  infrastructure as code

  (Terraform, CloudFormation).
• Exposure to

  microservices architecture and API integration

  .
• Understanding of

  automated testing frameworks

  and quality assurance practices.
• Familiarity with

  CI/CD security and compliance standards