Devops Engineer (BigData)

Key Responsibilities:

Primary DevOps Responsibilities

• Design, implement, and maintain CI/CD pipelines for applications and Big Data workloads.
  
• Automate infrastructure provisioning using Infrastructure as Code (Terraform/CloudFormation/ARM).
  
• Manage containerized environments using Docker and Kubernetes.
  
• Deploy and manage cloud-based Big Data platforms (AWS/Azure/GCP).
  
• Implement monitoring, logging, and alerting solutions.
  
• Ensure system reliability, scalability, and security compliance.
  
• Manage version control and release management processes.
  

Secondary – Big Data Platform Support

• Deploy and support Hadoop ecosystem components (HDFS, Hive, Spark, etc.).
  
• Support Big Data clusters (on-prem or cloud-based like EMR/Dataproc/HDInsight).
  
• Assist in managing data pipelines and workflow orchestration (Airflow/Oozie).
  
• Monitor performance of Big Data environments and optimize cluster configurations.
  
• Support Data Marts and Data Warehouse environments.
  

Required Skills (Must Have – DevOps Focus)

• Strong hands-on experience in:
  
• CI/CD tools (Jenkins/GitLab CI/Azure DevOps)
  
• Docker & Kubernetes
  
• Terraform / Infrastructure as Code
  
• Cloud platforms (AWS/Azure/GCP)
  
• Linux administration and shell scripting
  
• Monitoring tools (Prometheus, Grafana, ELK, Cloud monitoring tools)
  

Good to Have (Big Data Exposure)

• Hadoop ecosystem experience
  
• Spark cluster management
  
• Experience with EMR / Dataproc / HDInsight
  
• Knowledge of Data pipelines & orchestration tools (Airflow)
  
• Understanding of Data Marts / Data Warehousing concepts
  
• Kafka exposure is a plus
  

Preferred Profile:

• Experience supporting production Big Data platforms
  
• Strong troubleshooting and performance tuning skills
  
• Ability to work with Data Engineering and Architecture teams
  

If you'd like, I can also provide:

• A shorter 8–10 line JD (for quick sharing)
  
• A Visa-style structured JD format
  
• Screening checklist for recruiter shortlisting
  
• Interview questions for L1 technical screening