Big Data Engineer

Kmccorp

5 - 10 years

6 - 10 Lacs

Hyderabad

Posted:1 day ago| Platform:

Apply

Skills Required

Root cause analysis data security Scheduling Apache Resource management big data Troubleshooting Information technology YARN Monitoring

Work Mode

Work from Office

Job Type

Full Time

Job Description

Design and optimize batch/streaming data pipelines using Scala, Spark, and Kafka
Implement real-time tokenization/cleansing microservices in Java
Manage production workflows via Apache Airflow (batch scheduling)
Conduct root-cause analysis of data incidents using Spark/Dynatrace logs
Monitor EMR clusters and optimize performance via YARN/Dynatrace metrics
Ensure data security through HashiCorp Vault (Transform Secrets Engine)
Validate data integrity and configure alerting systems

Requirements

Technical Requirements

Programming :Scala (Spark batch/streaming), Java (real-time microservices)
Big Data Systems: Apache Spark, EMR, HDFS, YARN resource management
Cloud & Storage :Amazon S3, EKS
Security: HashiCorp Vault, tokenization vs. encryption (FPE)
Orchestration :Apache Airflow (batch scheduling)
Operational Excellence Spark log analysis, Dynatrace monitoring, incident handling, data validation

Mandatory Competencies

Expertise in distributed data processing (Spark on EMR/Hadoop)
Proficiency in shell scripting and YARN job management
Ability to implement format-preserving encryption (tokenization solutions)
Experience with production troubleshooting (executor logs, metrics, RCA)

More Jobs at Kmccorp

Java Full Stack Lead

Hyderabad

10 - 15 yrs

INR 13 - 18 Lacs

Senior Full Stack Developer (Frontend-Leaning)

Hyderabad

8 - 10 yrs

INR 8 - 13 Lacs

Java Full Stack Developer

Hyderabad

5 - 7 yrs

INR 7 - 9 Lacs

Senior QA Automation Engineer

Kolkata, Mumbai, New Delhi, Hyderabad, Pune, Chennai, Bengaluru

10.0 - 15.0 yrs

INR 10 - 14 Lacs

Platform Engineer

Hyderabad

7.0 - 12.0 yrs

INR 20 - 25 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Big Data Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.