Big Data Engineer

KMCCorp India Pvt Ltd

0 years

0 Lacs

Hyderabad Telangana India

Posted:2 days ago| Platform:

Apply

Skills Required

data design scala spark kafka tokenization microservices java apache airflow scheduling analysis metrics security vault integrity programming management storage encryption orchestration monitoring processing hadoop scripting troubleshooting timing

Work Mode

On-site

Job Type

Full Time

Job Description

Core Responsibilities

Design and optimize batch/streaming data pipelines using Scala, Spark, and Kafka
Implement real-time tokenization/cleansing microservices in Java
Manage production workflows via Apache Airflow (batch scheduling)
Conduct root-cause analysis of data incidents using Spark/Dynatrace logs
Monitor EMR clusters and optimize performance via YARN/Dynatrace metrics
Ensure data security through HashiCorp Vault (Transform Secrets Engine)
Validate data integrity and configure alerting systems

Requirements

Technical Requirements

Programming :Scala (Spark batch/streaming), Java (real-time microservices)
Big Data Systems: Apache Spark, EMR, HDFS, YARN resource management
Cloud & Storage :Amazon S3, EKS
Security: HashiCorp Vault, tokenization vs. encryption (FPE)
Orchestration :Apache Airflow (batch scheduling)
Operational Excellence Spark log analysis, Dynatrace monitoring, incident handling, data validation

Mandatory Competencies

Expertise in distributed data processing (Spark on EMR/Hadoop)
Proficiency in shell scripting and YARN job management
Ability to implement format-preserving encryption (tokenization solutions)
Experience with production troubleshooting (executor logs, metrics, RCA)

Benefits
Benefits

Insurance - Family
Term Insurance PF

Paid Time Off - 20 days
Holidays - 10 daysFlexi timingCompetitive SalaryDiverse & Inclusive workspace

More Jobs at KMCCorp India Pvt Ltd

Senior QA Automation Engineer

Telangana, India

Experience: Not specified

Salary: Not disclosed

Big Data Engineer

Hyderabad, Telangana, India

Experience: Not specified

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start Java Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.