Jobs

Interviews
Job Alerts
Tools

Upskill and Grow with AI

Mock Interview Practice interviews in realistic simulations

Coding Practice Improve your coding skills with challenges

Certification Earn certifications to validate your skills

AI Learning Get trained with AI expert sessions

Career Path AI insights for smarter career decisions

AI Job Match Score AI-Powered Job Match Against Your Resume and Optimize Your Resume

Career Tools and Resources

Resume Builder Build Professional Resume with Ease

ATS Friendliness Check Check Resume Friendliness for Applicant Tracking Systems

Auto Apply Apply to hundreds of jobs on any platform effortlessly

Co-Pilot (Chrome Extension) Your AI Assistant for Seamless Browsing Efficiency

Interview Questions Streamline interviews with ready-to-use questions

Salaries Discover market-driven salary insights across skillsets and geographies

Companies Explore leading companies actively hiring talent
For Employers

Home
>
Jobs in hyderābād
>
Pennant Technologies
>
Big Data Engineer

Big Data Engineer

Pennant Technologies

5 years

0 Lacs

hyderābād

Posted:17 hours ago| Platform: GlassDoor logo

Apply

Skills Required

data analytics design ml development etl apache airflow storage governance unity processing spark scala java sql server collaboration mlflow tracking model containerization docker kubernetes code engineering testing monitoring sensors tuning strategies deployment scaling debugging integration programming python git jenkins github aws gcp azure hive kafka compliance certifications certification databricks communication learning collaborative agile

Work Mode

On-site

Job Type

Part Time

Job Description

Job Title: Senior Big Data Engineer

Department: Data & Analytics

Employment Type: Full‑time

Role Overview

We are seeking an experienced Big Data Engineer to join our growing Data & Analytics team. In this role, you’ll design, build, and optimize our modern data platform—enabling scalable pipelines, reliable data lakes, and robust ML operations. You’ll work closely with data scientists, analysts, and stakeholders to turn raw data into actionable insights.

Key Responsibilities

Data Platform Development & Maintenance
Design, deploy, and maintain ETL/ELT pipelines using Apache Airflow and dbt/meltano.
Architect and manage data lakes on HDFS and object storage (e.g., S3/HDFS).
Implement table formats and governance with Apache Iceberg and Unity Catalog/Apache ranger/Atlas.
Big Data Processing
Develop distributed data processing jobs in Apache Spark (Scala/Java/Python).
Optimize Spark applications for performance, cost, and scalability.
Good working knowledge in Databases (Postgres, Mongo, SQL, Sql server etc)
Collaboration with Data science teams
Integrate MLflow for experiment tracking, model registry, and seamless deployment.
Collaborate with data science teams to operationalize models in production.
Containerization & Orchestration
Containerize data services and workflows using Docker/kubenetes.
Orchestrate deployments on Kubernetes clusters (EKS/GKE/AKS).
Center of Excellence & Best Practices
Contribute to and govern the dbt Center of Excellence: define standards, code reviews, and documentation.
Advocate and enforce data engineering best practices: CI/CD, testing, monitoring, and observability.
Cross‑Functional Collaboration
Partner with Analytics, Product, and Engineering teams to understand requirements and deliver solutions.
Mentor junior engineers and share knowledge through code reviews and technical talks.

Required Skills & Experience

5+ years of professional experience in Big Data engineering or similar roles.
Airflow: Hands‑on experience authoring DAGs, configuring operators/sensors, and managing scheduler/executor backends.
Apache Spark: Proficient in building and tuning large‑scale ETL jobs (Scala/Python).
Apache Iceberg & Unity Catalog: Familiarity with table formats, partitioning strategies, and catalog management.
Data Lake & HDFS: Solid understanding of storage architectures, file formats (Parquet/ORC), and data partitioning.
MLflow: Working knowledge of experiment tracking, model registry, and deployment workflows.
Kubernetes: Skilled in deploying, scaling, and debugging containerized applications and stateful workloads.
dbt COE: Experience with dbt for data modeling, CI/CD integration, and establishing a CoE for standards.
Strong programming skills in Python and/or Scala.
Experience with version control (Git), CI/CD pipelines (Jenkins/GitHub Actions), and monitoring tools (Prometheus, Grafana).

Nice‑to‑Have

Familiarity with cloud platforms (AWS, GCP, or Azure) and managed services (EMR, Dataproc, EKS, GKE).
Knowledge of additional data catalogs (e.g., AWS Glue, Hive Metastore).
Background in event‑driven architectures (Kafka, Pulsar).
Prior involvement in data governance or compliance initiatives.

Education & Certifications

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
Relevant certifications (e.g., Any cloud certification AWS/GCP/Azure/Databricks Certified Big Data – Specialty) are a plus.

Soft Skills

Excellent problem‑solving and analytical abilities.
Strong written and verbal communication; able to explain technical concepts to non‑technical stakeholders.
Proactive mindset with a passion for learning and driving innovation.
Collaborative team player who thrives in agile, fast‑paced environments.

Join us and help shape the future of our data platform—enabling smarter decisions, faster insights, and transformative business outcomes!

More Jobs at Pennant Technologies

Social Media Marketing Lead

Hyderābād

8.0 - 10.0 yrs

INR 3 - 5 Lacs

Lead Business Solutions Consultant - Lending (Retail & Corporate)

Hyderābād

5.0 - 10.0 yrs

Salary: Not disclosed

Assistant Vice President (AVP) - AI/ML Technologies

Hyderābād

10.0 - 10.0 yrs

INR 6 - 7 Lacs

Senior Manager, Presales

Hyderābād

5.0 - 5.0 yrs

Salary: Not disclosed

Lead Engineer, Support

Visakhapatnam

8.0 - 13.0 yrs

INR 15 - 30 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Java Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Enhance Your Java Skills

Practice Java coding challenges to boost your skills

Start Practicing Java Now

Pennant Technologies

Software Development

Hyderabad Telangana

Login to

Please Verify Your Phone or Email

Confirm Action

Big Data Engineer