Design, develop, and maintain scalable data pipelines using
Databricks
and
Apache Spark
.
Build and optimize ETL/ELT workflows for large-scale datasets.
Develop Delta Lake tables and manage data quality, reliability, and performance.
Integrate data from various sources (APIs, databases, cloud storage).
Collaborate with data scientists, analysts, and business stakeholders to support analytics and ML workloads.
Implement best practices for CI/CD, version control, and automation.
Monitor platform performance and troubleshoot issues in Databricks clusters and jobs.

Required Skills & Experience

3+ years of experience as a Data Engineer or similar role.
Hands-on experience with
Databricks
(Jobs, Workflows, Delta Lake, Unity Catalog, Notebooks).
Strong proficiency in
Apache Spark (PySpark/Scala)
.
Experience with at least one cloud platform:
Azure, AWS, or GCP
.
Strong SQL knowledge for data modelling, transformations, and performance tuning.
Experience with CI/CD tools (Git, Azure DevOps, Jenkins, etc.).
Solid understanding of data warehousing and data-lake architecture.

Preferred Qualifications

Databricks certification(s) (Data Engineer Associate/Professional).
Experience with MLflow or supporting data science workloads.
Knowledge of streaming technologies (Structured Streaming/Kafka).
Experience with orchestration tools (Airflow, ADF, AWS Glue, etc.).

What We Offer

Competitive compensation and benefits.
Opportunity to work with modern cloud and big-data technologies.
Collaborative, growth-oriented team environment.
Career development and certification support.

More Jobs at iDWTeam

Front Desk Representative

hyderabad, telangana, india

Experience: Not specified

Salary: Not disclosed

SAP PP Functional Consultant

hyderabad, telangana, india

Experience: Not specified

Salary: Not disclosed

Front Desk Representative

hyderabad, telangana, india

Experience: Not specified

Salary: Not disclosed

Information Technology Sales Executive

hyderabad, telangana

3.0 - 7.0 yrs

Salary: Not disclosed

Front Desk Representative

hyderabad, telangana

1.0 - 5.0 yrs

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.