Jobs
Interviews
UST logo

Lead II -Databrick Engineer

8 years

Thiruvananthapuram

Posted:16 hours ago

Apply

Job Description

Role Description

Lead Databricks Engineer-Lead Databricks Developers with streaming expertise. Key Responsibilities
  • Design and develop end-to-end data pipelines using Databricks (PySpark, Delta Lake).
  • Build real-time streaming ingestion pipelines from Kafka and other event-driven sources.
  • Implement layered data architecture (Bronze, Silver, Gold) to support transformation and reporting. Develop data processing logic for transactional data ingestion, validation, and enrichment. Apply data quality, deduplication, and business rules for healthcare datasets.
  • Support cloud migration from on-premise systems to Databricks + Snowflake ecosystem. Optimize pipeline performance to meet SLA-driven processing timelines. Implement and support CI/CD pipelines and DevOps integration. Collaborate with business and technical stakeholders to deliver scalable solutions. Required Skills
  • Strong expertise in Databricks (Streaming & Batch) Advanced proficiency in PySpark / Spark SQL/Spark Streaming
  • Experience in real-time data streaming (Kafka) Strong understanding of data modeling and ETL/ELT pipeline design
  • Hands-on experience with AWS (S3, Airflow, Lambda) Experience with CI/CD, Git, and deployment automation Domain Experience
  • Experience in Healthcare Payer domain (Claims, Membership, Coverage) Exposure to FHIR / healthcare data standards preferred Understanding of data privacy, security, and governance requirements (PII/HIPAA) Leadership Expectations
  • Operate at a Lead level driving design, development, and delivery Mentor team members and ensure delivery quality Own modules independently and collaborate across distributed teams Experience
  • 8+ years in Data Engineering
  • 3+ years hands-on experience with Databricks Prior experience in Tech Lead/Senior roles.

Skills

Apache Spark, PySpark, Databricks, Apache Kafka

Skills Required

Pyspark Kafka Data Architecture Data Processing Data Ingestion Data Quality Business Rules Healthcare Cloud Migration Ci/cd Pipelines Devops Spark

Learn the skills this job asks for

Free courses, interview questions and practice for each skill.

Job Type

Full Time

More Jobs at UST

Lead I - Software Engineering - JAVA Developer

Mumbai

5 - 7 yrs
|

Salary: Not disclosed

Lead II - Data Engineering

Pune

5 - 8 yrs
|

Salary: Not disclosed

Lead II -Databrick Engineer

Thiruvananthapuram

8 - 8 yrs
|

Salary: Not disclosed

Senior DevOps Engineer (Azure & Terraform)

Pune

6 - 10 yrs
|

Salary: Not disclosed

Junior Cloud Engineer – AWS & Linux

Thiruvananthapuram

2 - 2 yrs
|

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
UST logo
UST

IT Services and IT Consulting

Aliso Viejo CA

RecommendedJobs for You

UST logo

Lead II -Databrick Engineer

UST

Thiruvananthapuram

UST logo

Lead II -Databrick Engineer

UST

Thiruvananthapuram