Home
Jobs

Posted:1 week ago| Platform: Linkedin logo

Apply

Skills Required

data design kafka databricks architecture layers tuning optimization code versioning deployment automation governance azure spark processing management integration unity sql query configuration strategies logic capture

Work Mode

On-site

Job Type

Full Time

Job Description

Key Responsibilities • Design and implement streaming data pipelines integrating Kafka with Databricks using Structured Streaming. • Architect and maintain Medallion Architecture with well-defined Bronze, Silver, and Gold layers. • Implement efficient ingestion using Databricks Autoloader for high-throughput data loads. • Work with large volumes of structured and unstructured data, ensuring high availability and performance. • Apply performance tuning techniques such as partitioning, caching, and cluster resource optimization. • Collaborate with cross-functional teams (data scientists, analysts, business users) to build robust data solutions. • Establish best practices for code versioning, deployment automation, and data governance. Required Technical Skills: • Strong expertise in Azure Databricks and Spark Structured Streaming • Processing modes (append, update, complete) • Output modes (append, complete, update) • Checkpointing and state management • Experience with Kafka integration for real-time data pipelines • Deep understanding of Medallion Architecture • Proficiency with Databricks Autoloader and schema evolution • Deep understanding of Unity Catalog and Foreign catalog • Strong knowledge of Spark SQL, Delta Lake, and DataFrames • Expertise in performance tuning (query optimization, cluster configuration, caching strategies) • Must have Data management strategies • Excellent with Governance and Access management • Strong with Data modelling, Data warehousing concepts, Databricks as a platform • Solid understanding of Window functions Proven experience in: • Merge/Upsert logic • Implementing SCD Type 1 and Type 2 • Handling CDC (Change Data Capture) scenarios • Retail/Telcom/Energy any one industry expertise • Real time use case execution • Data modelling Show more Show less

More Jobs at Celebal Technologies

Nodejs Full-Stack Developer

Jaipur

3 - 6 yrs
|

INR 0 - 3 Lacs

Hr Analytics Specialist

Jaipur

4 - 8 yrs
|

INR 5 - 8 Lacs

Urgently Looking For Data Scientist|| Immediate Joiner|| Celebal Tech

Delhi NCR, Jaipur

3 - 7 yrs
|

INR 15 - 25 Lacs

Qa Automation Testing Engineer

Delhi NCR, Jaipur

2 - 5 yrs
|

INR 5 - 7 Lacs

Technical Recruiter

Pune

3 - 5 yrs
|

INR 4 - 8 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Data Interview Now

My Connections Celebal Technologies

Download Chrome Extension (See your connection in the Celebal Technologies )

chrome image
Download Now
Celebal Technologies
Celebal Technologies
www.celebaltechnologies.com

Technology Consulting and Services

Ahmedabad

201-500 Employees

39 Jobs

    Key People

  • Siddharth Ghosh

    Co-Founder & CEO

  • Vishal Kothari

    Co-Founder

RecommendedJobs for You

Roku logo

Senior Data Engineer

Roku

Bengaluru

Connectedx Techsol logo

Senior Data Engineer

Connectedx Techsol

Hyderabad

Blackbaud logo

Senior Data Engineer

Blackbaud

Hyderabad

Novo Nordisk logo

Senior Data Engineer

Novo Nordisk

Bengaluru

Thomson Reuters logo

Senior Data Engineer

Thomson Reuters

Hyderabad

Zeta Global logo

Senior Data Engineer

Zeta Global

Bengaluru

Alstom Transportation logo

Senior Data Engineer

Alstom Transportation

Bengaluru

Systems Plus logo

Senior Data Engineer

Systems Plus

Pune

Thomson Reuters logo

Senior Data Engineer

Thomson Reuters

Hyderābād

Circle K logo

Senior Data Engineer

Circle K

Gurgaon