Databricks Data Engineer

Crisil

3 years

0 Lacs

mumbai maharashtra india

Posted:1 week ago| Platform:

Apply

Skills Required

databricks data design analytics apache spark learning storage etl consistency reporting automate scheduling pipeline engineering software pyspark azure aws governance unity processing kafka git visualization power tableau certifications

Work Mode

On-site

Job Type

Full Time

Job Description

Position Overview:

Databricks Data Engineer

Key Responsibilities:

Develop and maintain
scalable data pipelines using Databricks and Apache Spark (primarily with PySpark).
Build and manage
Delta Lake tables and optimize them for analytics and machine learning workloads.
Integrate data from diverse sources
including APIs, cloud storage, databases, and streaming platforms like Kafka.
Transform r
aw data into clean, structured formats using best practices in ETL/ELT.
Implement data quality checks
and ensure consistency, completeness, and accuracy of data sets.
Collaborate
with data scientists, analysts, and business stakeholders to meet data requirements for reporting and analytics.
Automate workflows
using Databricks Workflows, job clusters, and scheduling tools.
Monitor and optimize Spark jobs for performance and cost efficiency.
Document and maintain data models, pipeline designs, and technical workflows.

Qualifications:

Education
: Bachelor’s degree in Computer Science, Engineering, or a related field.
Experience
:
- 3+ years of experience in data engineering or software development.
- 1+ years of hands-on experience working with Databricks and Apache Spark in production environments
Skills
:
- Proficient in PySpark and SQL.
- Solid understanding of data warehousing, ETL/ELT design, and big data best practices.
- Experience with cloud platforms such as Azure, AWS, or GCP.
- Experience with Delta Lake, Databricks Workflows, and MLflow.
- Exposure to data governance tools like Unity Catalog or Purview.
- Experience with real-time data processing (e.g., using Kafka, Structured Streaming).
- Familiarity with version control systems (e.g., Git) and CI/CD practices.
- Familiarity with data visualization tools (e.g., Power BI, Tableau) and BI integration
Certifications :
- Databricks Certified Data Engineer Associate or Professional (a plus).

More Jobs at Crisil

Operational Resilience - Vice President

Mumbai, United Kingdom

9 - 14 yrs

INR 33 - 38 Lacs

Associate Director

Mumbai

10 - 15 yrs

INR 28 - 33 Lacs

Cloud Engineer

Chennai, Pune

5 - 8 yrs

INR 9 - 13 Lacs

Cloud Engineer

Chennai, Pune

3 - 6 yrs

INR 8 - 12 Lacs

IT Risk and Governance

Chennai, Pune

5 - 8 yrs

INR 16 - 21 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.