MindBrain logo

Data Engineer – Databricks + GCP (5+ yrs)

0 years

0 Lacs

india

Posted:1 day ago| Platform: Linkedin logo

Apply

Skills Required

data databricks gcp pyspark etl design storage versioning retention relational analytics aws azure processing tuning spark query checks logging monitoring documentation devops deployment orchestration git airflow

Work Mode

Remote

Job Type

Contractual

Job Description

Data Engineer – Databricks + GCP (5+ yrs )

Remote

Contract


Skills: PySpark, Databricks, Delta Lake, SQL

•⁠ ⁠Build ETL pipelines

•⁠ ⁠Manage Delta Lake

•⁠ ⁠Data integrations

•⁠ ⁠Improve performance


Key Responsibilities
  • Design, build, and maintain robust

    ETL/ELT pipelines

    using

    PySpark

    on

    Databricks

    .
  • Manage and administer

    Delta Lake

    tables — including schema evolution, partitioning, ACID-compliant data storage, versioning, and data retention.
  • Ingest data from diverse sources (relational databases, flat files, APIs, streaming systems) — structured, semi-structured, and unstructured — and transform them into analytics-ready datasets.
  • Implement data integrations across systems and platforms — including cloud storage (e.g., AWS S3 / Azure Data Lake), data warehouses/lakehouses, and data catalogs.
  • Optimize data-processing workflows for performance and cost — tuning Spark jobs, optimizing partitioning, caching, query patterns, and resource utilization.
  • Build pipelines for both

    batch and streaming

    data workflows; implement data quality checks, validation, error handling, logging, and monitoring.
  • Collaborate with data analysts, data scientists, and business stakeholders to translate analytical requirements into scalable data solutions.
  • Maintain metadata, documentation, version control; work with DevOps/CI-CD pipelines for deployment and orchestration of workflows (e.g., via Databricks Workflows, Git, Airflow, etc.)

More Jobs at MindBrain

EDI CLEO Specialist

India

Experience: Not specified

|

Salary: Not disclosed

Senior Generative Ai Engineer

India

Experience: Not specified

|

Salary: Not disclosed

Sr. Dot Net (Azure Integration Developer )3 to 5 yrs

India

3.0 - 5.0 yrs
|

Salary: Not disclosed

Senior AI/ML Engineer

India

5.0 - 5.0 yrs
|

Salary: Not disclosed

Senior Python Developer

India

Experience: Not specified

|

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start PySpark Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Google Play App Store
coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
MindBrain logo
MindBrain

RecommendedJobs for You

MindBrain logo

Data Engineer – Databricks + GCP (5+ yrs)

MindBrain

india

MindBrain logo

Data Engineer – Databricks + GCP (5+ yrs)

MindBrain

india