Srijan Technologies PVT LTD logo

Senior/Lead Data Engineer - Databricks..

5 years

7 - 9 Lacs

gurgaon

Posted:4 days ago| Platform: GlassDoor logo

Apply

Skills Required

data databricks engineering programming architecture support analytics spark optimization management aws azure processing postgresql nosql mongodb design integration capture efficiency scalability integrity consistency accessibility governance devops drive pyspark etl gitlab pipeline communication collaboration

Work Mode

On-site

Job Type

Part Time

Job Description

Senior/Lead Data Engineer - Databricks


We are seeking a highly skilled and experienced Data Engineering Lead with a strong background in the retail domain and exceptional programming abilities. As a Lead, you will play a pivotal role in implementing, and optimizing data architecture to support our retail business operations and analytics initiatives. Your expertise in Spark programming, optimization techniques, and familiarity with Databricks and CI/CD practices will be instrumental in ensuring the efficient and effective management of our data ecosystem.

Imp Pointers as below skills should be in the CV

Data Platform (Data Engineers)

  • Data Lake: AWS S3 / Azure / Dell Cloud Data Lake with Delta Lake format
  • Data Warehouse: Databricks, IOMETE, Google BigQuery for analytical workloads
  • Batch Processing /Stream Processing: Batch/real-time processing of data pipelines
  • Database: PostgreSQL for transactional data, NoSQL (ex. MongoDB, Cassendra) for document storage

Responsibilities:


  • Design and develop data models, data integration processes, and data pipelines to capture, transform, and load structured and unstructured data from various retail sources.
  • Hands-on programming in Spark to develop and optimize data processing applications and analytics workflows.
  • Apply optimization techniques to enhance the performance and efficiency of data processing and analytical tasks.
  • Evaluate and implement appropriate tools and technologies, including Databricks, to streamline data operations and ensure scalability and reliability.
  • Work closely with other team members to ensure data integrity, consistency, and accessibility across the organization.
  • Define and enforce best practices for data governance and data management, including data quality, metadata management, and data security.
  • Collaborate with DevOps teams to establish and maintain CI/CD pipelines for data engineering and analytics workflows.
  • Peer Review of team members' deliverables
  • Stay updated with the latest advancements and trends in the retail domain, data architecture, and programming languages to drive continuous improvement.

Requirements:

  • At least 5+ years of experience in the data engineering domain.
  • Proven experience as a senior Data Engineer /Lead, preferably within the retail industry.
  • Strong programming skills with expertise in PySpark programming and optimization techniques.
  • Hands-on experience with Databricks, Deltalake and its components for data processing and analytics.
  • Hands-on experience in data modelling, data integration, and ETL/ELT processes.
  • Experience in working with Gitlab pipelines and an in-depth understanding of CI/CD pipeline designs.
  • Experience with data governance, data quality, and metadata management.
  • Strong analytical and problem-solving abilities with a detail-oriented mindset.
  • Excellent communication and collaboration skills to work effectively with cross-functional teams.
  • Ability to adapt to a fast-paced and evolving environment while managing multiple priorities.
  • Good to have experience in at least one of the Cloud Vendor (AWS / Azure / GCP)
  • Good to have experience with streaming technologies as well

More Jobs at Srijan Technologies PVT LTD

Lead Data Engineer (AWS Glue)

Gurgaon

Experience: Not specified

|

INR 6 - 10 Lacs

Technical Architect (AWS Glue)

Gurgaon

10.0 - 10.0 yrs
|

INR 4 - 7 Lacs

Technical Delivery Manager - Data Platform

Gurgaon

5.0 - 5.0 yrs
|

INR 2 - 10 Lacs

Associate - Market Research

Gurugram, Haryana

2.0 - 2.0 yrs
|

Salary: Not disclosed

Associate - Market Research

Gurgaon

2.0 - 2.0 yrs
|

INR 2 - 6 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start DevOps Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Google Play App Store
coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
Srijan Technologies PVT LTD logo
Srijan Technologies PVT LTD

RecommendedJobs for You

Srijan Technologies PVT LTD logo

Senior/Lead Data Engineer - Databricks..

Srijan Technologies PVT LTD

gurgaon

Srijan Technologies PVT LTD logo

Senior/Lead Data Engineer - Databricks..

Srijan Technologies PVT LTD

gurgaon