PEI Group is a subscriber-focused business intelligence company that focuses on private investment markets in real estate, infrastructure, private equity, private debt, and specialist sector-specific activities within private asset classes. We provide industry-leading journalism, data, and market insight to subscribing clients via a wide portfolio of specialist brands supported by our robust and scalable digital publishing, analytics, and database platform.

Since its inception in 2001, we have grown into a subscriber-focused business intelligence company with our multi-talented global team of over 400 people spread across EMEA, USA & Asia, and our purpose is to inform and connect investment professionals across global, specialised markets.

As a

Lead

Data Engineer

, you will lead, design, implement, and maintain data processing pipelines and workflows using Databricks on the Azure platform. Your expertise in PySpark, SQL, Databricks, test-driven development, and Docker will be essential to the success of our data engineering initiatives.

Roles and responsibilities:

Collaborate with cross-functional teams to understand data requirements and design scalable and efficient data processing solutions.
Develop and maintain data pipelines using PySpark and SQL on the Databricks platform.
Optimise and tune data processing jobs for performance and reliability.
Implement automated testing and monitoring processes to ensure data quality and reliability.
Work closely with data scientists, data analysts, and other stakeholders to understand their data needs and provide effective solutions.
Troubleshoot and resolve data-related issues, including performance bottlenecks and data quality problems.
Stay up to date with industry trends and best practices in data engineering and Databricks.

Key Requirements:

8+ years
of experience as a Data Engineer, with a focus on Databricks and cloud-based data platforms, with a minimum of
4 years
of experience in writing unit/end-to-end tests for data pipelines and ETL processes on Databricks.
Hands-on experience in
PySpark
programming for data manipulation, transformation, and analysis.
Strong experience in
SQL
and writing complex queries for data retrieval and manipulation.
Experience in Docker for containerising and deploying data engineering applications is good to have.
Strong knowledge of the Databricks platform and its components, including Databricks notebooks, clusters, and jobs.
Experience in designing and implementing data models to support analytical and reporting needs will be an added advantage.

More Jobs at THE Scalers

Lead - People and Culture

Bengaluru

5.0 - 7.0 yrs

INR 11 - 15 Lacs

Senior Java Developer

Bengaluru

3.0 - 8.0 yrs

INR 5 - 10 Lacs

Senior Data Engineer

Bengaluru

5.0 - 10.0 yrs

INR 7 - 12 Lacs

Mid Cloud Developer

Bengaluru

3.0 - 8.0 yrs

INR 5 - 10 Lacs

Lead Data Engineer

Bengaluru

8.0 - 13.0 yrs

INR 25 - 30 Lacs

Mock Interview

Practice Video Interview with JobPe AI

Start Business Intelligence Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

Job Application AI Bot

Apply to 20+ Portals in one click

Download Now

Download the Mobile App

Instantly access job listings, apply easily, and track applications.

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now

THE Scalers

Login to

Please Verify Your Phone or Email

Confirm Action

Search

Profile

Upskill and Grow with AI

Lead Data Engineer