Lead Data & Platform Engineer

Mastercard

9 - 13 years

17 - 22 Lacs

pune

Posted:3 days ago| Platform:

Apply

Skills Required

hive spark software engineering hadoop python bitbucket cloud technologies sql java jenkins etl big data etl development rally snowflake data analysis aws sagemaker impala data engineering nosql data bricks firewall hue sqoop agile aws

Work Mode

Work from Office

Job Type

Full Time

Job Description

Overview

We are looking for a self-motivated and enthusiastic Lead Cloud Data / Platform Engineer to join the Data Platforms and Engineering Services team. This individual will be responsible to work with our Cloud Architects, Cloud infrastructure, Information Security, Technical Product Managers, Senior Stakeholders and other Cloud data / platform engineers to ensure we are delivering the right solutions for our customers.
Data Platform provides real time streaming, batch data processing, pipeline orchestration, data lake management, data cataloging capabilities and deliver data on time with quality, supported by metrics. As a key player of our data platform team, you will have opportunity to use your expertise to work in solving big data problems, design, coding and analytical skills to build core capabilities, frameworks and data pipelines.

Role

Design, develop and implement large scale, high-volume, high-performance, highly available, scalable data platform infrastructure and pipelines for the Lake house data platforms in AWS cloud.
Work closely with senior data / platform engineers and data architects using Agile methodology, take ownership of the assigned tasks and deliver the results in timely manner.
Assist in troubleshooting and resolving issues with data platform /pipelines, AWS services ensuring they are running smoothly and efficiently.
Stay up to date with emerging technologies and trends in the data engineering and cloud space and willingness to learn and use new tools and platforms that can improve efficiency.

All About You:

Overall 9-13 years of career experience into Software Engineering or a related field
Architect and implement robust data pipelines across large-scale distributed systems using AWS native services.
Lead the development and maintenance of data platforms leveraging services such as:
--Data & Analytics: AWS Glue, EMR, MSK, Athena, Redshift
--Data Governance: Lake Formation, Glue Catalog
--Storage: S3 (General Purpose and S3 Tables)
--Monitoring: CloudWatch, CloudTrail
--Work hands-on with open table formats including Apache Iceberg and Parquet.
Design and deploy infrastructure using CloudFormation and Infrastructure as Code ( IaC ) principles.
Develop scalable solutions using at least one programming language (e.g., Python, Java).
Experience in data processing platforms on cloud such as Data bricks, Apache Spark, and Snowflake to handle large-scale data processing and distributed computing across massive datasets.
Collaborate with cross-functional teams to ensure data integrity, governance, and accessibility.
Utilize data modeling techniques for both SQL and NoSQL systems and apply ETL and data warehousing best practices.
Contribute to machine learning workflows and analytics using tools like Amazon SageMaker and MWA.
Proven experience in building and managing data platforms in cloud environments, especially AWS.
Experience and knowledge of Bit Bucket, Rally, and Jenkins a plus
Supporting all tiers of applications deployed in the cloud, configuring network, cloud native services, infrastructure, and privileges for development teams (AWS Accounts, IAM Users and Roles, VPCs, S3 buckets, cloud native services) and Knowledge of cloud networking concepts, subnets, routing, load balancing, firewalls, and cloud security.
Knowledge with Hadoop (CDP), relational databases and SQL, ETL development, data validation and testing, Data Analysis, and experience with Hadoop technologies such as Hive, Impala, Sqoop, Hue, Spark, Python of Hadoop Platform is a plus.
Strong communication and collaboration skills, with the ability to work effectively with data professionals and business stakeholders.
Detail-oriented with the ability to manage multiple priorities in a fast-paced, deadline-driven environment.
Bachelors degree or higher in Computer Science, Information Technology, or a related field.

More Jobs at Mastercard

Senior Analyst - Project Management

Pune

5 - 9 yrs

INR 13 - 18 Lacs

Sr. Principal Software Engineer

Pune

9 - 13 yrs

INR 20 - 25 Lacs

Lead Software Engineer-R-245726

Vadodara, Gujarat, India

2 - 4 yrs

Salary: Not disclosed

Consultant-2

Gurugram, Haryana

Experience: Not specified

Salary: Not disclosed

Consultant-2

Gurugram, Haryana

Experience: Not specified

Salary: Not disclosed

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.