Data Engineer

The TeamThe Data Engineering team is responsible for architecting, building, and maintaining our evolving data infrastructure, as well as curating and governing the data assets created on our platform. We work closely with various stakeholders to acquire, process, and refine vast datasets, focusing on creating scalable and optimized data pipelines. Our team possesses broad expertise in critical data domains, technology stacks, and architectural patterns. We foster knowledge sharing and collaboration, resulting in a unified strategy and seamless data management.
The Impact: This role is the foundation of the products delivered. The data onboarded is the base for the company as it feeds into the products, platforms, and essential for supporting our advanced analytics and machine learning initiatives.
Whats in it for youBe the part of a successful team which works on delivering top priority projects which will directly contribute to Companys strategy. Drive the testing initiatives including supporting Automation strategy, performance, and security testing. This is the place to enhance your Testing skills while adding value to the business. As an experienced member of the team, you will have the opportunity to own and drive a project end to end and collaborate with developers, business analysts and product managers who are experts in their domain which can help you to build multiple skillsets.
Responsibilities

Design, develop, and maintain scalable and efficient data pipelines to process large volumes of data.

To implement ETL processes to acquire, validate, and process incoming data from diverse sources.

Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and translate them into technical solutions.

Implement data ingestion, transformation, and integration processes to ensure data quality, accuracy, and consistency.

Optimize Spark jobs and data processing workflows for performance, scalability, and reliability.

Troubleshoot and resolve issues related to data pipelines, data processing, and performance bottlenecks.

Conduct code reviews and provide constructive feedback to junior team members to ensure code quality and best practices adherence.

Stay updated with the latest advancements in Spark and related technologies and evaluate their potential for enhancing existing data engineering processes.

Develop and maintain documentation, including technical specifications, data models, and system architecture diagrams.

Stay abreast of emerging trends and technologies in the data engineering and big data space and propose innovative solutions to enhance data processing capabilities.

What Were Looking For

5+ Years of experience in Data Engineering or related field

Strong experience in Python programming with expertise in building data-intensive applications.

Proven hands-on experience with Apache Spark, including Spark Core, Spark SQL, Spark Streaming, and Spark MLlib.

Solid understanding of distributed computing concepts, parallel processing, and cluster computing frameworks.

Proficiency in data modeling, data warehousing, and ETL techniques.

Experience with workflow management platforms, preferably Airflow.

Familiarity with big data technologies such as Hadoop, Hive, or HBase.

Strong Knowledge of SQL and experience with relational databases.

Hand on experience with AWS cloud data platform

Strong problem-solving and troubleshooting skills, with the ability to analyze complex data engineering issues and provide effective solutions.

Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams.

Nice to have experience on DataBricks

Preferred QualificationsBachelors degree in Information Technology, Computer Information Systems, Computer Engineering, Computer Science, or other technical discipline
Whats In It For
You?
Our Purpose:
Progress is not a self-starter. It requires a catalyst to be set in motion. Information, imagination, people, technologythe right combination can unlock possibility and change the world. At S&P Global we transform data into Essential Intelligence, pinpointing risks and opening possibilities. Our People:
Our Values:
Integrity, Discovery, Partnership
At S&P Global, we focus on Powering Global Markets. We start with a foundation of integrity in all we do, bring a spirit of
discovery to our work, and collaborate in close
partnership with each other and our customers to achieve shared goals.
Benefits:
We take care of you, so you cantake care of business. We care about our people. Thats why we provide everything youand your careerneed to thrive at S&P Global.

Health & WellnessHealth care coverage designed for the mind and body.

Flexible DowntimeGenerous time off helps keep you energized for your time on.

Continuous LearningAccess a wealth of resources to grow your career and learn valuable new skills.

Invest in Your FutureSecure your financial future through competitive pay, retirement planning, a continuing education program with a company-matched student loan contribution, and financial wellness programs.

Family Friendly PerksIts not just about you. S&P Global has perks for your partners and little ones, too, with some best-in class benefits for families.

Beyond the BasicsFrom retail discounts to referral incentive awardssmall perks can make a big difference.