Data Engineer -MHP-2528

Role Overview

Data Engineer

Key Responsibilities

• Data Pipeline Development:

  Build, maintain, and optimize data pipelines using

  Airflow

  for orchestration and scheduling.

• Data Ingestion & Transformation:

  Work with

  flat files

  (CSV, JSON, Mainframe-based files) and ensure accurate ingestion and transformation.

• Spark-based Processing:

  Use

  PySpark

  for large-scale data processing and implementing custom UDF (User Defined Functions) where needed.

• SQL Development:

  Create, optimize, and maintain SQL scripts for data manipulation, reporting, and analytics.

• Snowflake Data Integration:

  Load and manage final processed data layers in

  Snowflake

  .

• Data Quality & Metrics:

  Implement checks for file size limits, data consistency, and daily metric tracking.

• Collaboration & Requirements Gathering:

  Work with business and technical teams to understand requirements and deliver efficient data solutions.

Required Skills & Experience

• Proficiency in

  SQL

  (query optimization, joins, indexing).
• Strong

  Python

  programming skills, including writing reusable functions.
• Hands-on experience with

  PySpark

  (adding columns, transformations, cache usage, UDF functions).
• Proficiency in

  Airflow

  for workflow orchestration.
• Familiarity with

  Trino

  and

  Hive

  query engines.
• Experience with

  flat file formats

  (CSV, JSON, Mainframe-based files) and data parsing strategies.
• Understanding of

  data normalization

  , unique constraints, and caching strategies.
• Experience working with

  Snowflake

  or other cloud data warehouses.

Preferred Qualifications

• Knowledge of performance tuning in Spark and SQL.
• Understanding of data governance and security best practices.
• Experience with large file processing (including max size handling).