Sr Data Engineer

Senior Data Engineer - Cloud Data Platform

Position Overview

We are seeking a highly skilled Senior Data Engineer to join our data platform team, responsible for designing and implementing scalable, cloud-native data solutions. This role focuses on building modern data infrastructure using AWS and Azure services to support our growing analytics and machine learning initiatives.

Key Responsibilities

• Business Translation

  : Collaborate with business stakeholders to understand requirements and translate them into scalable data models, architectures, and robust end-to-end data pipelines

• ETL/ELT Implementation

  : Design, develop, and maintain high-performance batch and real-time data pipelines using modern frameworks like Apache Spark, Delta Lake, and streaming technologies

• Data Platform Management

  : Architect and implement data lakes, data warehouses, and Lakehouse architectures following medallion architecture principles (Bronze, Silver, Gold layers)

Data Operations & Quality

• Pipeline Orchestration

  : Implement complex workflow orchestration using tools like AWS Step Functions or Azure Data Factory

• Data Quality Assurance

  : Establish comprehensive data validation, monitoring, and quality frameworks using tools like Great Expectations or custom Python solutions

• Performance Optimization

  : Monitor, troubleshoot, and optimize data pipeline performance, implementing partitioning strategies and query optimization techniques

• Data Governance

  : Implement data lineage tracking, metadata management, and ensure compliance with data privacy regulations (GDPR)

Cloud Infrastructure & DevOps

Experience & Background

• 6+ years

  of hands-on experience in Data Engineering roles with demonstrated expertise in cloud-native data solutions

• Cloud Expertise

  : Strong preference for AWS/Azure data services

• Production Experience

  : Proven track record of building and maintaining production-grade data pipelines processing terabytes of data

Technical Skills

Programming & Development

• Python

  : Advanced proficiency in Python for data engineering (pandas, numpy, boto3, azure-sdk)

• SQL

  : Expert-level SQL skills including complex queries, window functions, CTEs, and performance tuning

• PySpark

  : Hands-on experience with PySpark for large-scale data processing and optimization

• Version Control

  : Proficient with Git workflows, branching strategies, and collaborative development

AWS Data Services (Primary Focus)

• Compute

  : Experience working on services like AWS Glue, Lambda Azure Data Factory, Synapse Analytics, Data Lake Storage Gen2

  ,

  S3 Redshift, RDS, DynamoDB

• Analytics

  : Power BI, Athena, Kinesis (Data Streams, Data Firehose, Analytics)

Security