Senior Data Engineer

Position Overview

We are seeking a highly skilled Senior Data Engineer to join our data platform team, responsible for designing and implementing scalable, cloud-native data solutions. This role focuses on building modern data infrastructure using AWS and Azure services to support our growing analytics and machine learning initiatives.

Key Responsibilities

• Business Translation

  : Collaborate with business stakeholders to understand requirements and translate them into scalable data models, architectures, and robust end-to-end data pipelines

• ETL/ELT Implementation

  : Design, develop, and maintain high-performance batch and real-time data pipelines using modern frameworks like Apache Spark, Delta Lake, and streaming technologies

• Data Platform Management

  : Architect and implement data lakes, data warehouses, and Lakehouse architectures following medallion architecture principles (Bronze, Silver, Gold layers)

Data Operations & Quality

• Pipeline Orchestration

  : Implement complex workflow orchestration using tools like AWS Step Functions or Azure Data Factory

• Data Quality Assurance

  : Establish comprehensive data validation, monitoring, and quality frameworks using tools like Great Expectations or custom Python solutions

• Performance Optimization

  : Monitor, troubleshoot, and optimize data pipeline performance, implementing partitioning strategies and query optimization techniques

• Data Governance

  : Implement data lineage tracking, metadata management, and ensure compliance with data privacy regulations (GDPR)

Cloud Infrastructure & DevOps

Experience & Background

• 6+ years

  of hands-on experience in Data Engineering roles with demonstrated expertise in cloud-native data solutions

• Cloud Expertise

  : Strong preference for AWS/Azure data services

• Production Experience

  : Proven track record of building and maintaining production-grade data pipelines processing terabytes of data

Technical Skills

Programming & Development

• Python

  : Advanced proficiency in Python for data engineering (pandas, numpy, boto3, azure-sdk)

• SQL

  : Expert-level SQL skills including complex queries, window functions, CTEs, and performance tuning

• PySpark

  : Hands-on experience with PySpark for large-scale data processing and optimization

• Version Control

  : Proficient with Git workflows, branching strategies, and collaborative development

AWS Data Services (Primary Focus)

• Compute

  : Experience working on services like AWS Glue, Lambda Azure Data Factory, Synapse Analytics, Data Lake Storage Gen2

  ,

  S3 Redshift, RDS, DynamoDB

• Analytics

  : Power BI, Athena, Kinesis (Data Streams, Data Firehose, Analytics)

• Security

  : IAM, KMS, VPC, Security Groups for secure data access patterns

Soft Skills & Collaboration

• Problem-Solving

  : Strong analytical and troubleshooting skills with attention to detail

• Communication

  : Excellent verbal and written communication skills for cross-functional collaboration

• Project Management

  : Ability to manage multiple priorities and deliver projects on time

• Mentorship

  : Experience mentoring junior engineers and promoting best practices

Preferred Qualifications (Nice to Have)

• CI/CD Pipelines: Experience with GitHub Actions, Jenkins, or Azure DevOps for automated testing and deployment

• Infrastructure as Code: Terraform or AWS CDK for infrastructure automation

• Containerization: Docker and Kubernetes for containerized data applications

• Testing Frameworks: Unit testing, integration testing, and data quality testing practices

• Cloud Certifications: AWS Certified Data Analytics, Azure Data Engineer Associate, or similar certifications

• Agile Methodologies: Experience working in Scrum/Kanban environments

What We Offer

• Opportunity to work with cutting-edge data technologies and modern cloud platforms
• Collaborative environment with opportunities for professional growth and learning
• Flexible work arrangements and comprehensive benefits package
• Conference attendance and certification reimbursement programs

This role requires the ability to work independently while collaborating effectively with cross-functional teams including Data Scientists, Analytics Engineers, Software Engineers, and Business Stakeholders.

Bottom of Form