Senior Data Engineer

Responsibilities:

Data Pipeline Development:

Design, develop, and maintain distributed data pipelines on AWS that ingest, process, and deliver data at scale. Implement both batch and event-driven data processing patterns using AWS-native services.

Event-Driven Architecture:

Build and support event-driven data solutions using asynchronous, decoupled architectures to enable near real-time processing and system scalability.

Technical Expertise:

Provide hands-on engineering using Python and Apache Spark to process large datasets efficiently. Apply best practices in distributed systems, performance optimization, and fault tolerance.

Data Quality, Lineage & Orchestration:

Ensure end-to-end data quality, implement validation and monitoring checks, establish data lineage, and manage orchestration workflows to ensure reliable and auditable data movement.

API Development:

Design and develop RESTful and event-driven APIs to expose data and data services to internal and external consumers.

Containers & Cloud-Native Development:

Build and deploy data services using Docker containers and manage workloads on Kubernetes following cloud-native and DevOps best practices.

Stakeholder Collaboration:

Collaborate with data consumers, architects, platform teams, and business stakeholders to gather requirements, design scalable solutions, and deliver high-quality outcomes.

Operational Excellence:

Contribute to monitoring, logging, alerting, and incident response for data platforms. Ensure systems meet reliability, performance, and security standards.

Requirements:

• BS in Computer Science, Engineering, or equivalent practical experience
• 8+ years of experience in data engineering or distributed systems development
• Strong hands-on experience with AWS cloud services
• Advanced proficiency in Python
• Hands-on experience with Apache Spark
• Experience building event-driven data pipelines
• Experience designing distributed data processing architectures
• Strong knowledge of data quality, data lineage, and orchestration
• Experience in API development
• Hands-on experience with Docker and Kubernetes