Senior Data Engineer

Experience

5+ Years

Job Purpose

• We are seeking an experienced Senior Big Data Engineer to take a leading role in designing, building, and managing the data driven apps for one of our customers. You will be responsible for understanding evolving customer needs and translating them into robust, scalable, and cost-effective data solutions on the AWS cloud. You will make critical architectural decisions, choosing the best-fit services from the AWS ecosystem to handle everything from real-time event streaming to large-scale batch processing.

Job Description / Duties and Responsibilities

• Lead design of data solutions on AWS, selecting optimal services for data processing (EMR, Glue, EMR Serverless, Lambda), storage (Cassandra, Elasticsearch), and messaging (Kafka, SNS/SES)
• Develop, test, and deploy scalable data pipelines using PySpark and Scala for batch and real-time streaming
• Engineer and optimize OLTP data persistence layer using Cassandra for high-throughput workloads
• Utilize Elasticsearch for search and analytics needs
• Build and maintain real-time data backbone using Apache Kafka for event streaming
• Implement AWS SNS/SES for asynchronous notifications
• Create and manage complex data workflows using Apache Airflow
• Work with product managers and developers to translate requirements into technical specifications
• Monitor, troubleshoot, and optimize data pipelines for performance, scalability, and cost-efficiency

Job Specification / Skills and Competencies

• 5+ years experience building and managing complex data systems
• Expert-level proficiency in PySpark and/or Scala
• Deep hands-on AWS experience (EMR, Glue, Lambda, S3, EC2)
• Strong experience designing data models in Cassandra and Elasticsearch
• Demonstrable experience with real-time streaming using Apache Kafka
• Proven ability with workflow orchestration using Apache Airflow
• Strong analytical skills for technology selection and system design
• Excellent problem-solving abilities and attention to detail
• Strong communication and interpersonal skills
• Ability to work independently and collaboratively
• Capacity to quickly learn and adapt to new technologies
• Good presentation skills
• Pro-active problem solver and self-leader
• 7+ years total experience

Any Additional Information/Specifics

• Bachelor’s/Master’s in Computer Science, Data Science, Engineering or related field
• Snowflake knowledge (good to have)
• Prior consulting/contracting experience with tight deadlines
• Ability to manage and nurture team of data engineers
• Adhere to Information Security Management policies and procedures