Senior Data Engineer

Role & responsibilities

Design and build a scalable data platform that can ingest, store, manage and stream massive amounts of data while simplifying its analysis and processing to enable rapid development of high-quality data products and services.
•Implement and test robust low latency data pipelines for preparing/curating data that will support various data as a service product. •You will work with various cross functional teams to explore data and figure out all the data wrangling needed to clean, curate the data and build the end-to-end data pipeline per requirements. •You will design necessary privacy and security preventive and detective controls into the CICD pipelines of each data pipeline. •You will have a strong bias for operational excellence ensuring error handling, restart-ability ensuring data consistency, logging, monitoring and alerting is built into the pipeline. •You will drive continual improvements for reliability, performance, scalability, quality and own associated KPI.

• 5+ years of experience in development and operation in production of Cloud native data streaming data systems (with an emphasis on scalability, robustness, data delivery low latency and data privacy and quality control)
  •5+ years experience working in Cloud native real-time streaming data ecosystem such as Spark, Flink, Kinesis, Lambda, Kafka, EMR/EKS platform, Lakehouse platform (i.e Delta.io, Databricks) •5+ years of experience in developing optimizing AWS data system architecture: Infrastructure as Code, integration and deployment automation, security.
• 5+ years of experience in ensuring operation reliability of cloud native data system in production.
  •5+ years in relational database concepts with a solid knowledge of SQL, SQL Tuning, Big Data NoSQL technologies •5+ years’ experience with languages like Spark-Scala, Py-Spark and /or Python, Java •Experience with development and deployment of containerized application using Docker, Kubernetes, Helm •Experience in building systems that monitor data losses and data quality control. •Demonstrated knowledge of data structures and algorithms. •Experience in designing foundations of Operational excellence and implementing. •Curiosity and passion to learn with a strong bias to action