Job
Description
Role & responsibilities Architect and lead the development of complex, scalable data pipelines using technologies like Scala, Spark, Python, and Airflow. Set technical standards and best practices across data engineering teams. Oversee AWS-based data infrastructure (S3, Redshift, Glue, EMR), ensuring reliability and performance at scale. Lead real-time streaming solutions (Kafka, Kinesis, Snowplow), optimizing for low-latency and high throughput. Mentor and guide teams on advanced system design, data observability, and best practices. Drive DataOps initiatives, focusing on data quality, lineage, and governance. Collaborate with cross-functional teams to align technical architecture with business goals. Lead the integration of ML platforms like SageMaker for advanced analytics. Champion operational excellence through incident resolution and proactive monitoring. Qualifications: 8+ years of hands-on experience in data engineering, with expertise in Scala, Spark, Python, and Airflow. Mastery in cloud data infrastructure (AWS Redshift, Glue, Athena, S3). Expertise in real-time streaming technologies (Kafka, Kinesis, Snowplow). Proven ability in DataOps, focusing on data observability, quality, and governance. Strong leadership and mentoring skills with a track record of guiding technical teams. Deep experience with large-scale distributed systems and Java, Vertx, Spring Boot is a plus. Familiarity with machine learning platforms (e.g., SageMaker) is preferred.
