Senior Data Platform Engineer

Required Skills:

Scala, Spark, and Kafka (any variant).

Proficient in working with distributed systems and have experience with different distributed processing frameworks that can handle data in batch and near real-time pipelines.

AWS cloud services and Databricks (preferred)

Capability to understand and implement metadata-based data processing framework and methodologies

Experience in working with CICD process - Jenkins, & GIT Actions

Experience in working with monitoring and control stack - Prometheus/Grafana

Key job responsibilities:

• Drive the backbone of our organizational data platform by building robust pipelines that turn complex data into actionable insights using AWS and the Databricks platform
• Design scalable, maintainable data systems with strong observability
• Monitor, log, alert, and ensure system robustness
• Build efficient pipelines for structured and unstructured data
• Interface with Product engineering, regulatory, and legal teams as necessary to ensure Lakehouses compliance adherence
• Write clean, efficient code that handles massive amounts of structured and unstructured data