Data Engineer - Scala expertise

As a Senior Data Engineer (Scala) at our technology services & software exports company, your primary responsibility will be to design, develop, and maintain scalable data pipelines using Scala and Apache Spark for batch and streaming workloads. Your role also involves implementing Kafka-based ingestion, processing streams with structured streaming, and ensuring data reliability across the analytics lifecycle. You will be expected to author efficient Spark SQL transformations, optimize job performance, and resource utilization on cluster platforms. Additionally, building robust ETL workflows, handling schema evolution, and implementing data quality checks and monitoring will be part of your daily tasks. Collaboration with data consumers, data scientists, and platform teams to translate requirements into reliable data products and APIs is essential. Participating in CI/CD for data pipelines, troubleshooting production incidents, and creating runbooks to improve operational maturity are also key responsibilities. Key Responsibilities: - Design, develop, and maintain scalable data pipelines using Scala and Apache Spark for batch and streaming workloads. - Implement Kafka-based ingestion, process streams with structured streaming, and ensure exactly-once or at-least-once guarantees as required. - Author efficient Spark SQL transformations and optimize job performance and resource utilization on cluster platforms. - Build robust ETL workflows, handle schema evolution, and implement data quality checks and monitoring. - Collaborate with data consumers, data scientists, and platform teams to translate requirements into reliable data products and APIs. - Participate in CI/CD for data pipelines, troubleshoot production incidents, and create runbooks to improve operational maturity. Qualifications Required: Must-Have: - Scala - Apache Spark - Spark SQL - Apache Kafka - Hadoop HDFS - SQL - ETL - AWS S3 Preferred: - Databricks - Apache Airflow - Delta Lake In this role, you will experience a fast-paced engineering culture with ownership of end-to-end data products. You will have opportunities to work on large-scale streaming and analytics for enterprise clients. There is structured career growth and technical mentorship within the data engineering practice. Please highlight your Scala + Spark project experience, examples of Kafka or streaming implementations, and any performance optimization or production debugging you have led when applying for this position.,