Data Engineer - Scala Spark

Role Overview: You will be responsible for designing, building, and optimizing large-scale ETL and data-processing pipelines within the Databricks ecosystem. Your role will involve migrating selected workloads to high-performance engines like Polars and DuckDB while maintaining strong engineering rigor. You will need to apply analytical thinking to address data reliability, performance, and scalability issues. Familiarity with AI concepts will be advantageous in this role. Key Responsibilities: - Develop and maintain distributed data pipelines using Scala, Spark, Delta, and Databricks. - Engineer robust ETL workflows for high-volume ingestion, transformation, and publishing. - Profile pipelines, identify and eliminate bottlenecks, and optimize compute, storage, and job orchestration. - Lead the migration of suitable workloads to high-performance engines such as Polars, DuckDB, or equivalent. - Implement CI/CD workflows with automated builds, tests, deployments, and environment gating. - Enforce coding standards through code coverage targets, unit/integration tests, and SonarQube rules. - Ensure pipeline observability by implementing logging, data quality checks, lineage, and failure diagnostics. - Apply analytical reasoning to troubleshoot complex data issues and provide root-cause clarity. - Collaborate with product, analytics, and platform teams to operationalize data solutions. Qualification Required: - 3+ years of experience in data engineering with a strong command of Scala and Spark. - Proven background in ETL design, distributed processing, and high-volume data systems. - Hands-on experience with Databricks, including jobs, clusters, notebooks, and Delta Lake. - Proficiency in workflow optimization, performance tuning, and memory management. - Experience with high-performance engines like Polars, DuckDB, or similar columnar/accelerated engines. - Knowledge of CI/CD discipline using Git-based pipelines, strong testing practices, and code-quality standards. - Familiarity with SonarQube, coverage metrics, and static analysis. - Strong analytical and debugging capabilities across data, pipelines, and infrastructure. - Exposure to AI concepts such as embeddings, vector stores, retrieval-augmented generation, fine-tuning, and agentic architectures. Additional Details: Experience with Azure cloud environments and metadata-driven or config-driven pipeline frameworks is preferred.,