Big Data Engineer - Spark/Scala

As a Spark Scala Developer, you will be responsible for designing and developing scalable data pipelines using Apache Spark and Scala. Your role will involve optimizing and troubleshooting Spark jobs for performance, working with massive datasets in on-prem Hadoop clusters or cloud platforms, and writing clean, modular Scala code using functional programming principles. Collaborating with data teams to integrate with platforms like Snowflake, Databricks, or data lakes will also be a key part of your responsibilities. It will be essential to ensure code quality, documentation, and CI/CD practices are followed diligently. The ideal candidate for this position should possess 3+ years of experience with Apache Spark in Scala and have a deep understanding of Spark internals including DAG, stages, tasks, caching, joins, and partitioning. Hands-on experience with performance tuning in production Spark jobs, proficiency in Scala functional programming, and a strong command over SQL are must-have skills. Additionally, experience with any major cloud platform such as AWS, Azure, or GCP is required. Having worked with Databricks, Snowflake, or Delta Lake, exposure to data pipeline tools like Airflow, Kafka, Glue, or BigQuery, familiarity with CI/CD pipelines and Git-based workflows, and comfort with SQL optimization and schema design in distributed environments are considered nice-to-have skills for this role. If you are someone with real-world experience working in Big Data environments, both on-prem and/or in the cloud, and possess the necessary skills and expertise in Spark Scala development, this opportunity may be the right fit for you.,