Senior Data Engineer

As a Data Engineer specializing in Spark & Delta Lake, your role will involve understanding key Spark concepts such as RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming. You will also be expected to have experience in optimizing Spark performance and a deep understanding of Delta Lake features like time travel, schema evolution, and data partitioning. Your responsibilities will include designing and implementing data pipelines using Spark and Delta Lake as the data storage layer. Proficiency in Python, Scala, or Java for Spark development and integration with ETL processes is essential. Additionally, you should possess knowledge of data ingestion techniques from various sources such as flat files, CSV, API, and databases. Understanding data quality best practices and data validation techniques is crucial for this role. Key Responsibilities: - Understand Spark core concepts like RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming - Experience in optimizing Spark performance - Deep knowledge of Delta Lake features like time travel, schema evolution, and data partitioning - Design and implement data pipelines using Spark and Delta Lake - Proficiency in Python/Scala/Java for Spark development and integration with ETL process - Knowledge of data ingestion techniques from various sources - Understanding of data quality best practices and data validation techniques Qualifications Required: - Proficiency in Python, Scala, or Java - Experience with Spark optimization techniques - Knowledge of data ingestion techniques - Understanding of data quality best practices - Familiarity with data warehouse concepts and data modeling techniques - Expertise in Git for code management - Familiarity with CI/CD pipelines and containerization technologies - Nice to have experience using data integration tools like DataStage, Prophecy, Informatica, or Ab Initio,