Spark & Delta Lake Specialist

You are a Spark & Delta Lake Specialist with 8+ years of experience and proficiency in Python, Spark (PySpark), SQL, and Delta Lake. Your role involves understanding Spark core concepts such as RDDs, DataFrames, DataSets, SparkSQL, and Spark Streaming. You have experience with Spark optimization techniques and deep knowledge of Delta Lake features like time travel, schema evolution, and data partitioning. Your responsibilities include designing and implementing data pipelines using Spark and Delta Lake as the data storage layer. You are expected to have proficiency in Python, Scala, Java for Spark development, and integration with ETL processes. Additionally, you have knowledge of data ingestion techniques from various sources including flat files, CSV, API, and databases. Understanding data quality best practices, data validation techniques, data warehouse concepts, and data modeling techniques is essential for this role. Expertise in Git for code management, familiarity with CI/CD pipelines, and containerization technologies are required. It would be a plus if you have experience using data integration tools like DataStage, Prophecy, Informatica, or Ab Initio. This is a contract position with our technology services client based in Hyderabad. There is a strong potential for conversion to full-time employment upon completion of the initial contract period. The notice period for this role is immediate to 15 days. If you are interested in this opportunity, please share your updated resume to vinod@s3staff.com.,