Spark/Scala Developer

You are a candidate with strong experience in Spark Core/Streaming, Spark RDD API, and Spark Dataframes API using programming languages such as Scala, Java, or Python. You have a background in Machine Learning or AI and possess a deep understanding of Data Structures, Algorithms, Data Transformation, Data Ingestion, and Optimization mechanisms. Additionally, you are well-versed in Big Data technologies including Hadoop, MapReduce, Kafka, and Cassandra. Your primary responsibility will be to design and develop scalable and high-performance data processing applications using Apache Spark (Core, Streaming) with Scala. You will work with Spark RDD and DataFrame APIs to create robust data pipelines and apply machine learning or AI techniques to solve real-world data challenges. Data ingestion from various structured and unstructured sources, optimizing Spark jobs for performance and resource utilization, and collaborating with cross-functional teams are integral parts of your role. You should have at least 4 years of experience in Big Data technologies, particularly in Apache Spark (Core/Streaming). Proficiency in Scala is preferred, along with knowledge of Java or Python. A strong understanding of Spark RDD API, DataFrame API, and practical experience with Machine Learning or AI frameworks and use cases is essential. Your expertise should also include data structures, algorithms, and performance optimization techniques, as well as hands-on experience with data ingestion, transformation, and processing pipelines. Given the scale of data handled in this role, you must enjoy tackling complex problems, conducting independent research, and collaborating with various teams to enhance the overall product experience. Your ability to ensure high code quality through unit testing and code reviews, along with designing and implementing data transformation, data ingestion, and ETL pipelines, will be key to your success in this position.,