Lead Data Engineer - Big Data & PySpark

As a Big Data Engineer at UST, you will be responsible for designing, developing, and maintaining scalable and distributed data architectures capable of processing large volumes of data. Your role will involve implementing and optimizing data storage solutions using technologies such as Hadoop, Spark, and PySpark. You will be required to develop and implement efficient ETL processes using PySpark to extract, transform, and load large datasets, as well as optimize PySpark applications for better performance, scalability, and resource management. To excel in this role, you should have proven experience as a Big Data Engineer with a strong focus on PySpark. A deep understanding of Big Data processing frameworks and technologies is essential, along with strong proficiency in PySpark for developing and optimizing ETL processes and data transformations. Experience with distributed computing and parallel processing will be beneficial, and the ability to collaborate in a fast-paced, innovative environment is crucial. Key Responsibilities: - Design, develop, and maintain scalable and distributed data architectures capable of processing large volumes of data. - Implement and optimize data storage solutions using technologies such as Hadoop, Spark, and PySpark. - Develop and implement efficient ETL processes using PySpark to extract, transform, and load large datasets. - Optimize PySpark applications for better performance, scalability, and resource management. Qualifications: - Proven experience as a Big Data Engineer with a strong focus on PySpark. - Deep understanding of Big Data processing frameworks and technologies. - Strong proficiency in PySpark for developing and optimizing ETL processes and data transformations. - Experience with distributed computing and parallel processing. - Ability to collaborate in a fast-paced, innovative environment. Skills required: - PySpark - Big Data - Python Join UST, a global digital transformation solutions provider, and work alongside the world's best companies to make a real impact through transformation. With over 20 years of experience, UST partners with clients from design to operation, embedding innovation and agility into their organizations. Be part of a team of over 30,000 employees in 30 countries, building for boundless impact and touching billions of lives in the process.,