Senior Lead Engineer (Python & Spark)

As a Senior Lead Engineer specializing in Python and Spark within the AWS environment, you will have the crucial responsibility of designing, building, and maintaining robust, scalable, and efficient ETL pipelines. Your primary focus will be on ensuring alignment with data lakehouse architecture on AWS and optimizing workflows using AWS services such as Glue, Glue Data Catalog, Lambda, and S3. Your expertise will play a key role in implementing data quality and governance frameworks to maintain reliable and consistent data processing across the platform. Collaboration with cross-functional teams to gather requirements, provide technical insights, and deliver high-quality data solutions will be essential in your role. You will drive the migration of existing data processing workflows to the lakehouse architecture by leveraging Iceberg capabilities and establishing best practices for coding standards, design patterns, and system architecture. Your leadership will extend to technical discussions, mentoring team members, and fostering a culture of continuous learning and innovation. Ensuring that all solutions are secure, compliant, and meet company and industry standards will be a top priority. Key relationships in this role will include interactions with Senior Management and Architectural Group, Development Managers, Team Leads, Data Engineers, Analysts, and Agile team members. Your extensive expertise in Python and Spark, along with strong experience in AWS services, data quality and governance, and scalable architecture, will be crucial for success in this position. Desired skills include familiarity with additional programming languages such as Java, experience with serverless computing paradigms, and knowledge of data visualization or reporting tools for effective stakeholder communication. Certification in AWS or data engineering would be beneficial. A bachelor's degree in Computer Science, Software Engineering, or a related field is helpful for this role, although equivalent professional experience or certifications will also be considered. Joining our team at LSEG means being part of a global financial markets infrastructure and data provider dedicated to driving financial stability, empowering economies, and enabling sustainable growth. Our culture, built on values of Integrity, Partnership, Excellence, and Change, guides our decision-making and actions daily. We value individuality, encourage new ideas, and are committed to sustainability across our global business. You will have the opportunity to contribute to re-engineering the financial ecosystem to support sustainable economic growth and the just transition to net zero. LSEG offers a range of tailored benefits and support, including healthcare, retirement planning, paid volunteering days, and wellbeing initiatives to ensure a collaborative and inclusive work environment. Please take a moment to review our privacy notice, which outlines how personal information is handled by London Stock Exchange Group (LSEG) and your rights as a data subject. If you are representing a Recruitment Agency Partner, it is essential to ensure that candidates applying to LSEG are aware of this privacy notice.,