Lead Engineer

As a Senior Lead Engineer specializing in Python and Spark in AWS, you will be responsible for designing, building, and maintaining robust, scalable, and efficient ETL pipelines. Your primary focus will be on ensuring alignment with data lakehouse architecture on AWS and optimizing workflows using services such as Glue, Lambda, and S3. Collaborating with cross-functional teams, you will gather requirements, provide technical insights, and deliver high-quality data solutions. Your role will involve driving the migration of existing data processing workflows to the lakehouse architecture, leveraging Iceberg capabilities, and enforcing best practices for coding standards and system architecture. You will play a key role in implementing data quality and governance frameworks to ensure reliable and consistent data processing across the platform. Monitoring and improving system performance, optimizing data workflows, and ensuring all solutions are secure, compliant, and meet industry standards will be crucial aspects of your responsibilities. Leading technical discussions, mentoring team members, and fostering a culture of continuous learning and innovation are essential for this role. You will also maintain relationships with senior management, architectural groups, development managers, team leads, data engineers, analysts, and agile team members. Key Skills and Experience: - Extensive expertise in Python and Spark for designing and implementing complex data processing workflows. - Strong experience with AWS services such as Glue, Lambda, S3, and EMR, focusing on data lakehouse solutions. - Deep understanding of data quality frameworks, data contracts, and governance processes. - Ability to design and implement scalable, maintainable, and secure architectures using modern data technologies. - Hands-on experience with Apache Iceberg and its integration within data lakehouse environments. - Expertise in problem-solving, performance optimization, and Agile methodologies. - Excellent interpersonal skills with the ability to communicate complex technical solutions effectively. Desired Skills and Experience: - Familiarity with additional programming languages such as Java. - Experience with serverless computing paradigms. - Knowledge of data visualization or reporting tools for stakeholder communication. - Certification in AWS or data engineering (e.g., AWS Certified Data Analytics, Certified Spark Developer). Education and Certifications: - A bachelor's degree in Computer Science, Software Engineering, or a related field is helpful. - Equivalent professional experience or certifications will also be considered. Join us at LSEG, a leading global financial markets infrastructure and data provider, where you will be part of a dynamic organization across 65 countries. We value individuality, encourage new ideas, and are committed to sustainability, driving sustainable economic growth and inclusivity. Experience the critical role we play in re-engineering the financial ecosystem and creating economic opportunities while accelerating the transition to net zero. At LSEG, we offer tailored benefits including healthcare, retirement planning, paid volunteering days, and wellbeing initiatives.,