Data Engineer - PySpark

As a Data Engineer - Pyspark at Barclays, you will be spearheading the evolution of the digital landscape by driving innovation and excellence. You will be responsible for harnessing cutting-edge technology to revolutionize digital offerings, ensuring unparalleled customer experiences. Working as part of a team of developers, you will deliver a technology stack, utilizing strong analytical and problem-solving skills to understand business requirements and provide quality solutions. **Key Responsibilities:** - Hands-on experience in Pyspark with a strong knowledge of Dataframes, RDD, and SparkSQL - Expertise in Pyspark performance optimization techniques - Experience in developing, testing, and maintaining applications on AWS Cloud - Proficiency in AWS Data Analytics Technology Stack including Glue, S3, Lambda, Lake formation, and Athena - Design and implementation of scalable and efficient data transformation/storage solutions with open table formats such as DELTA, Iceberg, and Hudi - Experience in using DBT (Data Build Tool) with Snowflake/Athena/Glue for ELT pipeline development - Proficient in writing advanced SQL and PL SQL programs - Building reusable components using Snowflake and AWS Tools/Technology - Worked on at least two major project implementations - Exposure to data governance or lineage tools such as Immuta and Alation - Experience with Orchestration tools like Apache Airflow or Snowflake Tasks - Knowledge of Ab-initio ETL tool is a plus **Qualifications Required:** - Ability to engage with stakeholders, elicit requirements/user stories, and translate requirements into ETL components - Understanding of infrastructure setup and providing solutions individually or in collaboration with teams - Good knowledge of Data Marts and Data Warehousing concepts - Possess good analytical and interpersonal skills - Implement Cloud-based Enterprise data warehouse with multiple data platforms along with Snowflake and NoSQL environment to build a data movement strategy This role is based out of Pune. As a Data Engineer - Pyspark at Barclays, your purpose will be to build and maintain systems that collect, store, process, and analyze data, such as data pipelines, data warehouses, and data lakes to ensure accurate, accessible, and secure data. **Additional Details:** You may be assessed on key critical skills relevant for success in the role, such as risk and controls, change and transformation, business acumen, strategic thinking, digital and technology, as well as job-specific technical skills. In this role, you will collaborate with data scientists to build and deploy machine learning models. You will also be responsible for building and maintaining data architecture pipelines, designing and implementing data warehouses and data lakes, and developing processing and analysis algorithms suitable for the intended data complexity and volumes. As a Data Engineer - Pyspark, you are expected to perform prescribed activities in a timely manner and to a high standard consistently driving continuous improvement. You will lead and supervise a team, guide and support professional development, allocate work requirements, and coordinate team resources. It is essential to demonstrate a clear set of leadership behaviors to create an environment for colleagues to thrive and deliver to a consistently excellent standard. All colleagues at Barclays are expected to demonstrate the Barclays Values of Respect, Integrity, Service, Excellence, and Stewardship, as well as the Barclays Mindset to Empower, Challenge, and Drive. As a Data Engineer - Pyspark at Barclays, you will be spearheading the evolution of the digital landscape by driving innovation and excellence. You will be responsible for harnessing cutting-edge technology to revolutionize digital offerings, ensuring unparalleled customer experiences. Working as part of a team of developers, you will deliver a technology stack, utilizing strong analytical and problem-solving skills to understand business requirements and provide quality solutions. **Key Responsibilities:** - Hands-on experience in Pyspark with a strong knowledge of Dataframes, RDD, and SparkSQL - Expertise in Pyspark performance optimization techniques - Experience in developing, testing, and maintaining applications on AWS Cloud - Proficiency in AWS Data Analytics Technology Stack including Glue, S3, Lambda, Lake formation, and Athena - Design and implementation of scalable and efficient data transformation/storage solutions with open table formats such as DELTA, Iceberg, and Hudi - Experience in using DBT (Data Build Tool) with Snowflake/Athena/Glue for ELT pipeline development - Proficient in writing advanced SQL and PL SQL programs - Building reusable components using Snowflake and AWS Tools/Technology - Worked on at least two major project implementations - Exposure to data governance or lineage tools such as Immuta and Alation - Experience with Orchestration tools like Apache Airflow or Snowflake