Data Engineering Lead

As a Data Engineering Lead at EY, you will collaborate with the Data Architect to design and implement scalable data lake architecture and data pipelines. Your responsibilities will include designing and implementing scalable data lake architectures using Azure Data Lake services, developing and maintaining data pipelines to ingest data from various sources, optimizing data storage and retrieval processes for efficiency and performance, ensuring data security and compliance with industry standards, collaborating with data scientists and analysts to facilitate data accessibility, monitoring and troubleshooting data pipeline issues to ensure reliability, and documenting data lake designs, processes, and best practices. You will also need experience with SQL and NoSQL databases, as well as familiarity with big data file formats like Parquet and Avro. **Key Responsibilities:** - Design and implement scalable data lake architectures using Azure Data Lake services. - Develop and maintain data pipelines to ingest data from various sources. - Optimize data storage and retrieval processes for efficiency and performance. - Ensure data security and compliance with industry standards. - Collaborate with data scientists and analysts to facilitate data accessibility. - Monitor and troubleshoot data pipeline issues to ensure reliability. - Document data lake designs, processes, and best practices. - Experience with SQL and NoSQL databases, as well as familiarity with big data file formats like Parquet and Avro. **Qualifications Required:** - Azure Data Lake - Azure Synapse Analytics - Azure Data Factory - Azure DataBricks - Python (PySpark, Numpy etc) - SQL - ETL - Data warehousing - Azure Devops - Experience in developing streaming pipeline using Azure Event Hub, Azure Stream analytics, Spark streaming - Experience in integration with business intelligence tools such as Power BI In addition to the technical skills mentioned, it is essential to have experience with Agile and DevOps methodologies and the software development lifecycle, be proactive and responsible for deliverables, escalate dependencies and risks, work with most DevOps tools with limited supervision, complete assigned tasks on time and provide regular status reporting. You should also be able to train new team members, have knowledge of cloud solutions such as Azure or AWS with DevOps/Cloud certifications, work with multi-cultural global teams virtually, and build strong relationships with project stakeholders. Join EY to build a better working world, where diverse teams across over 150 countries provide trust through assurance and help clients grow, transform, and operate. EY exists to create long-term value for clients, people, and society, and to build trust in the capital markets by asking better questions to find new answers for the complex issues facing our world today.,