Job
Description
We are looking for a highly skilled and experienced Data Engineering Lead to join our team. The ideal candidate will have 5-10 years of experience in designing and implementing scalable data lake architecture and data pipelines. ### Roles and Responsibility Design and implement scalable data lake architectures using Azure Data Lake services. Develop and maintain data pipelines to ingest data from various sources. Optimize data storage and retrieval processes for efficiency and performance. Ensure data security and compliance with industry standards. Collaborate with data scientists and analysts to facilitate data accessibility. Monitor and troubleshoot data pipeline issues to ensure reliability. Document data lake designs, processes, and best practices. Experience with SQL and NoSQL databases, as well as familiarity with big data file formats like Parquet and Avro. Must have skills: Azure Data Lake, Azure Synapse Analytics, Azure Data Factory, Azure DataBricks, Python (PySpark, Numpy etc), SQL, ETL, Data warehousing, Azure Devops, Experience in developing streaming pipeline using Azure Event Hub, Azure Stream analytics, Spark streaming, and integration with business intelligence tools such as Power BI. Good to have skills: Big Data technologies (e.g., Hadoop, Spark), Data security. General Skills: Experience with Agile and DevOps methodologies and the software development lifecycle, proactive and responsible for deliverables, escalates dependencies and risks, works with most DevOps tools, limited supervision, completes assigned tasks on time and provides regular status reports, trains new team members, and builds strong relationships with project stakeholders. ### Job Requirements Minimum 5 years of experience in designing and implementing scalable data lake architecture and data pipelines. Strong knowledge of Azure Data Lake, Azure Synapse Analytics, Azure Data Factory, Azure DataBricks, Python (PySpark, Numpy etc), SQL, ETL, Data warehousing, and Azure Devops. Experience in developing streaming pipelines using Azure Event Hub, Azure Stream analytics, and Spark streaming. Familiarity with big data file formats like Parquet and Avro. Ability to work with multi-cultural global teams and virtually. Knowledge of cloud solutions such as Azure or AWS with DevOps/Cloud certifications is desired. Proactive and responsible for deliverables. Escalates dependencies and risks. Works with most DevOps tools, limited supervision. Completes assigned tasks on time and provides regular status reports. Trains new team members and builds strong relationships with project stakeholders.
