Data Engineer Lead

Job Description

Position Overview: We are looking for a skilled and visionary Data Engineering Lead to join our growing team. In this role, you will be responsible for leading a team of data engineers in designing, developing, and maintaining robust data pipelines and infrastructure. You will work closely with cross-functional teams to support data-driven decision-making and ensure the availability, quality, and integrity of our data assets. Role & responsibilities: Build, develop, and maintain efficient and high-performance data pipelines across both cloud and on-premises environments. Ensure the accuracy, adequacy, and legitimacy of data. Prepare ETL pipelines to extract data from various sources and store it in a centralized location. • Analyse, interpret, and present results through effective visualization and reports. Identify critical metrics. Implement and instill best practices for effective data management. Monitor the use of data systems and ensure the correctness, completeness, and availability of data services. Optimize data infrastructure and processes for cost efficiency on AWS cloud and on-premises environments. Utilize Apache Airflow, NiFi, or equivalent tools to build and manage data workflows and integrations. Implement best practices for data governance, security, and compliance. Monitor and troubleshoot data pipeline issues to ensure timely resolution. Stay current with industry trends and emerging technologies in data engineering and cloud computing. Lead and mentor a team of data engineers, fostering a culture of collaboration, innovation, and continuous improvement. Define team objectives and performance metrics, conducting regular performance evaluations and providing constructive feedback. Facilitate knowledge sharing and professional development within the team. Preferred candidate profile: Up to 6-9 years of proven experience as a Data Engineering Lead or in a similar role. Extensive hands-on experience with ETL and ELT processes. Strong expertise in data integrity and quality assurance. Proficiency in optimizing AWS cloud services and on-premises infrastructure for cost and performance. Hands-on experience with Apache Airflow and NiFi. Strong programming skills in languages such as Python, Java, or Scala. Experience with SQL and NoSQL databases. Experience in building and maintaining a single source of truth. Familiarity with data warehousing solutions like Amazon Redshift, Snowflake, or BigQuery. Strong problem-solving skills and the ability to work under pressure. Hands-on experience with data visualization tools such as Tableau, Power BI, or Looker. Experience in financial services is a must. Skills Required: Team leading and team management. Strong analytical and problem-solving abilities. Excellent communication and interpersonal skills. Ability to work collaboratively in a fast-paced environment and manage multiple priorities. Educational Qualifications: Bachelor's degree in Computer Science, Information Technology, Data Science, or a related field.