Lead Data Engineer

Job Description

Role & responsibilities: Lead Data Engineer will be responsible for designing, implementing, and maintaining data solutions on the Microsoft Azure Data platform (Databricks,Azure Data Factory, Python) Leading a team of data engineers, collaborating with various stakeholders, and ensuring the efficient processing, storage, and retrieval of large volumes of data Maintaining and enhancing the data pipelines necessary to support business operations, analytics workloads, reporting needs and insights the impact the strategic and tactical implementation for the organization. Should be able to consume business and functional requirements and translation them into technical implementations. This role will also be responsible for providing technical expertise and direction to their team to ensure quality, standards, processes, and methodologies are being followed and work through technical challenges the team may face. Technical Expertise and Responsibilities : Design, build, and maintain scalable and reliable data pipelines. Should be able to design and build solutions in Azure data factory and Databricks to extract, transform and load data into different source and target systems. Should be able to analyze and understand the existing data landscape and provide recommendations/innovative ideas for rearchitecting / optimizing / streamlining to bring efficiency and scalability. Must be able to collaborate and effectively communicate with onshore counterparts to address technical gaps, requirement challenges, and other complex scenarios. Monitor and troubleshoot data systems to ensure high performance and reliability. Should be highly analytical and detail-oriented with extensive familiarity with database management principles. Optimize data processes for speed and efficiency. Ensure the data architecture supports business requirements and data governance policies. Define and execute the data engineering strategy in alignment with the companys goals. Integrate data from various sources, ensuring data quality and consistency. Stay updated with emerging technologies and industry trends. Understand the big picture business process utilizing deep knowledge in banking industry and translate them to data requirements. Enabling and running data migrations across different databases and different servers Perform thorough testing and validation to support the accuracy of data transformations and data verification used in machine learning models. Analyze data and different systems to define data requirements. Should be well versed with Data Structures & algorithms. Define data mapping working along with business and digital team and data team. Data pipeline maintenance/testing/performance validation Assemble large, complex data sets that meet functional / non-functional business requirements. Analyze and identify gaps on data needs and work with business and IT to bring in alignment on data needs. Understand impact of data conversions as they pertain to servicing operations.o Manage higher volume and more complex cases with accuracy and efficiency. Leadership Responsibilities : Provide leadership and guidance to the data engineering team, including mentoring, coaching, and fostering a collaborative work environment. Support team members with troubleshooting and resolving complex technical issues and challenges. Provide technical expertise and direction in data engineering, guiding the team in selecting appropriate tools, technologies, and methodologies. Stay updated with the latest advancements in data engineering and ensure the team follows best practices and industry standards. Align coding standards, conduct code reviews to ensure proper code quality level. Identify and introduce quality assurance processes for data pipelines and workflows. Stay up to date on the latest process and IT advancements to automate and modernize data systems for storage for performance, efficiency and cost savings with a cloud centric focus. Act as main point of contact to other teams/contributors. Work with product and business teams to understand and translate business requirements in to technical solutions. Implement proper security measures and procedures to prevent unauthorized entry and/or data loss. Promotes honest and open communication throughout the credit union. Demonstrate behaviors that are consistent with the credit unions values, philosophies, and leadership characteristics. Perform other duties as assigned. Foster a culture of collaboration, innovation, and continuous improvement within the team. Optimize data flow and collection for cross-functional teams. Work closely with Data counterparts at onshore, product owners, and business stakeholders to understand data needs and strategies. Collaborate with IT and DevOps teams to ensure data infrastructure aligns with overall IT architecture. Drive continuous improvement initiatives within the data engineering function. Required Skills: 5+ years of experience supporting big data platforms (cloud) Exceptional analytical and conceptual thinking skills coupled with strong written and verbal communication skills and ability to solve problems quickly. Strong technical skills in data engineering, including proficiency in programming languages and technologies such (Databricks,Azure Data Factory, Python) Should be proficient in SQL and Query optimization. Technical expertise understanding database design development, data models, and techniques for data mining. Expertise in working with various data tools and technologies, such as ETL frameworks, data pipelines, and data warehousing solutions. Preferred Cloud computing experience with preference for Microsoft Azure cloud technologies. Proven experience in leading a team of data engineers, providing guidance, mentorship, and technical support. In-depth knowledge of data management principles and best practices, including data governance, data quality, and data integration. Excellent interpersonal skills, with the ability to effectively collaborate with cross-functional teams, stakeholders, and senior management.• Drive automation efforts across the data analytics team utilizing Infrastructure as Code (IaC) using Terraform, Configuration Management, and Continuous Integration (CI) / Continuous Delivery (CD) tools such as Jenkins. Experience designing for data engineering models. Able to communicate effectively, both written and orally, with internal and external customers, vendors, and partners. Thrives in ambiguous technology situations. Ability to solve problems efficiently, think analytically, creatively, and logically. Ability to interact effectively with co-workers to accomplish daily tasks and to resolve interpersonal conflicts and miscommunications. Must be able to be bonded. Ability to maintain a high level of confidentiality. Should have experience as a technical lead mentoring data engineer and driving data-oriented projects. Should have worked in onshore offshore model managing challenging scenarios. Lead data architecture and implementation efforts to structure, integrate, govern, store, describe, model, and maintain data warehouse for optimal accuracy and usage. Ability to work with ambiguity and vague requirements and transform them into deliverables. Help build define architecture frameworks, best practices & processes. Collaborate on Data warehouse architecture and technical design discussions. Should have expertise in data life cycle, data ingestion, transformation, data loading, validation, and performance tuning