Big Data Engineer Lead

Job Description

Required Qualifications: Undergraduate degree or equivalent experience 5+ years of work experience on Big Data skills 5+ years of experience managing the team 5+ years of work experience on people management skills 3+ years of work experience on Azure Cloud skills Experience or knowledge Azure Cloud, Databricks, Terraform, CI/CD, Spark, Scala, Java, Hbase, Hive, Sqoop, GitHub, Jenkins, Elastic Search, Grafana, UNIX, SQL, OpenShift, Kubernetes and Oozie etc. Solid technical knowledge and work experience on Big Data skills and Azure Cloud skills Primary Responsibilities: Designing and developing large-scale data processing systems. Use the expertise in big data technologies to ensure that the systems are efficient, scalable, and secure Ensuring that the developed systems are running smoothly. Monitor system performance, diagnose and troubleshoot issues, and make necessary changes to optimize system performance Processing, cleaning, and integrating large data sets from various sources to ensure that the data is accurate, complete, and consistent Work closely with cross-functional teams, including data scientists, analysts, and business stakeholders. Collaborate with these teams to ensure that the systems they develop meet the organizations requirements and can support its goals Collaborating closely with senior stakeholders to understand business requirements and effectively translate them into technical requirements for the development team Planning and documenting comprehensive technical specifications for features or system design, ensuring a clear roadmap for development and implementation Designing, building, and configuring applications to meet business process and application requirements, leveraging your technical expertise and problem-solving skills Directing the development team in all aspects of the software development life cycle, including design, development, coding, testing, and debugging, to deliver high-quality solutions Writing testable, scalable, and efficient code, leading by example, and setting coding standards for the team Conducting code reviews and providing constructive feedback to ensure code quality and adherence to best practices Mentoring and guiding junior team members, fostering their professional growth, and encouraging the adoption of industry best practices Ensuring that software quality standards are met by enforcing code standards, conducting rigorous testing, and implementing continuous improvement processes Staying updated with the latest technologies and industry trends, continuously enhancing technical skills, and driving innovation within the development team Set and communicate team priorities that support the broader organization's goals. Align strategy, processes, and decision-making across teams Set clear expectations with individuals based on their level and role and aligned to the broader organization's goals. Meet regularly with individuals to discuss performance and development and provide feedback and coaching Develop the long-term technical vision and roadmap within, and often beyond, the scope of your teams. Evolve the roadmap to meet anticipated future requirements and infrastructure needs. Identify, navigate, and overcome technical and organizational barriers that may stand in the way of delivery Constantly improve the processes and practices around development and delivery Always think customer first, including striving to outperform their expectations Effectively work with Product Managers, Program Managers and other stakeholders to ensure the customer is benefiting from the work Foster and facilitate Agile methodologies globally and work in an agile environment using SCRUM or Kanban Work with Program Managers/leads to consume product backlog and generate technical design Leading by example on design and development of platform features Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so