Lead Data Engineer

About the Role:

As a Senior Lead Data Engineer, you will play a crucial role in designing, building, and optimizing our data pipeline and infrastructure. You will work closely with stakeholders, analysts, and other to ensure data integrity, availability, and security. This position requires a deep understanding of data engineering principles, advanced technical skills, and the ability to build data pipelines & ingestion workflows.

What You’ll Do:

• Design, develop, and maintain robust data pipelines and ETL processes to collect, process, and store large volumes of data from various sources.
• Experience in building and maintaining the existing data acquisition scripts using Go Language.
• Collaborate with stakeholders/product owners and analysts to understand data requirements and translate them into scalable solutions.
• Optimize and improve existing data solutions for better performance and scalability.
• Ensure data consistency, security and accuracy through data validation and cleansing techniques.
• Implement and maintain data warehouse solutions, including data modelling, schema design, and query optimization.
• Develop and maintain data integration solutions to support business intelligence and analytics needs.
• Monitor and troubleshoot data pipeline issues, ensuring timely resolution.
• Stay updated with the latest trends and technologies in data engineering and recommend improvements.
• Configure and monitor continuous integration/continuous deployment (CI/CD) pipelines and tools (e.g., Git, Jenkins).
• Mentor and provide technical guidance to junior data engineers.
• Proven ability to work independently and with a team.
• Document data processes, workflows, and best practices.

What You’ll Need:

• Bachelor’s or master’s degree in computer science, Engineering, or a related field.
• 10+ years of experience in data engineering, data warehousing, or a related field.
• Strong Hands-on experience on advanced Snowflake and PostgreSQL.
• Experienced at handling Data Warehouse technologies like Snowflake.
• Creating Integration with AWS S3.
• Configuring Data ingestion pipelines.
• Proficiency in programming languages such as Python, Go Language.
• Experience with cloud platforms (e.g., AWS, Azure) and their data services.
• Strong experience on data integration techniques, data modelling and ETL processes.
• Familiarity with data visualization tools (e.g., Tableau, Power BI) is a plus.
• Experience with real-time data processing and stream processing frameworks.
• Experience in configuring CI/CD pipelines and version control like GIT.
• Knowledge of containerization and orchestration tools.
• Knowledge of NLP and machine learning concepts.
• Understanding of data governance and compliance regulations.
• Strong problem-solving skills and the ability to work independently or as part of a team.
• Adaptability and a willingness to learn new technologies and techniques.
• Excellent communication and interpersonal skills.