Lead Consultant - Sr.Data Engineer (DBT+Snowflake)

Sr.Data

Responsibilities

• Develop, implement, and optimize data pipelines using Snowflake, with a focus on Cortex AI capabilities.

• Extract, transform, and load (ETL) data from various sources into Snowflake, ensuring data integrity and accuracy.

• Implement Conversational AI solutions using Snowflake Cortex AI to facilitate data interaction through ChatBot agents.

• Collaborate with data scientists and AI developers to integrate predictive analytics and AI models into data workflows.

• Monitor and troubleshoot data pipelines to resolve data discrepancies and optimize performance.

• Utilize Snowflake%27s advanced features, including Snowpark, Streams, and Tasks, to enable data processing and analysis.

• Develop and maintain data documentation, best practices, and data governance protocols.

• Ensure data security, privacy, and compliance with organizational and regulatory guidelines.

• Bachelor%27s degree in computer science, Data Engineering, or a related discipline.

• Experience in data engineering, including a minimum and working extensively with Snowflake.

• Demonstrated expertise with Snowflake Cortex AI, particularly in data extraction, chatbot development, and Conversational AI solutions.

• Strong skills in SQL, Python programming, and data modelling techniques.

• Hands-on experience with data integration platforms such as Matillion, Talend, or Informatica.

• Familiarity with cloud environments including AWS, Azure, or Google Cloud Platform.

• Excellent problem-solving abilities, with a focus on ensuring data quality and optimizing system performance.

• Effective communicator with proven ability to collaborate within cross-functional teams.

• Proficient in utilizing DBT's testing and documentation features to guarantee the accuracy and reliability of data transformations.

• Solid understanding of data lineage and metadata management, leveraging DBT's lineage tracking capabilities to document data workflows.

• Knowledge of software engineering best practices applied to DBT development, including version control, code reviews, and automated testing.

• Experience designing and building data ingestion pipelines.

• Skilled in using Snowflake utilities such as SnowSQL, Snowpipe, bulk copy, Snowpark, Tasks, Streams, Time Travel, Cloning, Optimizer, Metadata Manager, data sharing, stored procedures, UDFs, and Snowsight.

• Proven expertise in implementing Change Data Capture (CDC) or Slowly Changing Dimensions (SCD) Type 2.

• Experience with Airflow or similar workflow orchestration tools for scheduling and managing ETL processes.

• Familiarity with repository management tools like GitHub, GitLab, or Azure Repos is a plus.

Qualifications we seek in you!

Minimum qualifications

• B.E./ Masters in Computer Science, Information technology, or Computer engineering or any equivalent degree with good IT experience and relevant of working experience as a Sr. Data Engineer with DBT+Snowflake skillsets

• Skill Matrix:

• DBT (Core or Cloud), Snowflake, AWS/Azure, SQL, ETL concepts, Airflow or any orchestration tools, Data Warehousing concepts