Senior Data Engineer

Senior Data Engineer

Responsibilities

• Design, develop, and maintain robust, scalable, and efficient data pipelines using Apache Airflow and DBT.
• Experience with real-time or streaming data pipelines.
• Knowledge of data modeling, ETL/ELT best practices and Identifying and addressing performance bottlenecks, and optimizing code and systems for improved speed and efficiency.
• Provides technical leadership and mentorship for other team members. Reviews and critiques designs and implementations
• Writing high-quality, efficient, and maintainable code, often with a focus on solving complex technical challenges.
• Facilitates collaboration of working team and often engages with stakeholders and product owners.
• Maintaining comprehensive technical documentation, including system architecture, code comments, and documentation to assist in understanding and maintaining the software.
• Effectively communicating technical concepts and project status to non-technical stakeholders.

Experience and Knowledge

• Degree in Computer Science, Information Systems, or equivalent experience
• 5+ years of related professional experience
• Advanced knowledge in multiple areas of
• data transformations, data pipelines, workflow automation, and scheduling systems
• data-centric systems architecture
• database systems, data warehouses, distributed file storage and compute platforms

• Experience with several of the technologies we currently use:
• Cloud Providers:

  AWS

  , Azure
• Data Pipelines: Apache Airflow, DBT
• Data Tools:

  Pyspark

  ,

  Snowpark

  , AWS Glue, Pandas, Databricks
• Databases:

  Snowflake

  , SqlServer, Postgresql
• Containerization: Kubernetes, Docker
• Languages:

  Python

  , Bash, SQL
• CI/CD:

  Terraform,

  GitLab, Azure DevOps
• SCM: Git, Github, GitLab