Data Engineer

Website-https://intraedge.com/

Job Title: Data Engineer

Experience: 5+Years

Location: [Remote]

Senior Data Engineer

The Senior Data Engineer will design, develop, monitor and maintain a robust and scalable data platform used by other data analyst and engineering teams to deliver powerful insights to both internal and external stakeholders

Responsibilities

● Design, build and maintain data infrastructure that powers both batch and realtime processing of billions of records a day.

● Improve the data quality and reliability of data pipelines through monitoring, validation and failure detection

● Design, build and maintain a central data cataloging system to ease integration and discovery of datasets

● Develop data pipelines that provide fast, optimized, and robust end-to-end analytical solutions

● Automate manual processes and create a platform in favor of self-service data consumption

● Deploy and configure components to production environments.

● Participate in on-call schedule to provide emergency incident support

● Mentor and train teammates on design and operation of data platform

● Stay current with industry trends, making recommendations as needed to help the company excel

● Other job-related duties as assigned Skills,

Experience and Qualifications

● Bachelor’s Degree in Computer Science or Engineering a plus

● 5+ years of relevant industry experience in Data Engineering working with large scale data driven systems

● Deep knowledge of dimensional modeling and designing schemas and data sets optimized for OLAP environment

● Experience fine-tuning queries around large, complex data sets

● Extensive experience working with an MPP data platform (ie Snowflake, BigQuery, or Databricks)

● Experience designing and implementing workflows on a modern orchestration framework (ie Airflow, Prefect)

● Advanced Data Build Tool (DBT) experience (ie macro design, generalizing tests, using different incremental strategies)

● Deep understanding of SQL and data warehouse systems, especially Snowflake (SnowPro certification preferred)

● Experience working with and administering a BI tool (Tableau preferred)

● Expertise in object-oriented and/or functional programming languages (Python preferred)

● Strong programming skills, able to write modular, maintainable code

● Understanding of DevOps principles such as automating of CI/CD pipelines and Infrastructure as code

● Understanding of polyglot data persistence (relational, key/value, document, column)

● Excellent problem-solving skills and the ability to proactively solve issues

● Excellent communication and organizational skills and proven ability to complete tasks and meet deadlines

● Ability to be flexible with working in tandem with a team of engineers or alone, as required