Job
Description
Role Overview: At Markovate, we are looking for a highly experienced and innovative Senior Data Engineer who can design, build, and optimize robust, scalable, and production-ready data pipelines across both AWS and Azure platforms. As a Senior Data Engineer, you will be responsible for developing hybrid ETL/ELT pipelines, processing files from AWS S3 and Azure Data Lake Gen2, implementing event-based orchestration, creating scalable ingestion workflows, integrating with metadata and lineage tools, and collaborating with cross-functional teams to align solutions with AI/ML use cases. Key Responsibilities: - Design and develop hybrid ETL/ELT pipelines using AWS Glue and Azure Data Factory (ADF). - Process files from AWS S3 and Azure Data Lake Gen2, including schema validation and data profiling. - Implement event-based orchestration using AWS Step Functions and Apache Airflow (Astronomer). - Develop and maintain bronze silver gold data layers using DBT or Coalesce. - Create scalable ingestion workflows using Airbyte, AWS Transfer Family, and Rivery. - Integrate with metadata and lineage tools like Unity Catalog and Open Metadata. - Build reusable components for schema enforcement, EDA, and alerting (e.g., MS Teams). - Work closely with QA teams to integrate test automation and ensure data quality. - Collaborate with cross-functional teams including data scientists and business stakeholders. - Document architectures, pipelines, and workflows for internal stakeholders. Qualifications Required: - 9+ years of experience in data engineering and data architecture. - Excellent communication and interpersonal skills. - Strong problem-solving, decision-making, and conflict-resolution abilities. - Proven ability to work independently and lead cross-functional teams. - Ability to work in a fast-paced, dynamic environment with discretion and professionalism. - Strong work ethics, trustworthiness, and commitment to collaboration. - Experience with cloud platforms like AWS (Glue, Step Functions, Lambda, S3, CloudWatch) and Azure (ADF, ADLS Gen2). - Proficient in transformation and ELT tools such as Databricks (PySpark), DBT, Coalesce, and Python. - Familiarity with data modeling techniques including CEDM, Data Vault 2.0, and Dimensional Modeling. - Hands-on experience with orchestration tools like AWS Step Functions, Airflow, and ADF Triggers. - Expertise in monitoring and logging using CloudWatch, AWS Glue Metrics, MS Teams Alerts, and Azure Data Explorer. - Proficiency in version control and CI/CD using GitHub, Azure DevOps, CloudFormation, Terraform, and ARM templates. Additional Company Details (if any): At Markovate, collaboration, innovation, continuous learning, transparent communication, and flexible work arrangements are key aspects of our culture. We value every voice, invest in our team's learning, embrace challenges as opportunities, and recognize individual achievements. Markovate is committed to sustainable practices, positive community impact, and prioritizes the growth and well-being of our employees. (Note: The additional company details section has been included as part of the job description to provide insights into the company culture and values.) Role Overview: At Markovate, we are looking for a highly experienced and innovative Senior Data Engineer who can design, build, and optimize robust, scalable, and production-ready data pipelines across both AWS and Azure platforms. As a Senior Data Engineer, you will be responsible for developing hybrid ETL/ELT pipelines, processing files from AWS S3 and Azure Data Lake Gen2, implementing event-based orchestration, creating scalable ingestion workflows, integrating with metadata and lineage tools, and collaborating with cross-functional teams to align solutions with AI/ML use cases. Key Responsibilities: - Design and develop hybrid ETL/ELT pipelines using AWS Glue and Azure Data Factory (ADF). - Process files from AWS S3 and Azure Data Lake Gen2, including schema validation and data profiling. - Implement event-based orchestration using AWS Step Functions and Apache Airflow (Astronomer). - Develop and maintain bronze silver gold data layers using DBT or Coalesce. - Create scalable ingestion workflows using Airbyte, AWS Transfer Family, and Rivery. - Integrate with metadata and lineage tools like Unity Catalog and Open Metadata. - Build reusable components for schema enforcement, EDA, and alerting (e.g., MS Teams). - Work closely with QA teams to integrate test automation and ensure data quality. - Collaborate with cross-functional teams including data scientists and business stakeholders. - Document architectures, pipelines, and workflows for internal stakeholders. Qualifications Required: - 9+ years of experience in data engineering and data architecture. - Excellent communication and interpersonal skills. - Strong problem-solving, decision-making, and conflict-resolution abilities. - Proven ability to work independently and lead cross-functiona
