AGNOW Solutions logo

Senior Data Engineer – Data Ingestion & Pipelines (Snowflake | Kafka | Fivetran)

AGNOW Solutions

6 years

Delhi

Posted:4 days ago

Job Description

Company Description


Role Overview

We are seeking an experienced Data Engineer to design, build, and optimize scalable data ingestion pipelines for product telemetry and enterprise business applications data. This role operates in a fast‑paced environment, rapidly prototyping data pipelines to enable data exploration, analytics, and downstream consumption.

Responsibilities

  • Design and implement modern data ingestion pipelines for business systems data using:
  • Fivetran or Similar tool
  • Python and Bulk APIs
  • Object storage and S3‑based lakehouse architectures
  • Streaming ingestion techniques leveraging Kafka, Spark, and other open‑source frameworks
  • Build end‑to‑end automated pipelines from source systems to Snowflake, ensuring reliability, scalability, and observability.
  • Apply data curation techniques (advanced SQL) in Snowflake, including transformation, standardization, and preparation of analytics‑ready datasets.
  • Optimize ingestion performance for large‑scale, high‑volume datasets (terabytes of data) using:
  • API rate‑limit handling
  • Micro‑batching strategies
  • Incremental loading and CDC (Change Data Capture) techniques
  • Implement DevOps and CI/CD practices for data pipelines, including version control, deployment automation, and environment management.
  • Partner with data analysts, SMEs, and platform teams to quickly prototype data pipelines supporting exploration and evolving requirements.
  • Monitor, troubleshoot, and optimize ingestion jobs to ensure data freshness, accuracy, and performance.

Required Skills & Experience

  • 6+ years of strong hands‑on experience in data engineering and ingestion automation in cloud or hybrid environments.
  • Proven experience ingesting business application data (e.g., CRM, ERP, financial, consumption, renewals systems).
  • Advanced proficiency with Python, Fivetran (or Similar ETL/ELT tool) for building ingestion pipelines, automation, orchestration, and data validation.
  • Expertise with streaming and distributed data processing technologies such as Kafka and Spark. Experience working with S3 or lakehouse‑style storage architectures.
  • Strong proficiency with Snowflake, including data ingestion, transformation, and performance optimization.
  • Deep understanding of ingestion optimization techniques for large, voluminous datasets, including rate limits, micro‑batching, incremental loads, and CDC.
  • Experience applying DevOps practices to data pipelines, including CI/CD and operational automation.
  • Ability to work independently in a fast‑paced, iterative environment and deliver working solutions rapidly.

Desirable Skills

  • Experience with additional open‑source ingestion frameworks and connectors
  • Exposure to event‑driven or telemetry‑heavy architectures at scale


Skills Required

Data Ingestion Kafka Servicenow Consulting AI Solutions Digital Transformation Automation Itsm Portfolio Management Service Delivery Enterprise Business Analytics

Learn the skills this job asks for

See your skill gap for Data Engineer →

Free courses, interview questions and practice for each skill.

Job Type

Full Time

Mock Interview

Practice Video Interview with JobPe AI

Start Job-Specific Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Skills

Practice coding challenges to boost your skills

Start Practicing Now
AGNOW Solutions logo
AGNOW Solutions

RecommendedJobs for You