Data Engineer

Job Title : Data Engineer

Experience : 4-9 Years

Location : Noida, Chennai & Pune

Skills : Python, Pyspark, Snowflake & Redshift

Key Responsibilities

• Migration & Modernization

• Lead the migration of data pipelines, models, and workloads from Redshift to Snowflake/Yellowbrick.

• Design and implement landing, staging, and curated data zones to support scalable ingestion and consumption patterns.

• Evaluate and recommend tools and frameworks for migration, including file formats, ingestion tools, and orchestration.

• Design and build robust ETL/ELT pipelines using Python, PySpark, SQL, and orchestration tools (e.g., Airflow, dbt).

• Support both batch and streaming pipelines, with real-time processing via Kafka, Snowpipe, or Spark Structured Streaming.

• Build modular, reusable, and testable pipeline components that handle high volume and ensure data integrity.

• Define and implement data modeling strategies (star, snowflake, normalization/denormalization) for analytics and BI layers.

• Implement strategies for data versioning, late-arriving data, and slowly changing dimensions.

• Implement automated data validation and anomaly detection (using tools like dbt tests, Great Expectations, or custom checks).

• Build logging and alerting into pipelines to monitor SLA adherence, data freshness, and pipeline health.

• Contribute to data governance initiatives including metadata tracking, data lineage, and access control.

Required Skills & Experience

• 10+ years in data engineering roles with increasing responsibility.

• Proven experience leading data migration or re-platforming projects.

• Strong command of Python, SQL, and PySpark for data pipeline development.

• Hands-on experience with modern data platforms like Snowflake, Redshift, Yellowbrick, or BigQuery.

• Proficient in building streaming pipelines with tools like Kafka, Flink, or Snowpipe.

• Deep understanding of data modeling, partitioning, indexing, and query optimization.

• Expertise with ETL orchestration tools (e.g., Apache Airflow, Prefect, Dagster, or dbt).

• Comfortable working with large datasets and solving performance bottlenecks.

• Experience in designing data validation frameworks and implementing DQ rules.