Hr91 logo

Data Engineer

Hr91

7 - 12 years

Chennai

Posted:15 hours ago

Job Description

Role & responsibilities:

  • Design, develop, and maintain scalable data pipelines for high-volume data movement.
  • Develop data transfer frameworks using

    Python and JDBC

    for full load and incremental load processing.
  • Handle large-scale data movement between

    Oracle, PostgreSQL, MySQL, ClickHouse

    , and other analytical platforms.
  • Develop and maintain complex ETL/ELT workflows with multiple dependencies and processing stages.
  • Implement

    hash-based CDC / checksum-based change detection

    mechanisms for identifying source data changes.
  • Develop reusable Python frameworks for data ingestion, transformation, validation, reconciliation, and monitoring.
  • Optimize JDBC-based data transfers through appropriate batch size, fetch size, parallel processing, partitioning, and connection management.
  • Design and implement incremental load, full load, historical backfill, and data synchronization processes.
  • Develop data quality checks and reconciliation mechanisms between source and target systems.
  • Implement error handling, retry mechanisms, logging, audit tracking, and recovery processes.
  • Troubleshoot complex production issues related to data pipelines, database connectivity, data mismatches, and performance.
  • Perform SQL and database performance tuning for both source and target systems.
  • Monitor pipeline execution, failures, data quality issues, and SLA adherence.
  • Automate data engineering jobs using

    Jenkins or similar scheduling/orchestration tools

    .
  • Work with application, database, DevOps, reporting, and business teams to resolve data-related issues.
  • Participate in production deployments, release activities, and performance optimization.
  • Review code and provide technical guidance to other data engineers.
  • Take ownership of assigned data engineering modules and deliver solutions independently.

Preferred candidate profile

Python

  • Strong hands-on experience in

    Python development

    .
  • Experience developing production-grade Python applications and data processing frameworks.
  • Good knowledge of

    Pandas, NumPy

    , database connectors, and API integration.
  • Experience with file processing, automation, logging, exception handling, and multiprocessing/multithreading.
  • Ability to develop reusable and maintainable Python modules.

Data Engineering

  • Strong understanding of

    ETL/ELT architecture

    .
  • Hands-on experience with high-volume data movement.
  • Experience with

    full load, incremental load, CDC, and historical data processing

    .
  • Experience implementing hash/checksum-based change detection.
  • Strong understanding of data reconciliation and data quality.
  • Ability to handle complex transformations and business rules.

Databases & SQL

  • Advanced SQL skills.
  • Strong experience with one or more of:
    • Oracle
    • PostgreSQL
    • MySQL
  • Hands-on experience with

    ClickHouse or other columnar/analytical databases

    .
  • Experience with JDBC drivers and database connectivity.
  • Good knowledge of query optimization, indexing, partitioning, and performance tuning.

Big Data & High-Volume Processing

  • Experience working with

    large datasets and high-volume data pipelines

    .
  • Understanding of distributed data processing and parallel processing concepts.
  • Experience optimizing pipelines for throughput and scalability.
  • Exposure to technologies such as

    Spark, Kafka, Hadoop, or equivalent Big Data platforms

    is preferred.

Workflow & Automation

  • Strong experience handling

    complex workflows with multiple dependencies

    .
  • Experience with tools such as:
    • Jenkins
    • Apache Airflow
    • Apache NiFi
    • Control-M
    • Other workflow orchestration tools
  • Ability to implement dependency management, retry, failure handling, and recovery mechanisms.
  • Experience automating data pipeline execution and monitoring.

Data Quality & Monitoring

  • Develop source-to-target reconciliation processes.
  • Implement record count, checksum/hash, duplicate, null, and business-rule validations.
  • Build monitoring and alerting mechanisms for pipeline failures and data quality issues.
  • Maintain audit logs and processing statistics.
  • Investigate data discrepancies and identify root causes.

Additional Skills

  • Good understanding of data warehousing and dimensional modeling.
  • Experience with data lakes and analytical platforms.
  • Knowledge of REST APIs and system integration.
  • Good Linux/Unix and shell scripting knowledge.
  • Experience with Git and CI/CD practices.
  • Exposure to Docker and cloud platforms such as AWS, Azure, or OCI is an advantage.
  • Strong

Skills Required

ETL / Elt Spark Postgresql Python Data Engineering Pandas Numpy MySQL

Learn the skills this job asks for

See your skill gap for Data Engineer →

Free courses, interview questions and practice for each skill.

Work Mode

Work from Office

Job Type

Full Time

Mock Interview

Practice Video Interview with JobPe AI

Start Python Interview
cta

Start Your Job Search Today

Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.

coding practice

Enhance Your Python Skills

Practice Python coding challenges to boost your skills

Start Practicing Python Now
Hr91 logo
Hr91

RecommendedJobs for You

Hr91 logo

Chennai

Cardinal Health logo

Data Engineer

Cardinal Health

Bengaluru

Wenger & Watson logo

Data Engineer

Wenger & Watson

Bengaluru

Hanuma Infotech logo

Data Engineer

Hanuma Infotech

India

Tibil Solutions logo

Data Engineer

Tibil Solutions

Bengaluru

CoffeeBeans logo

Data Engineer

CoffeeBeans

Bengaluru