Senior Data Engineer - Python/SQL/ETL

About The Role

We are seeking an experienced Senior Data Engineer to architect, develop, and optimize enterprise-grade data platforms and pipelines.The ideal candidate will have deep hands-on expertise in Big Data, cloud platforms, real-time data streaming, and distributed data systems, along with strong problem-solving and collaboration skills.You will be responsible for designing high-performance data workflows that support analytics, ML models, and mission-critical business applications.

Key Responsibilities

• Architect, design, and build scalable, reliable, and high-throughput ETL/ELT pipelines.
• Design and develop batch and streaming pipelines using Spark, Kafka, and cloud-native services.
• Build data ingestion frameworks for structured, semi-structured, and unstructured data.
• Develop and maintain data APIs, data orchestration jobs, and workflow automation.
• Design and manage data lakes, data warehouses, data marts, and analytical data models.
• Optimize storage formats using Parquet, ORC, Delta Lake, and compression strategies.
• Implement partitioning, bucketing, clustering, and modern data modeling approaches (Star/Snowflake, Data Vault, Lakehouse).
• Tune Spark jobs, distributed queries, and resource allocation to ensure high performance.
• Identify and resolve data bottlenecks in pipelines, clusters, and storage layers.
• Implement indexing, caching, and query optimization for analytical workloads.
• Implement data quality checks, validation rules, and automated testing frameworks.
• Ensure governance practices around metadata management, lineage, cataloging, and access control.
• Enforce data security, privacy, encryption, and compliance with organizational standards.
• Work closely with Data Scientists, BI Analysts, ML Engineers, and Product teams to align data needs.
• Translate business requirements into scalable engineering solutions.
• Mentor junior engineers and contribute to best practices across the data team.
• Implement CI/CD pipelines for data workflows using Git-based deployment strategies.
• Automate cluster provisioning, job scheduling, and monitoring using tools like Airflow, Dagster, Prefect, or Oozie.
• Establish logging, monitoring, and alerting for data pipelines.
  

Required Skills & Technical Expertise

Programming & Core Skills :

• Strong proficiency in Python, Scala, or Java (minimum one is mandatory; experience in more is a plus).
• Expertise in building distributed data processing applications.
• Deep understanding and hands-on experience with :
• Apache Spark (Structured Streaming, Spark SQL, PySpark/Scala)
• Hadoop HDFS, YARN, Hive, HBase
• Kafka (event streaming, consumer groups, partitions, schema registry)
• Airflow / Dagster / Prefect for orchestration
• Hands-on experience with at least one cloud platform : AWS (preferred) / Azure / GCP
• Working knowledge of cloud-native services such as :
• AWS : S3, EMR, Glue, Athena, Redshift, Kinesis, Lambda
• Azure : Synapse, ADLS, Data Factory
• GCP : BigQuery, Dataflow, Dataproc, Pub/Sub
• Experience with Databricks / EMR / Snowflake is a strong advantage.
• Expertise in SQL ability to write complex, optimized queries for large datasets.
• Dimensional modeling
• Lakehouse architecture
• Modern table formats (Delta Lake, Iceberg, Hudi)
• Experience with distributed data systems, cluster management, and storage strategies.
• Understanding of containerization (Docker, Kubernetes) for data workloads.
• Proficiency with Git, branching strategies, and repo management.
• CI/CD exposure with GitHub Actions, Jenkins, GitLab CI, etc.
• Experience with data governance tools like AWS Glue Data Catalog, Apache Atlas, or Collibra.
• Ability to define SLAs, SLIs, and SLOs for data reliability
  

(ref:hirist.tech)