Lumiq - ETL Data Engineer - Python/Numpy

Job Purpose

Key Responsibilities

• Analyze existing ETL pipelines and their dependencies on Microsoft SQL Server as source systems.
• Design and implement modifications to repoint ETL extractions from PostgreSQL (CURA) while preserving the current transformations and load logic into Elasticsearch and MongoDB.
• Ensure end-to-end data integrity, quality, and freshness remain unaffected after the source switch.
• Write efficient and optimized SQL queries to extract data from the new source.
• Conduct performance testing to confirm no degradation of pipeline throughput or latency in production.
• Work closely with DevOps and platform teams to containerize, orchestrate, and deploy the updated ETLs using Docker and Kubernetes.
• Monitor post-deployment performance and handle any production issues proactively.
• Document design, code, data mappings, and operational runbooks.
  

Required Skills And Qualifications

• Strong experience building and maintaining large-scale distributed data systems.
• Expert-level proficiency in Python, especially data analysis/manipulation libraries like pandas, NumPy, and Polars.
• Advanced SQL development skills with proven experience in performance optimization.
• Working knowledge of Docker and Kubernetes.
• Familiarity with Elasticsearch and MongoDB as data stores.
• Experience working in production environments with mission-critical systems.