Senior Python Data Engineer

Role & responsibilities

Key Responsibilities

• Design, build and maintain Python plugins that encapsulate business rules and transformation logic such as custom aggregation/disaggregation, rule-based data enrichment, and time-grain manipulations.
  
• Implement data-cleaning, validation, mapping and merging routines using Pandas, NumPy and other Python libraries to prepare inputs for downstream analytics.
  
• Define clear interfaces and configuration schemes for plugins (for example, picklists for output grains and rule codes) and package them for easy consumption by implementation teams.
  
• Profile and optimize Python code pathsleveraging vectorized operations, efficient merges and in-memory transformationsto handle large datasets with low latency.
  
• Establish and enforce coding standards, write comprehensive unit and integration tests with pytest (or similar), and ensure high coverage for all new components.
  
• Triage and resolve issues in legacy scripts, refactor complex routines for readability and extensibility, and manage versioning across releases.
  

Collaborate with data engineers and analysts to capture requirements, document plugin behaviors, configuration parameters, and usage examples in code repositories or internal wikis.

Requirements

Must-Have Skills

• 57 years of hands-on experience writing well-structured Python code, with deep familiarity in both object-oriented and functional programming paradigms.
  
• Expert mastery of Pandas and NumPy for transformations (group-by aggregations, merges, column operations), plus strong comfort with Python’s datetime and copy modules.
  
• Proven experience designing modular Python packages, exposing configuration through parameters or picklists, and managing versioned releases.
  
• Ability to translate complex, rule-driven requirements (such as disaggregation rules, external-table merges, and priority ranking) into clean Python functions and classes.
  
• Proficiency with pytest (or equivalent), mocking, and integrating with CI pipelines (e.g., GitHub Actions, Jenkins) for automated testing.
  
• Skilled use of Python’s logging module to instrument scripts, manage log levels, and capture diagnostic information.
  
• Strong Git workflow experience, including branching strategies, pull requests, code reviews, and merge management.