Python Spark Developer

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Python and Apache Spark.
  
• Build high-performance data workflows leveraging pushdown optimization, streaming, replication, partitioning, and clustering.
  
• Assess and integrate modern data platforms and tools within the enterprise architecture.
  
• Engineer and maintain feature pipelines supporting real-time fraud detection systems.
  
• Create data models and processing strategies aligned with distributed-system principles to ensure scalability, consistency, and high performance.
  
• Deliver production-ready, maintainable solutions with strong observability and operational excellence.
  
• Apply clean code practices, SOLID principles, and proven architectural patterns to build robust, extensible systems.
  
• Participate in code reviews, testing, and deployment processes.
  
• Contribute to architectural decisions and continuous improvement initiatives.
  

Required Skills

• 3+ years of professional Python development experience with a strong focus on data engineering.
  
• Hands-on expertise with Apache Spark and large-scale data processing.
  
• Strong SQL skills with experience in both relational and distributed data systems.
  
• Solid understanding of software engineering principles, clean code practices, and design patterns.
  
• Experience designing and architecting scalable, data-intensive systems.
  
• Ability to evaluate emerging technologies and recommend enterprise-aligned solutions.
  
• Excellent analytical, problem-solving, communication, and collaboration skills.
  

Beneficial Skills (Nice to Have)

• Exposure to cloud platforms such as Snowflake, Databricks, or similar.
  
• Familiarity with Kafka, Redis, Airflow, Jenkins, and Git.
  
• Understanding of observability practices and tools (e.g., OpenTelemetry, Grafana, Loki, Tempo) with a focus on integrating observability into system design.
  
• Awareness of Kubernetes, Helm, GitOps, and containerization concepts.
  
• Background in real-time systems or financial services.
  
• A good sense of humor.