Senior Big Data Engineer

Purpose

Key Responsibilities

• Architect and implement ETL/ELT pipelines for streaming (Kafka/Spark) and batch log processing, including data enrichment (e.g., IP geolocation, user mapping) in high-volume scenarios.
• Build and optimize high-performance APIs (e.g., using FastAPI) to deliver processed logs and automate integrations for internal apps, ensuring low-latency access.
• Integrate external data sources, optimize queries for concurrent users, and design for scalability (e.g., handling 10x-100x growth in log data).
• Manage data lifecycle processes: compaction, migration to cold storage (on-prem MinIO equivalents), and cost/efficiency optimizations.
• Lead performance testing (load, concurrency) and document proposed solutions for ETL designs, API architectures, and automation workflows.
• Collaborate with infrastructure teams on shared tools like Spark/Kafka, focusing on development-side optimizations without delving into setup/monitoring.

Technical Requirements

• Core Skills:
• 5+ years in Python for big data applications (Spark/Hadoop; ELK/Elastic Stack experience is a plus for log handling).
• 2+ years developing APIs (FastAPI/Flask), streaming pipelines (Kafka/Spark Streaming), and ETL automations in on-premise environments.
• Proficiency in SQL, pandas, numpy, and on-prem storage solutions (e.g., MinIO).

• Design Skills

  : System architecture diagramming (e.g., UML), API integration, and rapid adaptation to tools like Airflow for orchestration.

• Flexibility

  : Experience with GCP/AWS tools (e.g., Kinesis, S3) is a plus for conceptual knowledge and adaptability.

Soft Skills

• Analytical approach to solving performance and scalability issues in big data ETL/APIs.
• Clear communication for documenting proposed architectures, solutions, and collaborating on cross-role integrations.
• Results-oriented, with a focus on meeting SLAs/KPIs (e.g., latency, throughput, data accuracy in high-volume logs).

Candidate Profile: