Data Engineer

We are seeking a highly skilled Data Engineer with strong hands-on experience in Python, PySpark, Apache Spark, Kafka, and Databricks to design, build, and maintain scalable and high-performance data pipelines. The role focuses on developing robust batch and streaming data solutions for large-scale distributed data environments.

Key Responsibilities

• Design, develop, and maintain batch and real-time data pipelines using PySpark and Apache Spark
• Build and manage scalable data workflows and jobs on Databricks
• Develop and optimize Python and SQL code to process large and complex datasets
• Implement Kafka-based streaming solutions for real-time data ingestion and processing
• Perform Spark job tuning, performance optimization, and resource management
• Ensure data reliability, scalability, and efficiency across pipelines
• Work closely with analytics, product, and engineering teams to support data requirements
• Monitor, troubleshoot, and resolve data pipeline and performance issues
• Maintain documentation for data pipelines, workflows, and processing logic

Required Experience & Qualifications

• 510 years of experience in Data Engineering roles
• Strong hands-on coding experience in Python and PySpark
• Extensive experience with Apache Spark and Databricks platforms
• Hands-on experience with Kafka for streaming data processing
• Strong SQL skills for data transformation and validation
• Experience working with large-scale, distributed data systems

Preferred Skills

• Experience with Delta Lake for data versioning and reliability
• Exposure to cloud platforms such as AWS, Azure, or GCP
• Experience with workflow orchestration tools such as Airflow