Data Engineer - Python, Spark, Scala

Job Title:

About the Role

We are looking for a passionate and experienced Data Engineer to join our growing data team. This is a hands-on role for someone who thrives in building scalable data platforms from the ground up. Youll work with cutting-edge technologies including Apache Spark, Google Cloud Platform (GCP), and modern data lake and warehouse architectures. If you enjoy solving complex data challenges and enabling data-driven decision-making, we’d love to hear from you.

Key Responsibilities

• Data Architecture & Design: Design and implement scalable data infrastructure including OLTP databases, data lakes, and cloud-based data warehouses.
• Spark-Based Data Engineering: Build and optimize data pipelines using Apache Spark (PySpark/Scala) for batch and real-time processing.
• Cloud Data Solutions: Leverage GCP services such as BigQuery, Dataflow, Dataproc, and Cloud Storage to build robust data solutions.
• ETL/ELT Development: Develop efficient ETL/ELT workflows to ingest, transform, and load data from diverse sources.
• Data Modeling & Governance: Define data models, enforce data quality standards, and implement governance practices across the data lifecycle.
• Automation & Monitoring: Automate workflows and implement monitoring to ensure data reliability, integrity, and performance.
• Cross-Functional Collaboration: Partner with software engineers, product managers, and data analysts/scientists to deliver data solutions aligned with business needs.
• Documentation & Best Practices: Maintain clear documentation and promote best practices in data engineering and architecture.

Required Skills & Qualifications

• Experience: 2+ years in data engineering with proven experience in building data platforms from scratch.
• Spark Expertise: Hands-on experience with Apache Spark (PySpark or Scala) in production environments.
• Cloud Proficiency: Strong experience with Google Cloud Platform data services (BigQuery, Dataflow, Dataproc, Cloud Storage).
• Big Data Ecosystem: Solid understanding of Hadoop, Hive, and distributed data processing.
• Programming: Proficient in Python or Scala for data engineering tasks.
• Databases: Expertise in SQL and experience with NoSQL databases (e.g., MongoDB, Cassandra).
• Data Warehousing: Familiarity with data warehouse design principles (Kimball/Inmon) and tools like BigQuery, Snowflake, or Redshift.
• Problem-Solving: Strong analytical skills and ability to make architectural decisions in ambiguous situations.
• Mindset: Self-starter with a builder mentality and a passion for scalable data systems.