Data Engineer

Company:

Website:

Business Type:

Company Type:

Business Model:

Funding Stage:

Industry:

Salary Range:

This is the permanent role with a Valued client of e-Hireo.

About The Role

highly autonomous Data Platform Engineer

Databricks, Spark technologies

Python-based data validation frameworks

Key Responsibilities

• Design, develop, and optimize data pipelines using PySpark and Spark SQL
• Build and manage scalable data solutions on Databricks with Unity Catalog
• Implement robust data quality checks using the Pandera framework
• Develop and integrate RESTful APIs to support platform and data operations
• Collaborate with data engineers, analysts, and DevOps to ensure seamless pipeline deployments
• Write clean, testable code and unit tests using frameworks like Nutter
• Automate infrastructure and pipeline provisioning using Terraform
• Work with version control, CI/CD pipelines, and deployment best practices
  

Must-Have Skills

• Strong experience with PySpark and Spark SQL
• Proven hands-on experience with Databricks, especially Unity Catalog
• Proficiency in Python, including Pandera for schema validation
• Experience with REST API frameworks (e.g., FastAPI, Flask, Django REST Framework)
• Ability to work autonomously and take end-to-end ownership of data tasks
  

Nice-to-Have / Preferred Skills

• Google Cloud Platform (GCP):
• Firestore, Google Cloud Storage (GCS)
• Cloud Build, BigQuery
• Experience with Airbyte or Fivetran for data ingestion
• Knowledge of Nutter for unit testing in Databricks
• Familiarity with Terraform for infrastructure as code (IaC)
• Understanding of CI/CD concepts and DevOps practices
• Knowledge of OAuth 2.0 protocol for secure API integration
  

Soft Skills & Attributes

• Strong problem-solving and analytical skills
• Ability to manage and prioritize multiple projects
• Excellent communication and stakeholder management skills
• Proactive and independent work ethic