Data Engineer

Description:

Architect and implement end-to-end data solutions using Databricks, Spark, Delta Lake, and

cloud services (AWS, Azure, GCP).

Lead data migration and transformation initiatives from legacy systems to modern lakehouse

architectures.

Collaborate with stakeholders to gather requirements and translate them into technical

designs.

Build and optimize data pipelines using PySpark, SQL, and Databricks Workflows.

Ensure robust data governance, security (RBAC, Unity Catalog), and compliance.

Guide teams in CI/CD practices using Databricks Asset Bundles or similar tools.

Provide technical leadership and mentorship to engineering teams.

Evaluate emerging technologies and contribute to innovation and continuous improvement.

Create documentation, training materials, and support post-deployment adoption.

Desired Profie:

7+ years of experience in data engineering or cloud architecture.

3+ years of hands-on experience with Databricks.

Should have experience in commercial datasets - IQVIA sales, Veeva CRM, HCP profiles,

Marketing, etc, 

Strong proficiency in Python, PySpark, SQL, and optionally Scala.

Expertise in Spark, Delta Lake, DLT, and Lakehouse architecture.

Experience with cloud platforms (AWS, Azure, GCP) and their integration with Databricks.

Familiarity with data governance, observability, and performance optimization.

Strong understanding of CI/CD, DevOps, and DataOps workflows.

Databricks certifications (e.g., Data Engineer Associate/Professional, Lakehouse Architect).

Experience with MLflow, TensorFlow, or other ML frameworks is a PLUS

Excellent communication and stakeholder management skills.

Required Skills:

Primary Skills:

Secondary Skills: