Data Engineering Lead

Description

The Data Engineering Lead is a senior-level position within our Data Engineering

team, responsible for spearheading the design, implementation, and optimization of

advanced data architectures using Databricks. This role involves leading a team of

data engineers, collaborating with business and data analysts, and business

stakeholders, and delivering scalable, secure data solutions tailored to the needs of

commercial analytics in the pharmaceutical industry. 

Responsibilities

1.    Architecture Design

•    Develop and maintain scalable data architectures using Databricks, including

data lakes, data warehouses, and real-time processing systems.

•    Create detailed blueprints for data processes and flows, ensuring alignment with

business objectives.

2.    Pipeline Development

•    Design and implement ETL/ELT pipelines using Databricks and Apache Spark to

process large-scale datasets efficiently.

•    Optimize pipelines for performance, scalability, and reliability.

3.    Data Governance

•    Implement and enforce data governance policies, security measures, and

compliance standards within the Databricks environment.

4.    Collaboration

•    Partner with data scientists, business analysts, and business stakeholders to

understand data needs and deliver solutions that drive business value.

•    Communicate complex technical concepts clearly to diverse audiences, fostering

alignment and collaboration.

5.    Integration

•    Integrate Databricks with cloud services (e.g., AWS S3, Azure Data Lake

Storage, Google Cloud Storage) and CRM/marketing automation platforms (e.g.,

Salesforce, Veeva) for seamless data flow.

•    Ensure interoperability with existing systems to create a cohesive data

ecosystem.

6.    Innovation

•    Stay updated on advancements in Databricks, Delta Lake, Databricks SQL, and

related technologies, applying best practices to enhance data capabilities.

•    Drive continuous improvement in data processes and tools.

7.    Leadership and Mentoring

•    Lead the implementation and optimization of Databricks for commercial pharma

analytics, tailoring solutions for sales, marketing, and patient outcomes.

•    Train and mentor team members on Databricks and analytics platforms, fostering

a culture of data literacy and innovation.

8.    Analytics Collaboration

•    Collaborate with analytics teams to develop reporting frameworks for monitoring

KPIs, such as engagement rates and campaign ROI.

•    Ensure seamless integration of Databricks with CRM and marketing automation

systems to support analytics workflows.

Desired Profile:

•    12+ years of experience in data engineering, with a strong focus on Databricks,

Apache Spark, and cloud platforms (AWS, Azure, or GCP).

•    Proficiency in Databricks, Python, Scala, SQL, and experience with ETL/ELT

tools and pipeline orchestration (e.g., Apache Airflow, ADF).

•    Pharma knowledge especially around entity HCP & HCO
•    Data modeling/Medallion
•    ETL/ELT pipeline design

•    Data privacy/security
•    Pharma entity (HCP/HCO) integration
•    Deep knowledge of data modeling, schema design, and database management.
•    Proven leadership in managing data engineering teams and projects, with strong

project management skills.

•    Excellent communication skills, with the ability to convey complex technical

concepts to non-technical stakeholders.

•    Strong understanding of syndicated data in pharmaceuticals.
•    Experience in commercial pharma analytics is highly preferred.
•    Bachelors degree in computer science, Data Science, or a related field;

advanced degree (e.g., Master’s) is a plus.

•    Certifications in Databricks, AWS, Azure, or related technologies is REQUIRED.