Databricks Data Architect

About the Role

The Databricks Data Architect will play a pivotal role in designing and scaling Lillyu2019s Data & AI platform on cloud environments such as AWS and Azure. This role focuses on leveraging Databricks Lakehouse to deliver secure, scalable, and high-performing data architectures that empower AI/ML, analytics, and data product development across global business functions.

The ideal candidate will blend deep technical expertise, data platform design experience, and strategic vision to enable Lillyu2019s transformation into an AI-driven enterprise.

Key Responsibilities

1. Platform & Architecture Leadership

• Design, implement, and optimize Lakehouse architectures using Databricks, Delta Lake, and Unity Catalog across AWS or Azure platforms.
• Architect and operationalize data pipelines that integrate structured and unstructured data sources for analytics and AI workloads.
• Build scalable frameworks to support machine learning, generative AI, and predictive analytics at enterprise scale.
• Ensure architectural alignment with cloud security, governance, and compliance standards.

2. AI/ML Enablement

• Collaborate with Data Scientists, ML Engineers, and AI Product Teams to operationalize models using Databricks MLflow and Feature Store.
• Design data readiness pipelines that enable AI/ML experimentation, model training, and inference.
• Implement MLOps practices for reproducibility, monitoring, and lifecycle management of ML models.

3. Data Governance & Optimization

• Implement data access, lineage, and stewardship frameworks using Unity Catalog, Purview, or AWS Glue Data Catalog.
• Optimize Databricks performance, scalability, and cost through intelligent cluster configuration and workload tuning.
• Champion metadata management, data observability, and quality frameworks to support enterprise-grade reliability.

4. Collaboration & Leadership

• Partner with Product Owners, Business SMEs, and Cloud Architects to design domain-aligned, reusable data products.
• Mentor and guide engineers on data engineering and cloud platform best practices.
• Serve as a technical thought leader, influencing data architecture standards and AI-enablement roadmaps.

Required Skills & Qualifications

• Bacheloru2019s or Masteru2019s degree in Computer Science, Engineering, or related discipline.

• 10+ years in Data Engineering / Architecture roles with 3+ years of hands-on Databricks experience.
• Strong proficiency in PySpark, SQL, Delta Lake, and Unity Catalog.
• Proven experience in designing and managing data solutions on AWS or Azure (e.g., S3, Redshift, Glue, Data Factory, Synapse).
• Deep understanding of data modeling, ETL orchestration, and performance tuning.
• Familiarity with MLOps, Feature Stores, and AI/ML lifecycle automation.
• Experience with unstructured data, vector databases, and GenAI pipelines is an added advantage.
• Exposure to pharma, life sciences, or regulated data environments preferred.

Preferred Certifications

• Databricks Certified Data Engineer / Architect (Associate or Professional).
• AWS Certified Data Analytics / Azure Data Engineer Associate.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form () for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lillyu00A0does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly