AVP Databricks Architect

Assistant Vice President- Lead Data Engineer

In this role, a Lead data engineer will lead the design and optimization of advanced data solutions. This role requires expertise in Databricks, Azure Data Factory (ADF), Python, PySpark and Unity Catalog to efficiently process and manage large datasets, along with a deep understanding of cloud architecture to build scalable, secure, and reliable data solutions on the Microsoft Azure platform. The primary responsibility of the lead data engineer with Unity Catalogue expertise is to apply advanced data engineering skills to optimize data integration, enhance data accessibility, and drive strategic decision-making through effective data governance, simplification, standardization, and innovative solutions across all supported units. This role will be implementing DevOps best practices and driving innovation using modern data platform capabilities such as Unity Catalog, MLflow, and Large Language Models (LLMs).

Responsibilities

• Design and development.

• Collaborate with business stakeholders and analysts to understand data requirements. Design, develop, and test data pipelines and workflows using Unity Catalogue to optimize end-to-end processes. Create reusable components, robust exception handling, and standardized frameworks for data solutions.

• Solution Design

• Develop and maintain robust data architectures using Lakehouse principles to ensure efficient data processing and storage. Comprehensive data architecture solutions using Databricks and Lakehouse principles to support advanced analytics and machine learning initiatives.

• Explore and integrate Large Language Models (LLMs) and Copilot tools to drive automation and agility.

• Leverage Databricks MLflow for model lifecycle management and operationalization

• Data Quality and Governance:

• Ensure data quality frameworks, lineage, and monitoring are in place.

• Implement data quality checks, validation rules, and governance policies to ensure the accuracy, reliability, and security of data assets.

• Data Integration and Analytics:

• Pull data from different sources, transform and stitch it for advanced analytics activities.

• Design, implement, and deploy data loaders to load data into the engineering sandbox.

• Collaborate with data scientists and analysts to support their data requirements and prepare machine learning feature stores.

• Leadership and Mentorship:

• Own complex, cross-functional data projects from ideation to production, including defining requirements, designing solutions, leading development, and ensuring successful deployment and long-term maintenance.

• Provide guidance and technical leadership to a team of data engineers through in-depth code reviews, mentoring junior and mid-level engineers, and fostering a culture of technical excellence.

• Mentor mid-level engineers and perform peer reviews.

• Process

• Drive continuous improvement initiatives in data processes and systems. Promote standardization and automation to enhance efficiency and accuracy. Support regional and global data projects

Qualifications We Seek in You!

Minimum Qualifications / Skills

• Bachelor's degree in computer science, Information Systems, or a related field.

• Experience in Databricks, Azure ADF, Python, Pyspark and Unity Catalog Dataflow, and Lakehouse architecture

• Deep hands-on expertise in Azure Data Services (e.g., Azure Data Lake, Azure Data Factory, Synapse, etc.) and Databricks.

• Strong experience in data pipeline design, ETL/ELT development, and data orchestration frameworks.

• Proficiency in DevOps tools and practices (CI/CD pipelines, IaC, monitoring).

• Knowledge of data lineage, cataloging, and enterprise data marketplace concepts.

• Familiarity with integrating 3rd party data sources and managing data quality frameworks.

• Ability to leverage LLMs and Copilot solutions to enhance data platform productivity.

• Experience in building self-healing architecture for data pipelines.

• Proven experience in managing data projects in complex environments, including global or multinational contexts

• Hands-on experience with data pipeline development and optimization

• Deep knowledge of data governance frameworks and tools, including Databricks Unity Catalog, to ensure data security, quality, and compliance at an enterprise level.

• A strong understanding of MLOps for building data foundations that support machine learning.

• Experience with DevOps practices to enhance data project delivery efficiency

Preferred Qualifications / Skills

• Prior track record of leadingenterprise HR/People platformsa plus

• Leads multiple pods, mentoring senior and mid-level engineers

• Experience in large-scale Lakehouse design, data mesh principles, and performance optimization

• Certifications in Azure data engineering, Databricks or related fields