AWS Databricks architect - 15+years @ Bangalore

As a Lead Databricks Data Platform Engineer, you will play a critical role in designing, scaling, and governing our unified Databricks environment within the Drug Innovation Platform. You will partner closely with product, engineering, and data science teams to enable reusable patterns, optimize performance, and ensure robust architecture and operations that align with our DSE@DBX vision.

This role requires a hands-on technical leader who brings deep Databricks expertise, extensive experience designing large-scale lakehouse architectures, and a collaborative mindset to connect siloed initiatives into a cohesive, shared platform.

Key Responsibilities

• Lead the end-to-end design, architecture, and engineering of our Databricks platform at scale, ensuring best practices in data management, security, governance, and cost optimization.
• Act as a subject matter expert and trusted advisor for all Databricks engineering topics from cluster management, pipelines, and Delta Lake patterns to ML workloads and performance tuning.
• Partner with domain teams to onboard new use cases onto the unified platform, ensuring they follow standard patterns and benefit from reusable assets.
• Build, document, and evangelize reusable templates, guardrails, and reference architectures to streamline new product onboarding and reduce silos.
• Provide hands-on technical guidance and mentorship to data engineers and analysts working within the Databricks ecosystem.
• Collaborate with AWS cloud engineers and architects to optimize integrations, networking, and data flows between Databricks and other key cloud services.
• Drive continuous improvement around platform reliability, cost efficiency, and performance by implementing robust monitoring and automation.
• Stay current with the latest Databricks capabilities and certifications; lead POCs to evaluate and adopt relevant new features for the organization.
• Contribute to the development and sharing of internal Databricks best practices, training materials, and community-of-practice sessions.
• Engage with stakeholders across data science, product, and platform teams to align the Databricks roadmap with broader DSE@DBX goals.

Key Skills & Qualifications

• Minimum 5 years of hands-on experience architecting, engineering, and operating Databricks platforms at enterprise scale.
• Multiple relevant Databricks certifications ideally including Databricks Certified Data Engineer Professional, Databricks Certified Machine Learning Professional, and Databricks Certified Solutions Architect.
• Deep expertise in Databricks components, including Unity Catalog, Delta Lake, DBSQL, Spark optimizations, cluster management, and CI/CD for pipelines.
• Strong understanding of AWS cloud architecture, networking, IAM, storage integrations (e.g., S3, Glue, Lake Formation), and security best practices.
• Experience defining and implementing data governance, lineage, and compliance frameworks within Databricks.
• Proven track record of leading platform initiatives and collaborating with cross-functional teams in a complex, regulated environment.
• Excellent communication and documentation skills — able to translate technical details into reusable guidance for a broad audience.

Self-starter with a mindset for platform thinking, reuse, and community enablement.