Data Engineer

we are passionate about enabling data teams to solve the world's toughest problems, from security threat detection to cancer drug development. We do this by building and running the world's best data and AI infrastructure platform, so our customers can focus on the high-value challenges that are central to their own missions. Our engineering teams build technical products that fulfill real, important needs in the world. We always push the boundaries of data and AI technology, while simultaneously operating with the resilience, security, and scale that makes our customers successful on our platform.

We develop and operate one of the largest-scale software platforms. The fleet consists of millions of virtual machines, generating terabytes of logs and processing exabytes of data per day. At our scale, we observe cloud hardware, network, and operating system faults, and our software must gracefully shield our customers from any of the above.

As a Data Engineer on the IT Data Engineering team, you will help design and scale the SaaS Application Inventory platform that provides a single source of truth for the applications Databricks relies on. You will integrate data from Okta, ZIP, IoT, and Finance into Databricks to deliver trusted, high-quality datasets that drive visibility, governance, and decision-making across the company.

The Impact You Will Have

Design and build reliable ETL pipelines to ingest and transform application data from multiple sources into Databricks tables.

Redesign and extend existing schemas and data models to support SaaS, custom, and IoT applications.

Implement metadata tagging and cataloging to enable better discovery, classification, and governance.

Improve data quality through validation, cleaning, and fuzzy matching of application attributes.

Support and evolve a Streamlit-based dashboard that provides stakeholders visibility into the application inventory. Document your work and ensure knowledge transfer to other engineers for sustainability.

What We Look For

35 years of industry experience in data engineering or related fields.

Proficiency in Python and experience maintaining data-driven applications.

Hands-on experience with ETL pipeline development, schema design, and data integration.

Familiarity with Databricks, SQL, and large-scale data platforms.

Exposure to BI/reporting tools such as Streamlit or similar frameworks. Understanding of workflow orchestration tools (e.g., Airflow, dbt) is a plus.

Excellent collaboration and communication skills to work across distributed teams.

Passion for building data infrastructure and enabling others by making data easier to access.