Uber s data infrastructure is composed of a wide variety of compute engines, scheduling/execution solutions, and storage solutions. Compute engines such as Apache Spark , Presto , Apache Hive , Neutrino, Apache Flink , etc., allow Uber to run petabyte-scale operations on a daily basis. Further, scheduling and execution engines such as Piper (Uber s fork of Apache Airflow ), Query Builder (user platform for executing compute SQLs), Query Runner (proxy layer for execution of workloads), and exist to allow scheduling and execution of compute workloads. Finally, a significant portion of storage is supported by HDFS, Google Cloud Storage (GCS),Apache Pinot , ElasticSearch , etc. Each engine supports thousands of executions, which are owned by multiple owners and sub-teams.
With such a complex and diverse big data landscape operating at petabyte-scale and around a million applications/queries running each day, it s imperative to provide the stakeholders a holistic view of the right performance and resource consumption insights.
DataCentral, is a comprehensive platform that provides users with essential insights into big data applications and queries. It empowers data platform users by offering detailed information on workflows and apps, improving productivity by reducing debugging time and improving the cost efficiency by providing detailed resource efficiency insights
As an engineer in the Data Central Team, you will be solving some of the most complex problems in Observability and efficiency of Distributed Data Systems at Uber scale.
What Youll Do
- Work with Uber data science and engineering teams to improve Observability of Batch Data use-cases at Uber.
- Leverage knowledge of spark internals to dramatically help improve customers Spark job performance.
- Design and implement AI based solutions to improve the application debuggability.
- Design and implement algorithms to optimize Resource consumption without impacting reliability
- Design and develop prediction and forecasting models to proactively predict system degradations and failures
- Work with multiple partner teams within and Uber and build cross-functional solutions in a collaborative work environment.
- Work with the community to upstream Ubers contributions to open source and also keep our internal fork up to date
What Youll Need
- Bachelor s degree in Computer Science or related field.
- 5+ years of experience building large scale distributed software systems.
- Solid understanding of Java for backend / systems software development.
- MS / PhD in Computer Science or related field.
- Experience managing production systems with a strong availability SLA.
- Experience working with Apache Spark or similar analytics technologies.
- Experience working with large scale distributed systems, HDFS / Yarn.
- Experience working with SQL Compiler, SQL Plan / Runtime Optimization.
- Experience working with Kubernetes
Ubers mission is to reimagine the way the world moves for the better. Here, bold ideas create real-world impact, challenges drive growth, and speed fuelds progress. What moves us, moves the world - let s move it forward, together.
Offices continue to be central to collaboration and Ubers cultural identity. Unless formally approved to work fully remotely, Uber expects employees to spend at least half of their work time in their assigned office. For certain roles, such as those based at green-light hubs, employees are expected to be in-office for 100% of their time. Please speak with your recruiter to better understand in-office expectations for this role.
*Accommodations may be available based on religious and/or medical conditions, or as required by applicable law.
