Lead Data Engineer

Data Engineer Lead:

• Bachelors Degree in Information Systems, Computer Science, or a quantitative discipline such as
• Mathematics or Engineering and/or equivalent formal training or work experience.
• Five to Seven (5 -7) years equivalent work experience in measurement and analysis, quantitative business problem-solving, simulation development and/or predictive analytics.
• Extensive knowledge in data engineering and machine learning frameworks including design, development and implementation of highly complex systems and data pipelines.
• Extensive knowledge in Information Systems including design, development and implementation of large batch or online transaction-based systems.
• Strong understanding of the transportation industry, competitors, and evolving technologies. Experience providing leadership in a general planning or consulting setting.
• Experience as a leader or a senior member of multi-function project teams. Strong oral and written communication skills.
• A related advanced degree may offset the related experience requirements.

Skill/Knowledge Considered a Plus

• Technical background in computer science, software engineering, database systems,
• distributed systems
• Fluency with distributed and cloud environments and a deep understanding of optimizing
• computational considerations with theoretical properties
• Experience in building robust cloud-based data engineering and curation solutions to create
• data products useful for numerous applications
• Detailed knowledge of the Microsoft Azure tooling for large-scale data engineering efforts and
• deployments is highly preferred. Experience with any combination of the following azure tools:
• Azure Databricks, Azure Data Factory, Azure SQL D, Azure Synapse Analytics
• Developing and operationalizing capabilities and solutions including under near real-time high-
• volume streaming conditions.
• Hands-on development skills with the ability to work at the code level and help debug hard to
• resolve issues.
• A compelling track record of designing and deploying large scale technical solutions, which
• deliver tangible, ongoing value
• Direct experience having built and deployed robust, complex production systems that implement modern, data processing methods at scale
• Ability to context-switch, to provide support to dispersed teams which may need an expert hacker” to unblock an especially challenging technical obstacle, and to work through problems as they are still being defined
• Demonstrated ability to deliver technical projects with a team, often working under tight time constraints to deliver value
• An ‘engineering’ mindset, willing to make rapid, pragmatic decisions to improve performance, accelerate progress or magnify impact
• Comfort with working with distributed teams on code-based deliverables, using version control systems and code reviews
• Ability to conduct data analysis, investigation, and lineage studies to document and enhance
• data quality and access
• Use of agile and devops practices for project and software management including continuous
• integration and continuous delivery

Demonstrated expertise working with some of the following common languages and tools:

• Spark (Scala and PySpark), Kafka and other high-volume data tools
• SQL and NoSQL storage tools, such as MySQL, Postgres, MongoDB/CosmosDB
• Java, Python data tools
• Azure DevOps experience to track work, develop using git-integrated version control patterns, and build and utilize CI/CD pipelines
• Working knowledge and experience implementing data architecture patterns to support varying business needs
• Experience with different data types (json, xml, parquet, avro, unstructured) for both batch and streaming ingestions
• Use of Azure Kubernetes Services, Eventhubs, or other related technologies to implement streaming ingestions
• Experience developing and implementing alerting and monitoring frameworks
• Working knowledge of Infrastructure as Code (IaC) through Terraform to create and deploy resources
• Implementation experience across different data stores, messaging systems, and data processing engines
• Data integration through APIs and/or REST service
• PowerPlatform (PowerBI, PowerApp, PowerAutomate) development experience a plus