About The Team
As a
Lead Data Engineer
, you will be joining the newly formed AI, Data & Analytics team, whose missionis to drive increased value from the data InvestCloud captures to enable a smarter financial future for ourclients, focused on enhanced intelligence. Ensuring we have fit-for-purpose modern capabilities is a keygoal for the team.This is a unique opportunity to shape the architecture and the tech execution patterns of a green fieldecosystem to create a next-generation advisor and client experience. We're building for scale. As such,much of what we design and implement today will be the technology/infrastructure which will servethousands of clients and petabyte-level volumes of data.
The core stack we use and are building is:
- AWS as our cloud provider
- Oracle as our legacy data warehouse
- Snowflake as our next-gen data warehouse
- Mage AI for data ingestion and processing
- Kafka as our message bus
- Terraform for building infrastructure
Key Responsibilities
- Ask the right questions to deeply understand stakeholder needs, ensuring solutions are aligned
with real problems.
- Drive the technical strategy of the team, and its execution through prioritization, and delivery
management.
- Lead a team to build reliable and scalable pipelines and capabilities across the platform, as well
as monitor and support the capabilities we offer.
- Inspire, teach and guide your team members; own design sessions, upskill the team where needed,
maintain the team's cohesion.
- Own complex architectures tying multiple services, SaaS tooling and third-party data together,
leveraging a strong understanding of a cloud-based stack.
- Uphold high-quality standards across documentation, testing, resiliency, monitoring, and code
quality.
- Drive towards efficiency, lower our cloud spend, tackle our tech debt and look for ways to simplify
code, infrastructure and data models across the platform.
- Promote data quality, governance and security as a first-class citizen of the platform, complying
with relevant regulations
Required Skills
- You have at least 10 years of relevant professional experience in Data Engineering
- You have shaped the architecture of a mature cloud data platform (AWS, GCP, Azure), designed
for different consumer types and providing quantifiable business value.
- You have served as a leader in technology in the past; you've built a roadmap, planned data/code
migrations and led execution, made mistakes and learned from them.
- You have interacted with a large community of stakeholders before; you understand the business
use cases and can tailor your communication to ICs and Senior Management.
- You have designed and implemented performant, reusable, and scalable data models in a cloud
data warehouse
(dbt, BigQuery, Snowflake)
and in a legacy ecosystem
(Oracle, Postgres)
- You have hands-on experience in building resilient batch (Airflow, Fivetran, Mage AI, Airbyte) and
streaming (Kafka, Kinesis, Flink, Spark) data pipelines at scale (> 1 TB/day)
- You can build and maintain your own infrastructure through IaC (Terraform, OpenTofu, Ansible),
containerization (Docker) and CI/CD (Jenkins, Github Actions).
- You preach operational procedures from data and infrastructure observability (Monte Carlo,
Datadog, Prometheus),
alerting and incident management (PagerDuty, incident.io)
- You bring deep expertise in SQL and Python, and use them to lead by example in technical
execution
- You're AI-proficient or AI-curious, eager to explore how emerging technologies can enhance team
productivity.
- You have experimented with Machine Learning frameworks (TensorFlow, PyTorch, Scikit-learn)
and LLM frameworks (e.g. Langchain)
Apply without meeting all requirements statementIf you don't meet every requirement but believe you'd thrive in this role, we'd still love to hear from you.We're always keen to speak to people who connect with our mission and values.
Location and Travel
The ideal candidate will be expected to work from the office on a regular basis (3 days minimum per week).Occasional travel may be required.
Compensation
The salary range will be determined based on experience, skills, and geographic location.Equal Opportunity Employer
InvestCloud is committed to fostering an inclusive workplace and welcomes applicants from all
backgrounds
