Position Overview
The Team Lead for Advanced Data Systems Delivery will lead a team responsible for delivering high-performance, scalable data solutions, with a focus on designing and optimizing data architectures for mission-critical Tier 1 systems.This role requires deep expertise in deploying real-time and batch processing systems, utilizing the ArcOne AI platform to architect scalable topologies, tune critical components, and ensure compliance with NFRs and SLAs, while maintaining a strong foundation in data architecture principles.
Key Responsibilities
Project Leadership & Delivery :
- Lead the end-to-end delivery of advanced data systems for mission-critical Tier 1 applications, focusing on revenue management and other high-impact use cases.
- Ensure solutions meet NFRs (e.g., scalability, performance, availability, security) and SLAs, delivering on time, within scope, and budget.
- Utilize the ArcOne AI platform to design, scale, and tune system topologies to support complex real-time and batch workloads.
Data Architecture Leadership
- Architect and oversee the implementation of robust, scalable, and secure data architectures to support mission-critical applications.
- Design data models, schemas, and storage solutions (e.g., data lakes, data warehouses, NoSQL databases) optimized for performance, scalability, and accessibility.
- Implement and optimize ETL/ELT pipelines, ensuring efficient data ingestion, transformation, and integration across distributed systems.
- Ensure data governance, integrity, and security standards are maintained across all data architectures.
Team Management
- Manage, mentor, and inspire a team of data engineers, architects, and performance engineers, fostering a culture of technical excellence and collaboration.
- Provide technical guidance on data architecture, advanced data systems, and system optimization, ensuring alignment with industry best practices.
- Conduct performance reviews, set team goals, and support career development for team members.
Technical Expertise
- Oversee the deployment of high-scale, mission-critical systems handling real-time (e.g., streaming, event-driven) and batch processing workloads.
- Leverage the ArcOne AI platform to design and scale system topologies, tuning components (e.g., compute, storage, network) to meet performance and scalability requirements.
- Optimize data processing frameworks (e.g., Apache Kafka, Spark, Hadoop, Flink) and databases (e.g., SQL, NoSQL) for low latency, high throughput, and reliability.
- Lead investigations into performance regressions, scalability issues, or system failures, ensuring rapid resolution and root cause analysis.
Stakeholder Collaboration
- Act as the primary point of contact for stakeholders, translating business requirements into technical solutions with a focus on robust data architectures.
- Collaborate with infrastructure, platform, AI, and product teams to align on system design, performance, and data architecture goals.
- Communicate project progress, risks, and performance metrics to senior leadership and clients.
System Design & Optimization
- Design and implement data architectures that support scalability, fault tolerance, and high availability for Tier 1 applications.
- Optimize real-time (e.g., Apache Kafka, Flink) and batch processing (e.g., Spark, Hadoop) workloads for high-scale environments.
- Ensure data pipelines are optimized for performance, leveraging the ArcOne AI platform to automate and enhance topology scaling and component tuning.
- Advocate for simplicity and rigor in system and data architecture design to address complex performance and scalability challenges.
Process & Standards
- Define and enforce best practices for advanced data systems delivery, data architecture, CI/CD pipelines, and performance testing strategies.
- Develop and monitor SLAs/SLOs for latency, throughput, data availability, and system reliability, ensuring compliance with mission-critical standards.
- Stay updated on industry trends in advanced data systems, data architecture, and AI technologies to drive innovation within the team.
Qualifications
Education : Bachelors or Masters degree in Computer Science, or a related field.
Experience
- 7+ years of experience in delivering advanced data systems or similar solutions, with at least 3 years in a leadership or team lead role.
- Proven track record of deploying high-scale, mission-critical Tier 1 systems with real-time and batch workloads.
- Extensive experience in designing and implementing data architectures for large-scale, distributed systems, including data modeling, ETL/ELT pipelines, and storage solutions (e.g., data lakes, data warehouses).
- Hands-on expertise with the ArcOne AI platform or similar AI-driven tools to design, scale, and tune system topologies.
- Deep knowledge of performance tuning for data processing frameworks (e.g., Apache Kafka, Spark, Hadoop, Flink) and databases (e.g., SQL, NoSQL, data lakes).
- Strong understanding of NFRs (e.g., scalability, availability, performance, security) and SLAs for Tier 1 systems.
- Experience in revenue management or similar domains (e.g., pricing, forecasting, optimization) is a plus.
Technical Skills
- Proficiency in programming languages such as Python, Java, or Scala.
- Expertise in data processing technologies (e.g., Apache Kafka, Spark, Hadoop, Flink) and cloud platforms (e.g., AWS, Azure, GCP).
- Strong experience with data architecture tools and platforms, including data modeling (e.g., ERD, dimensional modeling), data lakes (e.g., Delta Lake, Iceberg), and data warehouses (e.g., Databricks, Dremio, DuckDB, Trino).
- Proficiency with performance profiling tools (e.g., perf, eBPF) and observability systems (e.g., Prometheus, Grafana, OpenTelemetry).
- Deep understanding of OS internals, networking, storage, and compute optimization (e.g., GPU utilization, memory management).
- Experience with containerization (e.g., Docker, Kubernetes) and data orchestration tools (e.g., Airflow, Dagster).
- Knowledge of AI/ML frameworks (e.g., TensorFlow, PyTorch) for optimizing data-driven solutions is a plus.
Soft Skills
- Exceptional leadership, communication, and stakeholder management skills.
- Strong problem-solving abilities with a focus on simplicity, rigor, and collaboration.
- Ability to navigate ambiguity and drive alignment across cross-functional teams.
Preferred
- Experience with the ArcOne AI platform for system design and optimization.
- Familiarity with agile methodologies and tools like Jira or Trello.
- Certification in cloud platforms (e.g., AWS Certified Solutions Architect), data engineering, or data architecture (e.g., TOGAF, DAMA-DMBOK)
(ref:hirist.tech)
