LLM Systems Engineer

As an engineer in this role, you will be responsible for building and optimizing high-throughput, low-latency LLM inference infrastructure using open-source models such as Qwen, LLaMA, and Mixtral on multi-GPU systems like A100/H100. Your ownership will include performance tuning, model hosting, routing logic, speculative decoding, and cost-efficiency tooling. Key Responsibilities: - Deep experience with vLLM, tensor/pipe parallelism, and KV cache management - Strong grasp of CUDA-level inference bottlenecks, FlashAttention2, and quantization - Familiarity with FP8, INT4, and speculative decoding (e.g., TwinPilots, PowerInfer) - Proven ability to scale LLMs across multi-GPU nodes using TP, DDP, and inference routing - Proficiency in Python (systems-level), containerized deployments (Docker, GCP/AWS), and load testing (Locust) Qualifications Required: - Experience with any-to-any model routing (e.g., text2sql, speech2text) is a bonus - Exposure to LangGraph, Triton kernels, or custom inference engines - Previous experience in tuning models for less than $0.50 per million token inference at scale Please note that the company offers a very competitive rate card for the best candidate fit. As an engineer in this role, you will be responsible for building and optimizing high-throughput, low-latency LLM inference infrastructure using open-source models such as Qwen, LLaMA, and Mixtral on multi-GPU systems like A100/H100. Your ownership will include performance tuning, model hosting, routing logic, speculative decoding, and cost-efficiency tooling. Key Responsibilities: - Deep experience with vLLM, tensor/pipe parallelism, and KV cache management - Strong grasp of CUDA-level inference bottlenecks, FlashAttention2, and quantization - Familiarity with FP8, INT4, and speculative decoding (e.g., TwinPilots, PowerInfer) - Proven ability to scale LLMs across multi-GPU nodes using TP, DDP, and inference routing - Proficiency in Python (systems-level), containerized deployments (Docker, GCP/AWS), and load testing (Locust) Qualifications Required: - Experience with any-to-any model routing (e.g., text2sql, speech2text) is a bonus - Exposure to LangGraph, Triton kernels, or custom inference engines - Previous experience in tuning models for less than $0.50 per million token inference at scale Please note that the company offers a very competitive rate card for the best candidate fit.