LLM Systems Engineer

As an engineer in this role, you will be responsible for building and optimizing high-throughput, low-latency LLM inference infrastructure. This will involve using open-source models such as Qwen, LLaMA, and Mixtral on multi-GPU systems like A100/H100. Your main areas of focus will include performance tuning, model hosting, routing logic, speculative decoding, and cost-efficiency tooling. To excel in this position, you must have deep experience with vLLM, tensor/pipe parallelism, and KV cache management. A strong understanding of CUDA-level inference bottlenecks, FlashAttention2, and quantization is essential. Additionally, familiarity with FP8, INT4, and speculative decoding (e.g., TwinPilots, PowerInfer) will be beneficial. You should also possess a proven ability to scale LLMs across multi-GPU nodes using TP, DDP, and inference routing techniques. Proficiency in Python at a systems level, experience with containerized deployments using tools like Docker, GCP, or AWS, and knowledge of load testing with Locust are required skills for this role. Any experience with any-to-any model routing (e.g., text2sql, speech2text) and exposure to LangGraph, Triton kernels, or custom inference engines will be considered a bonus. Moreover, if you have successfully tuned models for less than $0.50 per million token inference at scale, it will be an advantage. In return for your expertise and contributions, we offer a very competitive rate card for the best candidate who fits this role.,