Summary
What you’ll impact
The Lead Inference Engineer owns end-to-end optimization of LLM, audio, and diffusion model inference to achieve sub-500 ms latency. The role uses Python, PyTorch, CUDA/Triton kernels, and serving frameworks like vLLM, SGLang, and TensorRT-LLM on GPUs. Key tasks include KV-cache policy design, INT8/INT4 quantization (GPTQ, AWQ), custom kernel work, and profiling tools to remove bottlenecks. They collaborate with research and infrastructure teams to ship optimized models and set performance standards across the stack.
Responsibilities
What you'll do
- Own end-to-end inference optimization across our model stack — LLMs, audio models, and diffusion-based components
- Implement and tune KV cache strategies for long-context conversations, including eviction policies, compression, and memory-efficient attention
- Evaluate, deploy, and extend inference serving frameworks (vLLM, SGLang, TensorRT-LLM, etc.) for our specific workloads
- Profile and benchmark end-to-end latency and throughput; identify and systematically eliminate bottlenecks
- Build internal tooling that makes optimization work faster and more rigorous — profiling viewers, end-to-end inference test harnesses, and other infrastructure that helps the team move quickly
- Accelerate diffusion model inference — consistency models, step distillation, caching strategies, and custom kernel optimizations
- Apply and develop quantization techniques (INT8, INT4, GPTQ, AWQ, and beyond) to reduce memory footprint and increase throughput without meaningfully degrading quality
- Work closely with research and infrastructure to ensure new models ship with optimized serving from day one
Requirements
What you’ll bring
- Significant hands-on experience with LLM inference optimization — you’ve shipped work on KV caching, memory layout, attention kernels, or batching strategies in a production or high-traffic research context
- Proven proficiency with inference serving frameworks — vLLM, SGLang, TensorRT-LLM, or similar — including going well beyond default configurations and adapting them to non-standard workloads
- Experience optimizing diffusion model inference (latency reduction, step distillation, caching, or kernel-level work)
- Strong Python and PyTorch skills; comfort reading and writing CUDA or Triton kernels is a significant plus
- A systematic approach to profiling and optimization — you measure first, then optimize
- Familiarity with speculative decoding or other inference-time acceleration techniques