Why This Field Matters

Training a model and shipping it to users fast and cheap are different skills. The person who owns the second one is the LLM serving systems engineer. How you place the same model on expensive GPUs decides throughput and cost per token by multiples. Now that inference sits at the center of SaaS cost of goods, the hands that close this gap become the margin.

The numbers carry the argument. In UC Berkeley’s PagedAttention paper, existing serving systems wasted 60–80% of KV-cache memory. By borrowing virtual-memory paging from operating systems, that waste dropped below 4%, and at the same latency, throughput jumped 2–4x over FasterTransformer and Orca. No model change — just the serving layer. A single GPU absorbing two to three times the concurrent requests also means buying that many fewer GPUs.

Required Skills

You have to know the inference engines. The 2026 standard splits three ways — vLLM, SGLang, and TensorRT-LLM — and all three support continuous batching, prefix caching, speculative decoding, quantization, and disaggregated serving out of the box. vLLM leans into GPU utilization and concurrency, TensorRT-LLM into low-level NVIDIA hardware optimization, and SGLang into Chinese open models like DeepSeek and Qwen plus multi-turn workloads. Deciding which engine to put behind which workload, and which flags to pass, is half the job.

The low-level instinct has to back it. Inference splits into two phases: prefill computes the prompt’s KV cache in one shot and is compute-bound, while decode emits tokens one at a time and is memory-bound. Run both on the same GPU and they interfere, degrading TTFT and TPOT together. That gave rise to disaggregated serving, which separates prefill and decode onto distinct GPU pools. NVIDIA’s measurements of TensorRT-LLM disaggregation on GB200 show 1.4–2.5x on DeepSeek R1 and up to 6.11x on Qwen 3 depending on input/output length. The role reaches down into KV-cache transfer over RDMA and NVLink, and cache-layout transforms across parallelism strategies (TP/PP). Python alone won’t cut it — the hot paths bring in Rust, C++, and CUDA.

Career Path

Juniors start by taking an existing inference engine, standing it up, and tuning it. Swap static batching for vLLM’s continuous batching to double throughput, then bolt speculative decoding onto a latency-tight single-user path to cut time-to-first-token by 2–3x. You build the eye for benchmarks and for reading TTFT, TPOT, and goodput first.

Seniority moves you from using the engine to fixing and building it. You design disaggregated serving architectures, implement KV-cache compression and transfer yourself, and own scheduling for multi-node deployments. NVIDIA and Google hire for this under titles like “AI Inference Performance Engineer” and “LLM Serving and GPU Performance.” U.S. LLM engineer pay runs $155K–$225K mid-level and $245K–$355K senior, stretching to $480K–$750K with equity at frontier labs. Inference engineering is named the fastest-growing discipline in AI — because the hands that cut cost per token are the ones companies need first.