Why This Field Matters

A few years ago, inference was a job for GPU servers. In 2026 the default is moving onto the device. llama.cpp and the GGUF format made running an LLM locally trivial. Q4 quantization shrinks a model to roughly a quarter of its FP16 size, so a 7B model fits inside 4GB of memory, and a GGUF build now lands within hours of nearly every new open-weight release. The same is true on phones. Meta’s MobileLLM 125M produces about 50 tokens per second on an iPhone, and small models like Llama 3.2’s 1B and 3B, Gemma 3 from 270M up, and Qwen2.5 at 0.5B and 1.5B run quantized to 4-bit on real handsets. On-device decoding sits under 20ms per token, while a cloud round-trip costs 200 to 500ms. Audio tells the same story. Kokoro, an 82M-parameter open TTS model with roughly 327MB of weights, runs faster than real time on a plain CPU including Apple Silicon, topped the TTS Arena leaderboard, and ships under Apache 2.0.

Why this turns into hiring is that three business drivers overlap. First, privacy. Data that never leaves the device cannot be breached. There are no server logs, no data-residency questions, no breach surface. As GDPR enforcement tightened, US state privacy laws multiplied, and Apple’s tracking limits landed, “send the user’s data to a server for AI processing” became a measurably worse default, and on-device inference does not trip any of those rules. Second, cost. Cloud inference bills per request, but pushing inference onto hardware the user already owns means the inference bill does not grow with traffic. Third, latency and offline. Real-time translation and AR cannot tolerate a network round-trip, and features that work without connectivity are a hard differentiator. At FAANG-scale orgs, ML platform teams are opening dedicated on-device tracks, and privacy-sensitive health and fintech companies hire this skill first.

Required Skills

This role sits in the narrow valley where ML systems knowledge overlaps native and mobile engineering, and you have to know both sides. First, quantization. INT8 and INT4 are table stakes, and you decide which of llama.cpp’s integer levels, from 1.5-bit up to 8-bit, to use. Dropping to 4-bit cuts memory to a quarter, but you learn from measurement where accuracy breaks, which tasks tolerate 4-bit, and which need 8-bit. Understanding in your bones why “train in 16-bit, deploy in 4-bit” became the industry standard is the starting point.

Second, runtimes and backends. You reach for llama.cpp and ggml for LLMs, Core ML on iOS and Apple Silicon, TFLite on Android, and ONNX Runtime for cross-platform. The core skill is having benchmarked, on real hardware, which backend wins for a given model and task. Mobile memory bandwidth runs 50 to 90 GB/s against a data center’s 2 to 3 TB/s, a gap of 30-fold or more, and even high-end phones cap usable RAM under 4GB, so thermal throttling and battery enter as variables. You track instruction-set extensions that bolt transformer inference onto the CPU, such as Qualcomm MX and ARM SME. In one case a 0.8B model completed a five-phase pipeline fully offline on a Snapdragon 7 Gen 1 in about 60 seconds. When the target is the browser, you layer WebGPU and Transformers.js and open a path up to 100x faster than WASM with a single device: 'webgpu' line.

Third, deployment and optimization. How to ship hundreds of megabytes of weights to the handset, on-device caching, the memory budget, cold start and fallback, and a TEE for model protection all fall inside the job. As Kokoro shows, one well-chosen small model can beat a cloud API on CPU, so the real weapon here is honing a small model to fit the device rather than scaling a heavy one.

Career Path

There are usually two ways in. An ML engineer goes deeper into device deployment, or a mobile or systems engineer picks up inference. Either way the first gate is the same: take an open quantized model, run it on a real phone or laptop, measure latency, memory, and battery, and write it up. One project at that level is enough to stand out in a junior portfolio. At the mid and senior level, a track record of running on-device inference inside a real product becomes the weapon. Results like “kept sensitive data off the server by processing on the device and erased the regulatory risk” or “moved inference to the handset and cut the monthly GPU bill by X percent” turn straight into negotiating leverage.

Experience tuning quantization loss, choosing backends, and designing model delivery and offline fallback separates this band. Above it, two directions open. One is on-device AI platform architect, owning the company’s device-inference standard and model deployment pipeline. The other is building a name by contributing directly to open-source ecosystems like llama.cpp and the Core ML tooling. The talent pool is still thin in this young area, so whoever enters now and builds references has a strong chance of being paid on scarcity for the next several years.