Running Transformer Models on Mobile NPUs: What Actually Works (and What Breaks)

The Problem

We have a 28-emotion sentiment classifier (SamLowe/roberta-base-go_emotions, 125M parameters) running on-device in SentiLog, an Android journaling app. It detects emotions as you type — English, German, Japanese, and Korean — using ONNX Runtime at 46.5ms per inference on CPU. We wanted NPU acceleration for better battery life and CPU offload.

The hardware is good: OnePlus 13, Snapdragon 8 Elite, Hexagon HTP v79 NPU capable of 45 TOPS. In theory, a transformer should fly on this. In practice, getting a transformer to actually run on an NPU — not just fall back to CPU through the NPU codepath — required understanding how the model maps to hardware at the op level.

Inside the Transformer: What the NPU Actually Sees

Before you can diagnose NPU failures, you need to understand what happens when a text like "I can't stop thinking about her" enters the model. Here's every compute step, with what the NPU can and can't handle:

This is why generic NPU dispatch produces 3/7 subgraphs on NPU instead of all 7. The MatMul ops go to NPU. Everything else — LayerNorm, Softmax, GELU — falls back to CPU. The memory transfer overhead between those 7 subgraphs is what makes it 290ms instead of 46ms.

What We Tried (and What Failed)

1 LiteRT with Accelerator.NPU (Generic Dispatch)

val model = CompiledModel.create(
    modelPath,
    CompiledModel.Options(Accelerator.NPU, Accelerator.CPU)
)

2 QualcommOptions Tuning (Burst + O3)

3 Smaller Models (DistilBERT / DistilRoBERTa)

4 Full Static INT8 Quantization + LiteRT

5 NNAPI Delegate

What Actually Works: Qualcomm AI Hub Two-Step Pipeline

The key insight: compile the model on Qualcomm's cloud servers, not at runtime on the device. AI Hub's compiler knows the Hexagon HTP v79 instruction set and fuses LayerNorm, Softmax, and GELU into NPU-native instructions before the binary is ever deployed.

import qai_hub as hub
from qai_hub import QuantizeDtype

# STEP 1: Quantize with calibration data
# (no truncation — works with int64 model + int64 calibration)
quantize_job = hub.submit_quantize_job(
    model="model_fp32.onnx",          # clean FP32, no pre-quantization
    calibration_data=cal_dict,        # 100 representative texts (int64 IDs)
    weights_dtype=QuantizeDtype.INT8,
    activations_dtype=QuantizeDtype.INT16,  # w8a16
)
quantize_job.wait()
# Download quantized ONNX; merge external weights into single file

# STEP 2: Compile to QNN context binary
# (truncation here — already quantized, no calibration needed)
compile_job = hub.submit_compile_job(
    model="model_w8a16_merged.onnx",
    device=hub.Device("Snapdragon 8 Elite QRD"),
    options="--target_runtime qnn_context_binary --truncate_64bit_io --truncate_64bit_tensors",
)
compile_job.wait()
compile_job.download_target_model("model_w8a16_qnn.bin")

Full Benchmark Results

Tested on OnePlus 13 (Snapdragon 8 Elite, Hexagon HTP v79, Android 16). Validation: 149 curated emotion texts, RoBERTa-base, top-1 accuracy.

Quantization Experiment: Where Accuracy Dies

The accuracy impact of each quantization step, measured against the 149-text validation set:

The 6.7 percentage-point difference between blind INT8 and calibrated w8a16 comes from how transformers use activations. LayerNorm and Softmax produce narrow-range outputs that INT8 clips aggressively without calibration. INT16 activations give these layers the numerical headroom they need.

SentiLog Design Decisions

Theory is cheap. Here's every architectural decision that shaped SentiLog's on-device AI, with the reasoning and the alternatives we rejected.

Trade-offs & Decisions

Every architectural choice involves a trade-off. Here's the decision matrix for SentiLog's AI stack:

Best Practices

1. Don't Rely on Generic Runtime Dispatch

LiteRT's Accelerator.NPU does per-op dispatch. For CNNs with all NPU-native ops, this works. For transformers, it fragments the graph into alternating NPU/CPU subgraphs with ~0.5–2ms per memory transfer. Always use ahead-of-time compilation via Qualcomm AI Hub or equivalent vendor tools (MediaTek NeuroPilot, Samsung ONE, etc.).

2. Fix Your Input Shapes Before Submission

NPUs require static tensor shapes. Convert dynamic ONNX axes to fixed values before submitting to AI Hub:

import onnx
model = onnx.load("model.onnx")
for inp in model.graph.input:
    dims = inp.type.tensor_type.shape.dim
    dims[0].dim_value = 1    # batch size
    dims[1].dim_value = 128  # sequence length
onnx.save(model, "model_fixed.onnx")

3. Use w8a16, Not Blind INT8

INT8 weights + INT16 activations preserves numerical precision in LayerNorm and Softmax. Blind INT8 drops accuracy by ~8%; calibrated w8a16 loses only ~1.4%. Always provide 100+ representative calibration samples — the quantizer needs them to correctly set activation clipping ranges.

4. Submit Clean FP32 to AI Hub

Don't pre-quantize with ONNX Runtime before submitting to AI Hub. ONNX-specific quantization ops (QLinearMatMul, etc.) confuse the QNN converter and produce exit code 255. Let AI Hub handle quantization from a clean FP32 model with int64 inputs.

5. Ship CPU as Fallback — Always

// Graceful degradation across backends:
val result = qnn?.analyze(text)            // QNN NPU: 67ms, 5-10x less battery
    ?: onnxCpu?.analyze(text)              // ONNX CPU: 46.5ms, reliable
    ?: keywordClassifier.analyze(text)     // Keywords: 0ms, 100% compatibility

QNN binaries are device-specific (compiled for Snapdragon 8 Elite) and ORT-version-specific. Any device running a different chipset or a different ORT version will fall through to ONNX CPU automatically.

The Real Value of NPU

For transformer models, a well-optimized CPU is actually faster than NPU (46.5ms vs 67ms). The real NPU value is in power efficiency and sustained performance:

Architecture

Tools Used

• LiteRT 2.1.1 — Google's TFLite successor with CompiledModel API and QualcommOptions
• ONNX Runtime 1.22.0 — CPU inference + QNN EP for NPU; 16KB page-aligned .so for Android 15+
• Qualcomm AI Hub — Cloud compilation for Snapdragon NPU (free account, ~$0.40/job)
• litert-torch 0.9.0 — PyTorch → TFLite conversion
• ai-edge-quantizer — Post-training quantization for TFLite path
• Kotlin 2.2.0 — Required for LiteRT 2.1.1 CompiledModel API

Device

OnePlus 13 — Snapdragon 8 Elite (SM8750), Hexagon HTP v79, 16GB LPDDR5X, Android 16. All benchmarks run on-device using a custom ADB Debug API with a curated 149-text multilingual validation dataset. Testing on mid-range devices (Snapdragon 7 series, Dimensity 9000, Exynos) is planned for v2.5; results will vary due to different NPU architectures and memory bandwidth.

This is based on real benchmarks from the SentiLog project (May 2026). Results are specific to Snapdragon 8 Elite — other chipsets will behave differently. Source code and test datasets are in the repository.