Skip to main content

Latency Estimator

Physics-based latency modeling for LLM inference. Captures the two fundamental bottlenecks: memory bandwidth (decode) and compute (prefill).

estimateTpot(arch, contextTokens, batchSize, hardware, kvPrecision?)

Time Per Output Token — each decode step streams the full KV cache from HBM.
TPOT scales linearly with context length and batch size. fp8 halves it vs bf16.

estimateTtft(arch, promptTokens, concurrentUsers, hardware)

Time To First Token — compute-bound prefill with queue congestion.
The model uses TTFT(C) = singlePrefill × (C+1)/2 — with C concurrent users, the average user waits for C/2 prefills ahead of them.

singlePrefillMs(arch, promptTokens, hardware)

Base prefill time for a single prompt with no queue contention.
Prefill is compute-bound: flops = (4 × N² × hiddenDim + 4 × N × ffnDim) × layers.

ttftBreachPoint(arch, hardware, ttftSlaMs, promptTokens?)

Maximum concurrent users before TTFT exceeds the SLA.
Adding NAND does not change the breach point — NAND doesn’t help prefill compute.

restoreLatency(kvSizeGb, nandBandwidthGBs, parallelism?)

Time to restore a cold session from NAND SSD to GPU HBM.
Parallel restore streams share the same SSD pipe — higher parallelism helps throughput but increases per-stream latency.