LLM Serving Estimator

Model Hugging Face repo

Any public repo — its config is read from huggingface.co.

Hardware

GPU

Number of GPUs

Workload

Concurrent users

Think time (s)

Prompt tokens

Output tokens

Think time is the pause between a user getting an answer and sending the next request. 0 means continuous load — the benchmark-style worst case.

Serving setup

Quantization

KV cache

Engine

Tensor parallel

Speculative decoding

Prefix-cache hit %

DP attention (MLA models such as DeepSeek, Kimi, GLM)

Service target — find the max users

Min tok/s per user

Max TTFT (s)

Gated or private model? Paste its config

config.json Total parameters, billions (optional)

How it works

Pick a model, GPUs and a workload. The estimator works out whether the weights and KV cache fit, then simulates decode and prefill on the hardware's memory bandwidth and compute, the model's MoE and attention structure, tensor-parallel communication and request queueing. Efficiency constants are fitted to public benchmarks; bands show the 10th–90th percentile error measured on held-out data.

Try: Llama-3.3-70B FP8 on 8×H100 · gpt-oss-120b on one H200 · DeepSeek-V4-Flash, 32k prompts, 8×B200

Median error	Per-user speed	TTFT	Throughput
Model + GPU measured	15%	32%	16%
Model family unseen	25%	45%	23%
New publishers & hardware	17%	59%	49%