Break-even is a gate for benchmarking, not a deployment answer
The calculator compares the token API bill with GPU rental plus an operations amount and reports the token volume where those cash costs meet. If the current workload is far below break-even, a self-hosting project is unlikely to repay its complexity. If it is above break-even, the result justifies a technical capacity benchmark—it does not prove the proposed GPUs can serve the workload.
Use a blended API rate weighted by actual input, output, cache, and batch usage. For GPU cost, include the complete host, accelerator count, billable hours, idle headroom, storage, network, observability, engineering, and support. A bare GPU list price understates production cost; a reserved commitment may lower rate but adds utilization and lock-in risk.
Prove throughput under the real request distribution
Benchmark the exact model weights, precision or quantization, serving engine, tensor parallel setup, context lengths, input/output mix, batch policy, concurrency, and latency objective. Tokens per second changes with prompt length, decode length, batching, memory pressure, speculative decoding, and hardware. An average throughput number cannot size an interactive service by itself.
Measure time to first token, inter-token latency, requests per second, tokens per second, GPU memory, utilization, queue depth, failure rate, and accepted-task quality at several concurrency levels. Reserve capacity for spikes, rolling upgrades, and a failed node. If the API route and self-hosted model are not quality-equivalent, financial break-even is not comparable.
- Use regional SKU prices for the full machine, not a detached accelerator headline.
- Model Spot or interruptible capacity with checkpoint, queue, and failover cost.
- Amortize setup and recurring on-call work instead of calling engineering free.
- Keep a provider fallback until self-hosted availability and quality are proven.
Keep infrastructure behind a small serving contract
The product should call an internal inference interface for generation, streaming, usage, cancellation, and typed errors. Provider APIs and self-hosted engines implement that contract. Routing, authentication, retries, and model-specific parameters stay inside adapters, which prevents infrastructure experiments from leaking across application code.
Use idempotency for queued work, cap retries, and never blindly duplicate a timed-out generation. Fail fast when model artifacts, tokenizer, GPU capacity, or required configuration are missing. Degrade to a known provider route or a visible unavailable state; do not silently switch to a different model with different quality or data policy.
Common comparison mistakes
- Using a GPU hourly rate without the VM, CPU, memory, storage, network, or operations.
- Sizing capacity from one short-prompt throughput number.
- Comparing different model versions, quantizations, or quality levels.
- Assuming 100% utilization while reserving no latency or failure headroom.
- Removing the API fallback before self-hosted reliability is demonstrated.
Questions developers ask
When is self-hosting an LLM cheaper than an API?
Only after complete GPU and operations cost is below the comparable blended API bill and a benchmark proves capacity, latency, quality, reliability, and policy parity.
What does the break-even token volume mean?
It is the monthly token volume where your entered API rate equals your entered self-hosted monthly cost. It is a financial threshold, not a capacity forecast.
Should Spot GPU pricing be used?
Use it only with an interruption-aware queue, checkpoint or restart plan, availability model, and fallback cost. Dynamic capacity cannot be treated like guaranteed on-demand service.