Enterprises building vertical AI solutions must carefully weigh the cost of inference. At the beginning, it often feels easiest to rely on cloud-based GPU clusters—the fastest chips in the largest clusters appear to be the best choice. But reality sets in when the bills arrive.
A recent MIT Media Lab report, The GenAI Divide: State of AI in Business 2025, found that despite significant investment in generative AI, 95% of enterprise pilots failed to deliver measurable returns, leaving most companies stuck in “pilot purgatory.” Beyond workflow gaps and data quality issues, cost remains one of the biggest barriers to enterprise AI transformation.
Why Cost Matters So Much
Token-based pricing is unpredictable, with variable input lengths driving unpredictable spend.
Latency requirements often force enterprises to use expensive GPU instances, even for simple inference tasks.
Data transfer costs add up quickly in high-volume applications.
Free Utility For You
Estimate Your Inference Needs
While training gets a lot of attention, the ongoing cost of inference is what determines long-term viability. During pilots, organizations optimize for capability. But when it’s time to scale to thousands or millions of daily inferences, cost-efficiency becomes critical.
This raises a key question: Do you need to send every inference request through a cloud API? The answer is often no. There are alternatives that can deliver better production economics:
Alternative AI Inference Solutions
- On-Device / Edge Inference — Best for low-latency, privacy-sensitive applications. Run fine-tuned, lightweight models directly on user devices or local hardware.
- Self-Hosted or On-Premises Inference — Deploy models on your own servers or private cloud to reduce dependency on public APIs.
- Hybrid Approaches — Mix cloud, edge, and on-prem solutions depending on task complexity and cost tradeoffs.
Estimating AI Inference Chip Requirements
If you choose on-device inference, the natural next question is: what kind of chip performance do you actually need?
This is where metrics like TOPS (trillions of operations per second) come into play. But raw TOPS numbers are not enough—you must consider:
- Model size — Number of parameters and complexity (e.g., MobileNet vs. LLaMA).
- Precision — INT8, FP16, or 4-bit quantization dramatically change compute needs.
- Latency targets — Real-time (e.g., 30 FPS video) requires higher throughput than batch analytics.
- Throughput requirements — How many inferences per second or tokens per second must be supported.
- Memory bandwidth — Often the true bottleneck, especially for large language models.
In practice, estimating chip requirements means calculating the operations per inference (based on your model), dividing by the latency you’re targeting, and adding a safety margin for overhead. Only then can you match those needs to hardware specs and decide whether a mobile NPU, edge accelerator, or GPU is the right fit.
A quick 3-step sizing recipe
- Quantify the workload.
- Vision models: use published MACs/FLOPs per image.
- Encoders (e.g., BERT): FLOPs per sequence length.
- LLMs:
- Prefill (reading the prompt): ~2 × N_params × prompt_tokens FLOPs (rule of thumb).
- Decode (each new token): ~2 × N_params FLOPs/token (dense, non-sparse, rough).
- Set a latency/throughput goal. e.g., 30 FPS video; or 100 ms per request; or 20 tokens/sec.
- Compute required throughput ≈ ops per inference ÷ target latency, then add ×2 headroom for overheads.
Handy reference numbers
(Per single input unless noted; numbers vary by variant/implementation, but these are widely used orders of magnitude.)
- ResNet-50 (224×224): ~4 GFLOPs/img
- YOLOv5s: ~16–17 GFLOPs/img
- MobileNetV2: ~0.3 GFLOPs/img
- BERT-base (seq 128): ~11–12 GFLOPs/inference
- LLM rule of thumb:
- Per generated token ≈ 2 × N_params FLOPs
- Example: 7B params → ~14 GFLOPs/token (decode); 13B → ~26 GFLOPs/token
- Prefill adds 2 × N_params × prompt_len FLOPs up front.
Worked examples
1) Real-time camera with YOLOv5s @ 30 FPS
- Cost ≈ 17 GFLOPs/frame × 30 = 510 GFLOPs/s ≈ 0.51 TOPS (compute only).
- With overhead and INT8/FP16 inefficiencies, plan ~1–2 TOPS sustained (peak device >2–3× that).
2) BERT-base QA, seq 128, 10 ms latency target
- ~12 GFLOPs / 0.01 s = 1.2 TFLOPs ≈ 0.0012 TOPS (compute only).
- Add batching/overheads → provision ~0.01–0.1 TOPS sustained per stream.
3) LLM 7B decoding at 20 tokens/sec
- ~14 GFLOPs/token × 20 = 280 GFLOPs/s ≈ 0.28 TFLOPs (compute only).
- Prefill: if prompt=1,000 tokens → ~2 × 7B × 1,000 ≈ 14 TFLOPs burst (one-off).
- With KV cache, memory traffic is often the bottleneck; provision multi-TFLOPs (or several TOPS INT8) of peak plus strong memory bandwidth to comfortably hit 20 tok/s.
4) LLM 13B at 30 tokens/sec
- 26 GFLOPs/token × 30 ≈ 780 GFLOPs/s ≈ 0.78 TFLOPs (compute only).
- In practice, plan several TFLOPs sustained; for edge NPUs quoted in INT8 TOPS, look for >10–20 TOPS peak to have headroom.
Quick Cheatsheet
Object detection (small/medium YOLO) at 30 FPS: ~0.5–5 TOPS sustained, depending on input size/model.
Encoder NLP (BERT-base) real-time: <0.1 TOPS sustained per stream.
LLMs 7B–13B (decode 10–30 tok/s): compute suggests <1 TFLOP, but real deployments typically want multi-TFLOPs / 10–50 INT8 TOPS peak for headroom, KV cache bandwidth, and batching.
Bigger LLMs (30B–70B): scale roughly with params; expect order(s) of magnitude more memory and bandwidth; usually GPU/TPU-class.
To make it simple, I provide a Sizing Tool.
Additional Considerations
Precision/quantization (INT8/INT4) can cut compute & memory dramatically, but may shift bottlenecks to memory bandwidth.
Batch size improves hardware utilization but hurts latency.
Sequence/context length balloons prefill cost for LLMs.
Sparsity/pruning/Flash-Attention/KV cache reduce effective ops.
I/O and memory often dominate; a “high-TOPS, low-bandwidth” chip can still underperform.
Bottom line: Don’t default to the biggest cloud cluster. First, estimate your inference needs realistically—based on your model, your workload, and your performance goals. Then decide whether cloud, on-device, on-prem, or a hybrid approach gives you the best balance of cost, latency, and scalability.
