16 — Migration and upgrade: when a new GPU generation is worth it

The decision this document serves: you are running a serving workload on generation N. Generation N+1 is available. Should you upgrade? The answer depends on whether your workload is bound by the thing the new generation improves, and whether the price increase is offset by the throughput increase. The records give enough data to answer this for three specific generational transitions.

1. The one-paragraph version

A new GPU generation is worth it when your workload is bound by the specific resource the new generation improves — and the price/throughput ratio favours the new part. The records show three transitions where the answer is different:

• H100 → H200: worth it only if you are capacity-bound (KV cache or weights don't fit on 80 GB). At batch 1 on an 8B model, H200 is 1.14x faster but 2.02x more expensive per hour — the upgrade makes $/Mtok worse by 1.77x.
• H100 → B200: worth it if you need the FP4 tensor-core path or the 8 TB/s bandwidth for a workload that is compute-bound at high batch. At batch 1 on a 70B model, B200 serves 6,725 tok/s on a single GPU where H100 needs TP=2 for 3,708 tok/s — but B200 costs 3.2x more per hour.
• A100 → H100: worth it for almost every workload. H100 is 3.2x the dense bf16 FLOPS (989.5 vs 312) and 1.64x the bandwidth (3,350 vs 2,039 GB/s), and the rental price spread ($2.47 vs $3.43) is only 1.39x.

2. H100 → H200: the capacity upgrade that is not a speed upgrade

2.1 What changes

H200 is purely a memory-capacity and bandwidth uplift — same compute, same power. The record's own summary: "it wins wherever KV cache or weights do not fit, and loses on nothing."

2.2 The measured speedup at batch 1

TensorRT-LLM: Llama 3.1 8B FP8 on 1x H100 SXM 80GB, ISL/OSL 1000/1000: 14,991.62 output tok/s (H100, 1 GPU) TensorRT-LLM: Llama 3.1 8B FP8 on 1x H200 SXM 141GB, ISL/OSL 1000/1000: 17,162.49 output tok/s (H200, 1 GPU)

Speedup: 1.14x. The 1.43x bandwidth increase yields only 1.14x throughput because at batch 1 on an 8B model the step is launch-overhead-bound, not bandwidth-bound (Per-kernel launch and dispatch overhead at batch 1).

2.3 The cost-per-token penalty

At rental rates, H200's $/Mtok is 1.77x worse than H100's. At AWS on-demand the prices are close enough that the 1.14x speedup makes H200 slightly better. The answer is channel-dependent.

2.4 When H200 is worth it

• KV cache does not fit on 80 GB. A 70B model at 128k context with bf16 KV needs ~2.9 GB per sequence for the cache alone (KV cache movement: block copy, prefix sharing, prefill-to-decode handoff, and offload gives the per-token formula). At 200 concurrent sequences that is 580 GB — fits on H200's 141 GB with room for weights, does not fit on H100's 80 GB.
• Weights do not fit on 80 GB. A 405B model in bf16 needs 810 GB; in FP8 it needs 405 GB. Neither fits on a single H100. H200's 141 GB does not change this — you need TP=8 either way. But if you are at TP=2 with an 80 GB card and the model is 130 GB, H200 lets you stay at TP=2 where H100 would force TP=4.
• You are serving long-context and the bandwidth is the binding constraint. At 32k+ context, decode attention is bandwidth-bound on KV reads (Attention over the KV cache in the decode step: AI = g, flat in context). H200's 1.43x bandwidth increase translates directly to 1.43x decode throughput at long context.

2.5 When H200 is not worth it

• Batch-1 serving of a model that fits on 80 GB. The 1.14x speedup does not cover the 2.02x price increase. Stay on H100.
• High-concurrency serving where H100 is already compute-bound. Above batch ~295, H100 is compute-bound on the weight GEMM (Dense weight GEMMs in the decode step (the per-token matmul tax): B_ridge = 295). H200 has the same compute, so there is no throughput gain at high batch.

3. H100 → B200: the FP4 path and the bandwidth leap

3.1 What changes

3.2 The measured speedup

TensorRT-LLM: Llama 3.3 70B FP8 on 2x H100 SXM 80GB (TP=2), ISL/OSL 1000/2000: 3,708.93 output tok/s (H100, TP=2) TensorRT-LLM: Llama 3.3 70B FP4 on 1x B200 180GB, ISL/OSL 1000/2000: 6,725.03 output tok/s (B200, TP=1)

Speedup: 1.81x — but this compares FP8 on H100 to FP4 on B200, so it blends the 2.53x FLOPS increase with the 2x FP4-vs-FP8 precision gain and the TP=2→TP=1 simplification. The records do not decompose the three.

3.3 The cost-per-token comparison

B200's $/Mtok is 1.78x worse than H100's for 70B serving. The single-GPU simplification (TP=1 vs TP=2) saves networking and complexity but not cost.

3.4 When B200 is worth it

• You need FP4 precision. B200 is the only part with a native FP4 tensor-core path (NVFP4 (Blackwell block-scaled E2M1 + E4M3): native_support is Blackwell-only). If your quality target met FP4, B200 is the only option.
• You are serving a model that fits on 186 GB but not on 80 GB at your target precision. A 70B model in FP4 needs ~35 GB of weights — fits on a single B200 with room for KV. On H100 you need TP=2 for the same model in FP8.
• You are compute-bound at high batch and need the 2.53x FLOPS increase. Above the ridge point, B200's compute advantage is real and measurable.

3.5 When B200 is not worth it

• You are serving small models at batch 1. The launch overhead dominates and the extra FLOPS and bandwidth are wasted.
• Your workload is memory-bandwidth-bound on weights, not compute-bound. B200's bandwidth increase (2.39x) is real but costs 3.2x per hour. If you are already bandwidth-bound on H100, the upgrade is worth it; if you are launch-bound, it is not.

4. A100 → H100: the clear upgrade

4.1 What changes

4.2 The price comparison

H100 is 3.17x the FLOPS for 2.0-3.0x the price. At AWS on-demand the $/FLOP is 1.58x better on H100. At Azure it is 1.0x (no advantage). The upgrade is clearly worth it at AWS, marginally worth it at Azure.

4.3 The FP8 factor

A100 has no native FP8 tensor-core path (FP8 E4M3 (per-tensor / per-block scaled): "NOT native on Ampere"). H100 has 1,979 dense FP8 TFLOPS. If your workload can use FP8, the effective FLOPS ratio is 6.3x (3.17x bf16 × 2x fp8), and the upgrade is worth it at any price ratio below 6.3x.

5. The migration checklist

Before upgrading, answer these questions with data:

1. What is my binding constraint? If it is memory capacity, only a capacity upgrade (H200, B200) helps. If it is compute, only a FLOPS upgrade (B200) helps. If it is launch overhead at batch 1, nothing helps (Per-kernel launch and dispatch overhead at batch 1).
2. What is my current utilization? If below 50%, the idle power floor (Modeled US volume-server idle/maximum power dynamic range: 0.44 minimum by 2020 from measured SERT data, 0.28 in the Best Practices scenario, with an explicit ~20-25% idle-power latency floor: 0.44 of max) means you are already paying for capacity you don't use. Upgrade the workload before upgrading the hardware.
3. Does the new part change my $/Mtok? Compute it: price_per_hour / (tok/s × 3600) × 1e6. If the answer is worse, the upgrade is a capacity decision, not a cost decision.
4. Does my engine support the new part? vLLM supports ROCm but with specific caveats. TensorRT-LLM is CUDA-only. Check 03-engine-selection.md before assuming the new part works.
5. Does my quantization format have a hardware path on the new part? FP4 is Blackwell-only. FP8 is Hopper+. Check 04-quant-selection.md before assuming the format carries over.

Related documents

• 08-cost-per-token.md — the $/Mtok arithmetic used throughout.
• 01-hardware-selection.md — the per-workload hardware table.
• 02-flop-map.md §3.1 — the ridge points that determine what binds.