inference/research
verifiedhigh · 0.90updated 2026-10-05

MI300X 192GB HBM3 memory pooling and KV cache capacity advantage

memory bound

Fields

id
flops/amd-mi300x-memory-pooling
Class
kv_transfer
Bound by
memory
Scales with
model_sizebatch_sizesequence_lengthkv_cache_dtype
Hardware parameters
parametermemory_bandwidtheffect(5) The memory bandwidth advantage (1.59x H100) directly translates to higher decode throughput in memory-bound scenarios.basisrecordedmagnitude1.59xvalues_foramd-instinct-mi300x,amd-instinct-mi325x
Workarounds
MI300X's 192GB HBM3 (2.4x H100's 80GB) enables single-node serving of large modelsFP8 KV cache halves KV memory usage vs BF16, doubling effective batch capacityAMD recommends 8 instances of TP=1 for models <=70B to maximize memory utilization
notes
MI300X's 192GB HBM3 memory is 2.4x the NVIDIA H100's 80GB and 1.59x the bandwidth (5.3 TB/s vs 3.35 TB/s). This large memory pool is the key inference advantage: (1) It enables single-node serving of Llama-3.1 405B and DeepSeek v3/R1 (671B) without multi-node communication. (2) It supports larger KV caches for large batch sizes and long sequences, reducing evictions. (3) AMD recommends serving models <=70B in 8 instances of TP=1 (one per GPU) to maximize throughput, as this avoids inter-GPU communication overhead entirely. (4) FP8 KV cache (supported via ROCm 6.2) halves KV memory usage, doubling effective batch capacity. (5) The memory bandwidth advantage (1.59x H100) directly translates to higher decode throughput in memory-bound scenarios. MI325X increases this to 256GB HBM3E at 6 TB/s, and MI350X/MI355X to 288GB HBM3E at 8 TB/s. RESOLVED (2026-10-04 cross-reference repair): the bare part names in affected_by_hardware were expanded to their record ids - mi300x -> amd-instinct-mi300x, mi325x -> amd-instinct-mi325x. Each name matched exactly one accelerator record in data/accelerators/, so no variant was chosen between siblings; the id is forced by the part name. MI350X/MI355X are discussed in this record's notes but were deliberately NOT added to the field, since the record's subject is the 192GB CDNA3 memory pool. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.

Notes

MI300X's 192GB HBM3 memory is 2.4x the NVIDIA H100's 80GB and 1.59x the bandwidth (5.3 TB/s vs 3.35 TB/s). This large memory pool is the key inference advantage: (1) It enables single-node serving of Llama-3.1 405B and DeepSeek v3/R1 (671B) without multi-node communication. (2) It supports larger KV caches for large batch sizes and long sequences, reducing evictions. (3) AMD recommends serving models <=70B in 8 instances of TP=1 (one per GPU) to maximize throughput, as this avoids inter-GPU communication overhead entirely. (4) FP8 KV cache (supported via ROCm 6.2) halves KV memory usage, doubling effective batch capacity. (5) The memory bandwidth advantage (1.59x H100) directly translates to higher decode throughput in memory-bound scenarios. MI325X increases this to 256GB HBM3E at 6 TB/s, and MI350X/MI355X to 288GB HBM3E at 8 TB/s. RESOLVED (2026-10-04 cross-reference repair): the bare part names in affected_by_hardware were expanded to their record ids - mi300x -> amd-instinct-mi300x, mi325x -> amd-instinct-mi325x. Each name matched exactly one accelerator record in data/accelerators/, so no variant was chosen between siblings; the id is forced by the part name. MI350X/MI355X are discussed in this record's notes but were deliberately NOT added to the field, since the record's subject is the 192GB CDNA3 memory pool.

HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.

Cross-references

Hardware-dependent (2)