inference/research
verifiedmedium · 0.70updated 2026-10-10

Embedding lookup and output projection over the vocabulary

memory bound

Fields

id
flops/embedding
Class
embedding
Bound by
memory
Arithmetic intensity
Input embedding is a pure gather: ~0 flops against hidden_width*2 bytes read per token, so AI is effectively zero and it is strictly a memory-access pattern problem. The output side (LM head, untied) is a real GEMM: flops = 2*hidden_width*V against bytes = 2*hidden_width*V + 2*V, giving AI ~1 flop/byte at batch 1 - the same regime as decode_gemm, because it is literally a weight matrix of vocab_width. Tied embeddings mean one weight matrix serves both directions, but the lookup and the GEMM pull opposite access patterns: random row gather versus dense streaming. A batched input-embedding gather across B tokens has AI still ~0 - bytes scale with B exactly as flops do (there are no flops).
Scales with
vocab_sizehidden_widthbatch_size (gather bytes scale linearly; AI unchanged)tying (whether one matrix serves lookup and projection)
Workarounds
Keep the embedding table in the fastest memory tier and co-locate it with the sampler that reads the logitsFused embedding gather that avoids a separate gather-then-copy passQuantize the embedding / LM-head matrix - it is a large weight matrix and behaves like any other decode GEMMCPU-resident or unified-memory embeddings for small vocabularies where the GPU table would not be reused enough to pay for its footprint
notes
Design consequence: this class is why a decoding engine's per-token time does not scale down with model optimizations that only touch transformer layers. On small models, especially small-vocab small-hidden ones, the embedding and LM head are a non-trivial fraction of the bandwidth budget. It is also the class most sensitive to memory tiering rather than to FLOPs: the input gather is a scattered read whose latency, not bandwidth, sets the cost, so parts with large caches and good random-access latency win. On unified-memory parts (Apple silicon, AMD APU designs) embedding tables and small LM heads can be shared with the host rather than duplicated, which matters when VRAM is the scarce resource. DANGLING REF, LEFT UNRESOLVED (2026-10-04 cross-reference repair): affected_by_hardware carries 'amd-ryzen-strix-point', which is not a record id. The nearest record is amd-radeon-8060s-ryzen-ai-max, whose architecture is 'rdna3.5 (Strix Halo iGPU, gfx1151)' - Strix HALO, a different product from Strix POINT (the laptop/mobile APU), so rewriting it there would assert the wrong silicon. The record's own note only claims dependence on 'AMD APU designs' generically. Left unchanged; the honest options are adding the missing accelerator record or dropping the token. === RE-EXAMINED 2026-10-10 (w10-xref-consistency) - STILL DANGLING, AND THE DECISION ABOVE STANDS. === Re-checked against data/accelerators/: 'amd-ryzen-strix-point' still resolves to nothing, and it is the ONLY non-architecture dangling value in all 609 affected_by_hardware values across data/flops/ (the other 8 are gfx942/gfx950/gfx1151, which are ROCm LLVM target names rather than products and remain budgeted debt). Two options were tested again and both fail: (1) REPOINT to amd-radeon-8060s-ryzen-ai-max - rejected. That record is Strix HALO (RDNA 3.5 iGPU, gfx1151, Ryzen AI Max+ 395); Strix POINT is gfx1150, the laptop/mobile APU, with a different GPU, different memory and a different ROCm target. The 2026-10-04 note already establishes the gap and it has not closed. Repointing would assert silicon this record does not claim. (2) DELETE the token - not chosen, because this record's unified-memory argument names 'AMD APU designs' and no APU record is carried anywhere else in affected_by_hardware, so deletion would silently drop the only pointer to that hardware class. The correct fix is an accelerator record for Strix Point, which is a data/accelerators/ write and outside this wave's scope; it is recorded here so a later pass does not re-derive it. No number in this record changed.

Notes

Design consequence: this class is why a decoding engine's per-token time does not scale down with model optimizations that only touch transformer layers. On small models, especially small-vocab small-hidden ones, the embedding and LM head are a non-trivial fraction of the bandwidth budget. It is also the class most sensitive to memory tiering rather than to FLOPs: the input gather is a scattered read whose latency, not bandwidth, sets the cost, so parts with large caches and good random-access latency win. On unified-memory parts (Apple silicon, AMD APU designs) embedding tables and small LM heads can be shared with the host rather than duplicated, which matters when VRAM is the scarce resource. DANGLING REF, LEFT UNRESOLVED (2026-10-04 cross-reference repair): affected_by_hardware carries 'amd-ryzen-strix-point', which is not a record id. The nearest record is amd-radeon-8060s-ryzen-ai-max, whose architecture is 'rdna3.5 (Strix Halo iGPU, gfx1151)' - Strix HALO, a different product from Strix POINT (the laptop/mobile APU), so rewriting it there would assert the wrong silicon. The record's own note only claims dependence on 'AMD APU designs' generically. Left unchanged; the honest options are adding the missing accelerator record or dropping the token.

=== RE-EXAMINED 2026-10-10 (w10-xref-consistency) - STILL DANGLING, AND THE DECISION ABOVE STANDS. === Re-checked against data/accelerators/: 'amd-ryzen-strix-point' still resolves to nothing, and it is the ONLY non-architecture dangling value in all 609 affected_by_hardware values across data/flops/ (the other 8 are gfx942/gfx950/gfx1151, which are ROCm LLVM target names rather than products and remain budgeted debt). Two options were tested again and both fail: (1) REPOINT to amd-radeon-8060s-ryzen-ai-max - rejected. That record is Strix HALO (RDNA 3.5 iGPU, gfx1151, Ryzen AI Max+ 395); Strix POINT is gfx1150, the laptop/mobile APU, with a different GPU, different memory and a different ROCm target. The 2026-10-04 note already establishes the gap and it has not closed. Repointing would assert silicon this record does not claim. (2) DELETE the token - not chosen, because this record's unified-memory argument names 'AMD APU designs' and no APU record is carried anywhere else in affected_by_hardware, so deletion would silently drop the only pointer to that hardware class. The correct fix is an accelerator record for Strix Point, which is a data/accelerators/ write and outside this wave's scope; it is recorded here so a later pass does not re-derive it. No number in this record changed.

Cross-references

Hardware-dependent (3)