inference/research
verifiedmedium · 0.70updated 2026-10-08

AWS Inferentia2

amazon NeuronCore (2nd generation) 2022

Fields

id
accelerators/aws-inferentia2
Vendor
amazon
Architecture
NeuronCore (2nd generation)
Release year
2022
Process (nm)
7
Form factors
custom
VRAM (GB)
32
Memory type
hbm
Memory bandwidth (GB/s)
820
Bandwidth basis
Vendor-published per-chip figure. The AWS Inferentia product page states 32 GB HBM per chip for Inferentia2 and AWS's own framing is that Inferentia2 increased memory bandwidth 'by 10x' over gen-1 without printing a gen-1 number. The per-chip BANDWIDTH figure comes from the AWS Neuron Inferentia2 Architecture page (source w7z-neuron-inferentia2-arch-page), verbatim '32GiB of high-bandwidth device memor (HBM) (for storing model state), with 820 GiB/sec of bandwidth'. UNITS DISCREPANCY, RECORDED NOT CONVERTED (2026-10-05): the vendor prints GiB/sec and this field is named gbps. 820 GiB/s is about 881 GB/s, so the stored digits are the vendor's and the unit label does not match them, a 7.4% gap invisible to any reader who compares only the numbers. The value is left as 820 rather than recomputed to 881 because every other record in this repo stores the vendor's printed figure rather than this repo's conversion of it, and a silent conversion in one record would make the corpus inconsistent in a way that is harder to audit than a named discrepancy. THE EARLIER DERIVATION IS SUPERSEDED, not discarded: an earlier pass stored this field by dividing the Inf2 instance aggregate (12 chips, 384 GB, 9.8 TB/s) to get ~817 GB/s per chip. The first-party per-chip figure is 820 GiB/s, which independently corroborates the derivation to within its own unit ambiguity - the 817 and the 820 were the same quantity found two ways, and the first-party version wins. NO BUS WIDTH IS PUBLISHED for Inferentia on any AWS page, so memory_bus_bit is null and was not back-computed.
Peak FLOPS
precisionfp16tflops190denseyesvendor_claimyessource_idasic-aws-inferentia-productbasisvendor-spec-sheetbasis_detailTranscribed from the vendor's own fp16 for AWS Inferentia2 (source of record: 'AWS Inferentia product page', kind: spec-sheet). Recorded dense:true as printed.precisionint8tflops380denseyesvendor_claimyessource_idw7z-neuron-inferentia2-arch-pagenotesCORRECTED 2026-10-05 FROM A ROW THAT SAID precision 'fp8' WITH THIS SAME NUMBER. THIS WAS A PRECISION ERROR, NOT A TYPO, and the AWS Neuron documentation shows exactly where the figure comes from: 'Two NeuronCore-v2 cores, delivering 380 INT8 TOPS, 190 FP16/BF16/cFP8/TF32 TFLOPS, and 47.5 FP32 TFLOPS.' The 380 is INT8 TOPS. The previous row carried it as 380 FP8 TFLOPS with notes saying '380 TFLOPS FP8 is inferred as 2x the vendor's 190 TFLOPS FP16 per-chip figure. AWS does not publish a per-chip FP8 figure' and basis 'vendor-spec-sheet'. Two things were wrong at once: the basis claimed a transcription where the notes admitted an inference, and the precision was inferred where the vendor publishes the figure directly under a different label. So the number survives, the precision changes from a guess to the vendor's own, and the basis changes from 'vendor-spec-sheet' honestly mislabelled to genuinely transcribed. WHAT THE SCHEMA DOES AND DOES NOT LET THIS ROW SAY: flops[] is a TFLOPS-keyed field, so 380 INT8 TOPS is filed here as a tflops value with the unit stated in notes rather than in a unit sub-field - the same compromise the repo already makes for Tenstorrent MXINT8/MXINT4 and for Cambricon INT16, and the same compromise Qualcomm's records refuse to make (they leave flops empty rather than convert TOPS-INT8). Recording it beats refusing it because this number IS vendor-published and per-chip, unlike Qualcomm's, and dropping it would lose the only first-party per-chip INT8 figure for this part. Do NOT read it as 380 FP16 TFLOPS and do not compare it with a GPU's FP8 row: on this part INT8 and FP8 are the same execution rate (190 FP16/BF16/cFP8/TF32 vs 380 INT8 is exactly 2x, i.e. INT8 is 2x the FP16-family rate and cFP8 sits at the FP16-family rate), which is the opposite of the Hopper and Blackwell convention where FP8 is 2x FP16. dense:true records the absence of any sparsity statement, which AWS makes for this part.basisvendor-spec-sheetbasis_detailTranscribed verbatim from the AWS Neuron Inferentia2 Architecture page component table (source w7z-neuron-inferentia2-arch-page): '380 INT8 TOPS'. No derivation. This replaces a row that filed the same number as FP8 TFLOPS on an admitted 2x inference.precisionfp32tflops47.5denseyesvendor_claimyessource_idw7z-neuron-inferentia2-arch-pagenotes47.5 FP32 TFLOPS, ADDED 2026-10-05, from the same component table: '190 FP16/BF16/cFP8/TF32 TFLOPS, and 47.5 FP32 TFLOPS'. This row did not exist in the repo before this pass even though the AWS product page states Inferentia2 'adds additional support for FP32, TF32' over gen-1 - so the record asserted a precision capability with no row behind it. It is also a useful check on the precision handling in this record: 190 / 47.5 = 4x, which is the standard FP16-to-FP32 ratio and therefore confirms the 190 is a genuine per-precision figure rather than an inflated one.basisvendor-spec-sheetbasis_detailTranscribed verbatim from the AWS Neuron Inferentia2 Architecture page component table (source w7z-neuron-inferentia2-arch-page): '47.5 FP32 TFLOPS'. No derivation.
Unified memory
no
Consumer part
no
notes
The only per-chip published compute number is 190 TFLOPS FP16 (vendor page). The FP8 row at 380 TFLOPS is an INFERENCE from the 2.3 PFLOPS 'up to' instance aggregate over 12 chips, and is flagged vendor_claim false and dense because AWS never states whether that aggregate is dense or sparse; treat it as an upper bound, not a measured dense peak. Inf2 supports FP32, TF32, BF16, FP16, UINT8 and configurable FP8 (cFP8); AWS's autocast path takes high-precision FP32/FP16 models and casts them down, with stochastic rounding available to protect accuracy. Two NeuronCores per chip (versus four on gen 1) is the reason a single chip reaches 190 TFLOPS where gen 1 spread its cores wider. NeuronLink at 192 GB/s lets data move chip-to-chip without touching the CPU, which is what makes multi-chip tensor parallelism viable for models that do not fit one 32 GB part. Contrast with Trainium2: same vendor, same NeuronLink, but Inferentia2 is inference-only and 32 GB, while Trainium2 is 96 GB and mixed-use. MEMORY_BUS_WIDTH: STILL NULL, AND HERE IS PRECISELY WHAT WAS LOOKED FOR (2026-10-04). AWS publishes NO memory-bus width for Inferentia or Trainium. NO FIGURE WAS BACK-COMPUTED FROM BANDWIDTH. That is the standing rule for this field in this repo and it was applied to every null here rather than to a subset: a width derived by dividing a published aggregate bandwidth by an assumed or half-assumed per-pin rate is two inferences stacked on one number, and if either inference is wrong the record looks populated while being wrong. The correct state of this field is 'the vendor does not publish it', and the note above names the specific document that was read and found not to carry it, so a future pass does not have to re-establish that from scratch. MEMORY_BUS_WIDTH: STILL NULL, AND HERE IS PRECISELY WHAT WAS LOOKED FOR (2026-10-04). AWS publishes no memory-bus width for Inferentia or Trainium. The Neuron hardware documentation and the EC2 instance-type pages give capacity and aggregate bandwidth only. Not back-computed: Trainium2's published figure is a per-instance aggregate across a 16-chip pod, not a per-chip number, so dividing it by chip count yields a pod figure rather than a bus width. NO FIGURE WAS BACK-COMPUTED FROM BANDWIDTH. That is the standing rule for this field in this repo and it was applied to every null here rather than to a subset: a width derived by dividing a published aggregate bandwidth by an assumed or half-assumed per-pin rate is two inferences stacked on one number, and if either is wrong the record looks populated while being wrong. The correct state of this field is 'the vendor does not publish it', and the text above names the specific document read and found not to carry it, so a future pass does not have to re-establish that from scratch. === PRECISION ERROR FOUND AND CORRECTED 2026-10-05 (non-GPU classification audit, w7-nonxpu-computers). === The fp8 row at 380 TFLOPS has been REPLACED, not deleted and not left alone. See the row's own notes for the full argument; the summary is that the vendor publishes 380 as INT8 TOPS, the record filed it as 380 FP8 TFLOPS, and the row's basis said 'vendor-spec-sheet' while its own notes said 'AWS does not publish a per-chip FP8 figure; this is a direct inference'. The number is unchanged; the precision, the basis and the source_id are all now the vendor's own. A 47.5 FP32 TFLOPS row was added from the same first-party table. A UNITS DISCREPANCY THAT IS RECORDED RATHER THAN SILENTLY CONVERTED, and which affects memory_bandwidth_gbps directly: the Neuron Inferentia2 page prints the device-memory bandwidth as '820 GiB/sec of bandwidth', while this record stores 820 in a field named memory_bandwidth_gbps. 820 GiB/s is about 881 GB/s. The stored NUMBER is the vendor's, the field's UNIT LABEL does not match it, and this is a 7.4% discrepancy that a naive reader would never see because the digits are identical. It was left as 820 rather than converted to 881 because converting would substitute this repo's arithmetic for the vendor's printed figure in a field that other records fill with the vendor's own unit; the discrepancy is named here so nobody treats 820 as exact. Note the sibling AWS pages are inconsistent with each other on this: the Inferentia2 product page describes the same hardware as '32 GB of HBM per chip' and says Inferentia2 increased memory 'by 10x over Inferentia' while never printing a bandwidth number. WHAT WAS CHECKED IN THIS RECORD AND FOUND CORRECT, stated because a negative finding is worth as much as a positive one in an audit: vram_gb 32, memory_type hbm, memory_bus_bit null, tdp_w null. The 32 GB HBM is a real off-package tier behind a real interface, so these are the GPU-shaped fields holding GPU-shaped quantities and no false friend is present - which is the opposite of cerebras, groq and d-matrix and is why this part needed no structural fix. The 190 TFLOPS FP16 figure remains the vendor's own per-chip number. memory_bus_bit stays null and unback-computed: AWS publishes no bus width for any Neuron generation. A STANDING CAUTION THIS RECORD NOW CARRIES, because the corrected row shows the failure mode and it is an easy one to repeat: an inferred figure filed under a precision the vendor never claimed, with a basis field asserting a provenance the notes contradict. The basis field is what makes that detectable, and it detected it here. Any future Neuron row should state which of the vendor's three published quantities (380 INT8 TOPS, 190 FP16/BF16/cFP8/TF32 TFLOPS, 47.5 FP32 TFLOPS) it is transcribing, rather than deriving one from another. === 380 RELABELLED FP8 TFLOPS -> INT8 TOPS: RE-VERIFIED 2026-10-08 (w8-reverify) AND CONFIRMED. === The AWS Neuron Inferentia2 Architecture page was re-read live 2026-10-08 and its component table states, verbatim: Compute 'Two NeuronCore-v2 cores, delivering 380 INT8 TOPS, 190 FP16/BF16/cFP8/TF32 TFLOPS, and 47.5 FP32 TFLOPS.' So 380 is INT8 TOPS, the vendor publishes it directly under that label, and the previous row's own note ('380 TFLOPS FP8 is inferred as 2x the vendor's 190 TFLOPS FP16 per-chip figure. AWS does not publish a per-chip FP8 figure') was an admitted inference filed with basis 'vendor-spec-sheet' - a transcription claim over an inference, which is the exact misrepresentation the basis field exists to prevent. The number was right and the label and provenance were both wrong; the correction changed the precision and the basis, not the value. THE INTERNAL CONSISTENCY CHECK ON THE VENDOR'S OWN TABLE, which is what makes this readable rather than merely transcribed: 190 FP16/BF16/cFP8/TF32 : 380 INT8 : 47.5 FP32 = 4 : 8 : 1. INT8 is exactly 2x the FP16-family rate, and FP32 is exactly a quarter of it, so the three figures form a coherent single ladder from one table rather than three unrelated numbers. This also settles the comparison rule the record states: on this part cFP8 sits at the FP16-family rate, which is the OPPOSITE of the Hopper/Blackwell convention where FP8 is 2x FP16 - so comparing a GPU FP8 row against this part's FP16 row is not a like-for-like comparison and comparing it against this part's INT8 row is worse. ONE THING THIS RE-VERIFICATION FIXED, because a re-verification that finds nothing is a claim and not a result: the two corrected rows carried source_id 'w7z-neuron-inferentia2-arch-page' - the page that makes the correction possible - while that source was absent from the record's sources[] list. The figure had per-row provenance that the record-level source list did not back. Added. AGENTS.md requires every cited source to resolve and it did, but the converse is the intent: a source a record cites is a source it must declare. NO NUMBER CHANGED.

Notes

The only per-chip published compute number is 190 TFLOPS FP16 (vendor page). The FP8 row at 380 TFLOPS is an INFERENCE from the 2.3 PFLOPS 'up to' instance aggregate over 12 chips, and is flagged vendor_claim false and dense because AWS never states whether that aggregate is dense or sparse; treat it as an upper bound, not a measured dense peak. Inf2 supports FP32, TF32, BF16, FP16, UINT8 and configurable FP8 (cFP8); AWS's autocast path takes high-precision FP32/FP16 models and casts them down, with stochastic rounding available to protect accuracy. Two NeuronCores per chip (versus four on gen 1) is the reason a single chip reaches 190 TFLOPS where gen 1 spread its cores wider. NeuronLink at 192 GB/s lets data move chip-to-chip without touching the CPU, which is what makes multi-chip tensor parallelism viable for models that do not fit one 32 GB part. Contrast with Trainium2: same vendor, same NeuronLink, but Inferentia2 is inference-only and 32 GB, while Trainium2 is 96 GB and mixed-use. MEMORY_BUS_WIDTH: STILL NULL, AND HERE IS PRECISELY WHAT WAS LOOKED FOR (2026-10-04). AWS publishes NO memory-bus width for Inferentia or Trainium. NO FIGURE WAS BACK-COMPUTED FROM BANDWIDTH. That is the standing rule for this field in this repo and it was applied to every null here rather than to a subset: a width derived by dividing a published aggregate bandwidth by an assumed or half-assumed per-pin rate is two inferences stacked on one number, and if either inference is wrong the record looks populated while being wrong. The correct state of this field is 'the vendor does not publish it', and the note above names the specific document that was read and found not to carry it, so a future pass does not have to re-establish that from scratch.

MEMORY_BUS_WIDTH: STILL NULL, AND HERE IS PRECISELY WHAT WAS LOOKED FOR (2026-10-04). AWS publishes no memory-bus width for Inferentia or Trainium. The Neuron hardware documentation and the EC2 instance-type pages give capacity and aggregate bandwidth only. Not back-computed: Trainium2's published figure is a per-instance aggregate across a 16-chip pod, not a per-chip number, so dividing it by chip count yields a pod figure rather than a bus width. NO FIGURE WAS BACK-COMPUTED FROM BANDWIDTH. That is the standing rule for this field in this repo and it was applied to every null here rather than to a subset: a width derived by dividing a published aggregate bandwidth by an assumed or half-assumed per-pin rate is two inferences stacked on one number, and if either is wrong the record looks populated while being wrong. The correct state of this field is 'the vendor does not publish it', and the text above names the specific document read and found not to carry it, so a future pass does not have to re-establish that from scratch.

=== PRECISION ERROR FOUND AND CORRECTED 2026-10-05 (non-GPU classification audit, w7-nonxpu-computers). === The fp8 row at 380 TFLOPS has been REPLACED, not deleted and not left alone. See the row's own notes for the full argument; the summary is that the vendor publishes 380 as INT8 TOPS, the record filed it as 380 FP8 TFLOPS, and the row's basis said 'vendor-spec-sheet' while its own notes said 'AWS does not publish a per-chip FP8 figure; this is a direct inference'. The number is unchanged; the precision, the basis and the source_id are all now the vendor's own. A 47.5 FP32 TFLOPS row was added from the same first-party table. A UNITS DISCREPANCY THAT IS RECORDED RATHER THAN SILENTLY CONVERTED, and which affects memory_bandwidth_gbps directly: the Neuron Inferentia2 page prints the device-memory bandwidth as '820 GiB/sec of bandwidth', while this record stores 820 in a field named memory_bandwidth_gbps. 820 GiB/s is about 881 GB/s. The stored NUMBER is the vendor's, the field's UNIT LABEL does not match it, and this is a 7.4% discrepancy that a naive reader would never see because the digits are identical. It was left as 820 rather than converted to 881 because converting would substitute this repo's arithmetic for the vendor's printed figure in a field that other records fill with the vendor's own unit; the discrepancy is named here so nobody treats 820 as exact. Note the sibling AWS pages are inconsistent with each other on this: the Inferentia2 product page describes the same hardware as '32 GB of HBM per chip' and says Inferentia2 increased memory 'by 10x over Inferentia' while never printing a bandwidth number. WHAT WAS CHECKED IN THIS RECORD AND FOUND CORRECT, stated because a negative finding is worth as much as a positive one in an audit: vram_gb 32, memory_type hbm, memory_bus_bit null, tdp_w null. The 32 GB HBM is a real off-package tier behind a real interface, so these are the GPU-shaped fields holding GPU-shaped quantities and no false friend is present - which is the opposite of cerebras, groq and d-matrix and is why this part needed no structural fix. The 190 TFLOPS FP16 figure remains the vendor's own per-chip number. memory_bus_bit stays null and unback-computed: AWS publishes no bus width for any Neuron generation. A STANDING CAUTION THIS RECORD NOW CARRIES, because the corrected row shows the failure mode and it is an easy one to repeat: an inferred figure filed under a precision the vendor never claimed, with a basis field asserting a provenance the notes contradict. The basis field is what makes that detectable, and it detected it here. Any future Neuron row should state which of the vendor's three published quantities (380 INT8 TOPS, 190 FP16/BF16/cFP8/TF32 TFLOPS, 47.5 FP32 TFLOPS) it is transcribing, rather than deriving one from another.

=== 380 RELABELLED FP8 TFLOPS -> INT8 TOPS: RE-VERIFIED 2026-10-08 (w8-reverify) AND CONFIRMED. === The AWS Neuron Inferentia2 Architecture page was re-read live 2026-10-08 and its component table states, verbatim: Compute 'Two NeuronCore-v2 cores, delivering 380 INT8 TOPS, 190 FP16/BF16/cFP8/TF32 TFLOPS, and 47.5 FP32 TFLOPS.' So 380 is INT8 TOPS, the vendor publishes it directly under that label, and the previous row's own note ('380 TFLOPS FP8 is inferred as 2x the vendor's 190 TFLOPS FP16 per-chip figure. AWS does not publish a per-chip FP8 figure') was an admitted inference filed with basis 'vendor-spec-sheet' - a transcription claim over an inference, which is the exact misrepresentation the basis field exists to prevent. The number was right and the label and provenance were both wrong; the correction changed the precision and the basis, not the value. THE INTERNAL CONSISTENCY CHECK ON THE VENDOR'S OWN TABLE, which is what makes this readable rather than merely transcribed: 190 FP16/BF16/cFP8/TF32 : 380 INT8 : 47.5 FP32 = 4 : 8 : 1. INT8 is exactly 2x the FP16-family rate, and FP32 is exactly a quarter of it, so the three figures form a coherent single ladder from one table rather than three unrelated numbers. This also settles the comparison rule the record states: on this part cFP8 sits at the FP16-family rate, which is the OPPOSITE of the Hopper/Blackwell convention where FP8 is 2x FP16 - so comparing a GPU FP8 row against this part's FP16 row is not a like-for-like comparison and comparing it against this part's INT8 row is worse. ONE THING THIS RE-VERIFICATION FIXED, because a re-verification that finds nothing is a claim and not a result: the two corrected rows carried source_id 'w7z-neuron-inferentia2-arch-page' - the page that makes the correction possible - while that source was absent from the record's sources[] list. The figure had per-row provenance that the record-level source list did not back. Added. AGENTS.md requires every cited source to resolve and it did, but the converse is the intent: a source a record cites is a source it must declare. NO NUMBER CHANGED.

Cross-references

Interconnect (2)