AI Inference Silicon

Dedicated application-specific processors and accelerators engineered for high-throughput, low-latency AI model inference in data centers and edge deployments. Architectures overcome the memory wall by utilizing massive on-chip SRAM, high-bandwidth memory (HBM3e), and specialized matrix processing engines. Commercial implementations include Etched Sohu ASICs, Groq Language Processing Units (LPUs), Tenstorrent Wormhole, and Cerebras CS-3 systems.

Key technical architectures and manufacturing innovations include:

• NVIDIA Blackwell Platform: Full-stack accelerated computing platform (GPU architecture, NVLink, networking, software) for trillion-parameter generative AI training and inference. Succeeds Hopper; named for mathematician David Harold Blackwell. Consumer GeForce RTX 50 Series (RTX 5090 CES 2025-01-06) is the same architecture family.
• NVIDIA Hopper Architecture: NVIDIA data-center GPU architecture announced GTC 2022-03-22, succeeding Ampere. First SKU H100 (Transformer Engine, HBM3); H200 (2023-11-13) is the HBM3e memory refresh. Still the training fleet for Llama 3.1-class models; Ornn still prices H100 SXM and H200 as free-tier OCPI GPUs in Aug 2026.
• NVIDIA A100 Tensor Core GPU: Ampere-architecture data-center GPU in full production and shipping 2020-05-14 (>54 billion transistors, 7nm, MIG, third-gen NVLink, TF32). Meta pretrained Llama 2 on A100-80GB clusters (3.3M GPU hours; arXiv 2307.09288, 2023-07-18). Ornn OCPI A100 SXM4 $1.06/hr as of 2026-08-22T14:02:53Z — cheapest data-center SKU on the free-tier index because a large depreciated Ampere fleet still runs fine-tune, inference, and smaller training jobs after Hopper/Blackwell took frontier pretrain.
• NVIDIA H100 Tensor Core GPU: Hopper-architecture data-center GPU announced GTC 2022-03-22 (80 billion transistors, Transformer Engine, first HBM3). Full production 2022-09-20. Meta trained Llama 3.1 405B on over 16 thousand H100 GPUs (Meta blog, 2024-07-23). Ornn OCPI H100 SXM $2.85/hr as of 2026-08-22T14:02:53Z. SemiAnalysis InferenceX (NVIDIA DGX B200 page, as of Apr 2026) cites Hopper inference at about $0.09/MTok vs Blackwell $0.02/MTok on GPT-OSS-120B — remaining financial impact is the installed training/inference fleet, not leading token TCO.
• NVIDIA GeForce RTX 5090: Consumer Blackwell GeForce GPU announced CES 2025-01-06 (92 billion transistors, 3,352 AI TOPS, 32GB GDDR7, 21,760 CUDA cores, 575W TGP) and available 2025-01-30 at $1,999. NVIDIA markets it for local LLM inference (LM Studio, AnythingLLM) and NVIDIA's Oct 23, 2025 Unsloth blog documents QLoRA fine-tune of Llama 3.1 8B-class models on a single 5090, with a claim that a single Blackwell GPU can fine-tune up to 40B-parameter models. No sourced evidence that any frontier/SOTA foundation model was pretrained on RTX 5090 — 32GB GDDR7 is not an HBM training SKU. Ornn OCPI $0.53/hr hourly (2026-08-22T14:02:53Z) and $0.52/hr settled daily (2026-08-21) — cheapest of the five free-tier GPUs, reflecting consumer/local-AI and small-job rental rather than cluster pretrain.
• Etched frontier inference cluster: Rack-scale inference cluster using Low Voltage Inference (LVI) for compute density at the same power and Cluster Scale Memory (CSM), a shared memory pool across the cluster. First customer Jane Street; >$1B in contracts claimed.

Leading industrial manufacturers, hyperscalers, and engineering operators include:

• NVIDIA Corporation: Meta Llama 3.1 405B trained on over 16 thousand H100 GPUs (Jul 23, 2024) — To enable training runs at this scale and achieve the results we have in a reasonable amount of time, we significantly optimized our full training stack and pushed our model training to over 16 thousand H100 GPUs, making the 405B the first Llama model trained at this scale.
• Meta: Meta MTIA 300 presented at ISCA 2026 — We present MTIA 300, Meta's first AI training chip optimized for Deep Learning Recommendation Models (DLRMs).
• Intel: Vector Core fully disaggregated inference demonstration (Jun 2, 2026) — Xeon 6 orchestration, SambaNova SN40 decode, NVIDIA Blackwell prefill from Vector Core LA data center; Together.ai first commercial customer.
• OpenAI: US Patent Office Publishes OpenAI Patent US20260147536A1 on Hardware Alignment Accelerators (May 28, 2026) — Techniques for data alignment in hardware accelerators, including aligning mantissa bits and exponent scaling within compute-in-memory (CIM) macro arrays for high-efficiency neural network inference.
• Broadcom: OpenAI Unpacks Jalapeño Custom LLM Inference ASIC at Hot Chips 2026 — Jalapeño is OpenAI's custom inference ASIC co-designed with Broadcom in a nine-month RTL-to-tapeout development cycle. Optimized for LLM inference efficiency and throughput per watt to overcome data center power constraints.
• Etched: Etched delivers first inference rack to Jane Street (Aug 18, 2026) — Etched shipped its first rack last month to Jane Street, and the quantitative trading firm is actively deploying the technology into its workloads.

Major industrial breakthroughs and commercial milestones include:

• NVIDIA A100 Ampere GPU in full production (GTC 2020): NVIDIA today announced that the first GPU based on the NVIDIA Ampere architecture, the NVIDIA A100, is in full production and shipping to customers worldwide.
• NVIDIA Hopper architecture and H100 announced (GTC 2022): Named for Grace Hopper, a pioneering U.S. computer scientist, the new architecture succeeds the NVIDIA Ampere architecture, launched two years ago.
• NVIDIA Blackwell platform and GB200 NVL72 announcement (GTC 2024): NVIDIA today announced that the NVIDIA Blackwell platform has arrived — enabling organizations everywhere to build and run real-time generative AI on trillion-parameter large language models at up to 25x less cost and energy consumption than its predecessor.
• Meta Llama 3.1 405B trained on over 16 thousand H100 GPUs (Jul 23, 2024): To enable training runs at this scale and achieve the results we have in a reasonable amount of time, we significantly optimized our full training stack and pushed our model training to over 16 thousand H100 GPUs, making the 405B the first Llama model trained at this scale.
• US Patent Office Publishes OpenAI Patent US20260147536A1 on Hardware Alignment Accelerators (May 28, 2026): Techniques for data alignment in hardware accelerators, including aligning mantissa bits and exponent scaling within compute-in-memory (CIM) macro arrays for high-efficiency neural network inference.
• OpenAI Unpacks Jalapeño Custom LLM Inference ASIC at Hot Chips 2026: Jalapeño is OpenAI's custom inference ASIC co-designed with Broadcom in a nine-month RTL-to-tapeout development cycle. Optimized for LLM inference efficiency and throughput per watt to overcome data center power constraints.