← All posts

The 0.01% Guide to AI Infrastructure: Understanding LLM Internals and GPU Execution

2026-05-06

Originally published on LinkedIn.

Generative AI platforms like ChatGPT, Perplexity, and Gemini have become ubiquitous, fundamentally transforming the modern technological landscape. Across industries, enterprises are deploying AI at an unprecedented scale, integrating Large Language Models (LLMs) into complex data pipelines and establishing sophisticated MLOps practices. Today, specialized tech organizations are fine-tuning foundational models to tackle domain-specific challenges in healthcare, law, and finance, while developers globally have embraced AI-powered coding interfaces to accelerate their workflows.

Yet, despite this massive wave of adoption, a critical knowledge gap remains. For most, the underlying mechanics of these systems are treated as a black box. Very few professionals truly grasp the internal intricacies of LLM architecture—such as embedding spaces, multi-head attention, and Mixture of Experts (MoE) routing. Even fewer understand the complex AI infrastructure required to load these massive models into GPU clusters for optimized inference.

This article bridges that gap. By breaking down the complete lifecycle—from how text tokens are processed in neural network layers to how matrix multiplications are executed at the hardware level—this guide will demystify the system architecture. By the end, you will join the 0.01% of professionals who truly understand the robust engineering and infrastructure that makes modern AI possible.

From Theory to Silicon: Introducing GPT-oss 120B

This article breaks down how a reference large language model such as GPT-oss 120B works, from its internal Transformer architecture to the way it is packaged, stored, loaded from disk, and executed on GPU compute infrastructure.

It is aimed at readers who already have a basic understanding of LLMs, although anyone with a general technical background should be able to follow by reviewing a few concepts as they go.

GPT-oss 120B is built on a Transformer architecture that combines attention mechanisms with a Mixture of Experts (MoE) design. Instead of engaging the full network for every token, the model selectively activates specialized experts, increasing inference efficiency and enabling large-scale deployment on modern GPU systems.

The GPT-oss 120B model architecture

The GPT-oss 120B model architecture

Step 1: Tokenization and Embeddings

Before the model can process text, it must convert human language into a mathematical format. The input sentence is divided into discrete segments called text tokens, which are assigned unique numerical IDs. A word lookup table then maps these IDs into dense, high-dimensional vector embeddings—in this architecture, an array of 8,192 numbers per token. This step captures the foundational semantic meaning of each word before it enters the computational pipeline.

Step 2: The Transformer Layers

The core engine of the model is a sequential tower of 36 transformer layers where the actual "reasoning" occurs. As the token embeddings pass through each layer sequentially, they accumulate deeper contextual understanding through three primary mechanisms:

Step 3: The Language Model Head

After the token vector has traversed all 36 layers, the highly enriched mathematical representation reaches the final stage: the Language Model (LM) Head. This component translates the 8,192-dimensional vector back into human language by calculating next-word probabilities. It outputs a distribution chart across the model's entire vocabulary, allowing the system to accurately predict and generate the most statistically likely next token.

Mapping Architecture to Disk: The Safetensors Format

Safetensors and disk storage

Safetensors and disk storage

Understanding the abstract architecture of GPT-oss 120B is only half the equation. To deploy the model, those mathematical structures must be serialized and stored on physical media (NVMe/SSDs). Modern LLMs utilize the Safetensors format, which organizes the model’s massive parameter count into secure, zero-copy memory-mapped files designed for rapid loading. Because a 120-billion-parameter model is too large for a single file, it is broken down, or "sharded," into logical components.

The Storage Sharding Strategy

When you download an LLM to disk, you are downloading a structured directory of these Safetensors shards. For GPT-oss 120B, the model maps perfectly from its architectural stack to roughly 38 specific files:

Exploded View: Inside a Single Layer Shard

If we inspect the internal payload of a single shard (such as Layer 0), we find the raw multi-dimensional matrices required for GPU compute:

By sharding the model this way, the system can selectively memory-map (mmap) individual layers into RAM and GPU VRAM precisely when needed, rather than attempting to load a monolithic 115 GB file into memory all at once.

The Inference and training Pipeline: From Storage to Silicon

Single-node inference pipeline

Single-node inference pipeline

Having mapped the LLM architecture into discrete Safetensors shards on disk, the next critical infrastructure challenge is moving those massive datasets into the execution environment. This explanation applies both for single-user inference and for training at any stage. Both use cases require a carefully orchestrated pipeline that shuttles data from slow non-volatile storage into ultra-fast GPU memory, and ultimately into the compute cores.

The Megatron Bridge: CPU to GPU Transfer

The journey begins at the NVMe/SSD storage level. The CPU orchestrates the loading of the Safetensors model shards into system RAM. However, simply copying data to the GPU is not enough for massive models like GPT-oss 120B.

Weight Buffering in HBM3e VRAM

Once across the PCIe bus, the model data lands in the GPU's High Bandwidth Memory (HBM3e VRAM).

Execution: L2 Cache to Streaming Multiprocessors (SMs)

VRAM is fast, but execution requires the data to be placed as close to the compute units as possible. During inference, the specific weight blocks required for the current token are pulled from VRAM into the GPU's L2 Cache.

By pipeline-lining this process—streaming data from NVMe to RAM, across the Megatron Bridge to VRAM, through the caches, and into the Tensor Cores—modern AI infrastructure can sustain the massive throughput required to generate text in real time.

Scaling Up: Multi-GPU Inference on an NVIDIA DGX H200 Node

Multi-GPU inference pipelijne (DGX H200)

Multi-GPU inference pipelijne (DGX H200)

While the internal architecture of an LLM and the loading process into a single GPU are complex, running a massive model like GPT-oss 120B in production requires scaling across multiple GPUs. A single GPU, even a powerful one, often lacks the VRAM to hold the entire model or the compute power to serve tokens at acceptable latency. To solve this, enterprise architectures rely on nodes like the NVIDIA DGX, which houses 8 high-performance GPUs (in this case, H200 SXM5 modules) working as a single unified engine.

Breaking Down the Workload: Parallelism Strategies

To make 8 GPUs act as one, the model's workload and memory must be divided. The system employs two primary strategies to distribute the massive parameters of GPT-oss 120B: Tensor Parallelism and Pipeline Parallelism.

  1. Tensor Parallelism (TP): Splitting the Math Instead of giving one GPU the entire matrix multiplication task, Tensor Parallelism slices the mathematical operations vertically (column-wise matrix sharding).
  1. Pipeline Parallelism (PP): An Assembly Line for Layers While TP slices the matrices, Pipeline Parallelism divides the sequence of the model's 36 layers across different GPUs, creating a sequential assembly line.

Advanced Deep Dive (Optional Reading): The Synergy of Tensor and Pipeline Parallelism

When you combine Tensor Parallelism (TP) and Pipeline Parallelism (PP)—often called "3D Parallelism" when combined with Data Parallelism—they work together in a grid-like fashion.

Imagine you have an 8-GPU node and you configure it with:

Here is exactly how the GPUs are assigned:

  1. Stage 1 of the Pipeline (Layers 0–17): This entire block of layers is assigned to a "TP Group" consisting of GPUs 0, 1, 2, and 3.
  2. Stage 2 of the Pipeline (Layers 18–35): This second block of layers is assigned to the second "TP Group" consisting of GPUs 4, 5, 6, and 7.

The Flow of Execution

As data moves through the system, here is what happens:

Why do both?

Tensor Parallelism requires GPUs to talk to each other constantly (often multiple times per layer) to share partial matrix math results. This requires extreme bandwidth, which is why it is almost exclusively done inside a single server using NVLink.

Pipeline Parallelism only requires GPUs to talk to each other once per stage (just passing the result of Layer 17 to Layer 18). Because the communication requirement is much lower, PP is often used to bridge across different servers (e.g., Server A handles the first half of the model, Server B handles the second half).

By combining them, you keep the ultra-fast TP math confined to GPUs physically wired together, while using PP to scale the model's depth across more hardware without causing a massive network bottleneck

The NVLink Fabric: The Backbone of Synchronization

Because Tensor Parallelism requires constant sharing of partial results, traditional PCIe connections are far too slow. This is where the NVLink 5.0 Fabric becomes critical.

VRAM Layout and Distributed MoE

Inside each of the 8 H200 GPUs, the 141GB of HBM3e VRAM is carefully partitioned. A portion holds shared embeddings, another holds the specific sharded layer matrices assigned to that GPU, and a critical segment is reserved for the KV Cache (which remembers the context of the user's prompt).

Furthermore, the Mixture of Experts (MoE) architecture scales beautifully in this environment. Instead of storing all 128 experts on every GPU, the experts are distributed across the node. When the router determines which 4 experts a token needs, it sends the "token chunks" over the NVLink fabric to whichever specific GPUs happen to house those required experts, computing them in parallel before gathering the results back.

By mastering this orchestration—ingesting data via PCIe, splitting math via Tensor Parallelism, dividing layers via Pipeline Parallelism, and synchronizing at terabytes per second over NVLink—systems architects can deploy 120-billion parameter models that generate responses in milliseconds.

Enter vLLM: The Software Engine for High-Throughput Inference

Production-scale inference serving with vLLM

Production-scale inference serving with vLLM

So far, this article has focused on the physical and architectural journey of the model—how GPT-oss 120B is mapped from layers into Safetensors, loaded across the PCIe bus, stored in HBM3e VRAM, and distributed across 8 GPUs via Tensor and Pipeline Parallelism.

However, hardware alone does not make a production-ready AI service. To serve thousands of users simultaneously (like ChatGPT or Perplexity does), you need highly optimized software to manage the traffic, memory, and queuing of incoming requests. This is where vLLM comes in. vLLM is a state-of-the-art, open-source inference server designed to squeeze the maximum throughput out of the GPU hardware we just discussed.

Key vLLM Optimizations

The image highlights several critical software optimizations that vLLM applies on top of the physical infrastructure:

The Difference: Architecture vs. Serving

To understand the complete pipeline, it is crucial to distinguish between what we learned earlier and what vLLM does:

Megatron Bridge vs. vLLM: Are they the same?

A common point of confusion is the relationship between NVIDIA's Megatron and inference servers like vLLM.

Megatron-LM (and Megatron Core) is primarily a framework designed for the massive undertaking of pre-training and fine-tuning foundational models. It provides the low-level building blocks (like the Megatron Bridge) to orchestrate Tensor, Pipeline, and Sequence parallelism across thousands of GPUs during the training phase.

Can Megatron be used for inference? Yes, technically Megatron has inference scripts, but it is generally used as a reference implementation or for testing the model immediately after training. It is not optimized for serving thousands of concurrent users in a production environment.

vLLM, on the other hand, is purpose-built exclusively for production inference. Interestingly, vLLM actually borrows Megatron-LM's Tensor Parallelism algorithms to distribute the model across GPUs. Today, the standard enterprise workflow is to use Megatron to train the model, export the finished checkpoints (often using tools like NeMo), and then load those weights into vLLM to serve the model to end-users with maximum efficiency and lowest latency.

The Strategic Imperative of AI Infrastructure

As generative AI transitions from experimental workflows to core enterprise operations, the industry conversation must shift from simply asking “which model is smartest” to “how robust is our underlying infrastructure.” True competitive advantage in the AI space is no longer just about prompt engineering or having the best model weights. It is about mastering the deployment stack.

Understanding this end-to-end pipeline—from optimizing NVMe-to-VRAM loading and orchestrating multi-GPU parallelism, to maximizing concurrent user throughput with advanced serving engines like vLLM—is what separates a fragile proof-of-concept from a highly profitable, scalable production environment. As AI architectures continue to grow in complexity, the organizations that win will be those that treat MLOps and infrastructure architecture not as an afterthought, but as the foundation of their business strategy.

I’d love to hear from other engineering leaders and systems architects in my network: As you scale your LLM deployments, what are the biggest hardware or serving bottlenecks you are currently navigating in your clusters? Let’s discuss in the comments.The Model Architecture

References & Further Reading

Images generated with: Gemini and Perplexity