flozi00 Logoflozi00 TechHub
by flozi00

NVIDIA NVLink Solutions

Overview

NVIDIA NVLink is a high-bandwidth, energy-efficient interconnect technology that enables ultra-fast communication between multiple GPUs in server systems. With bandwidths far exceeding traditional PCIe connections, NVLink is essential for AI training, inference, and HPC workloads that require massive parallel computing power.

Key Benefits

  • Ultra-High Bandwidth: Up to 1.8 TB/s per GPU (NVLink 5th generation)
  • Low Latency: Direct GPU-to-GPU communication bypassing CPU overhead
  • Scalability: Connect up to 576 GPUs in a fully connected fabric
  • Memory Coherence: Shared memory access across multiple GPUs
  • Energy Efficiency: 14x more bandwidth than PCIe Gen 5
  • Bandwidth per GPU: 1,800 GB/s total
  • Links per GPU: Up to 18 NVLink connections
  • Architecture: NVIDIA Blackwell (GB300 series)
  • Use Cases: Trillion-parameter AI models, test-time reasoning, large-scale inference
  • Bandwidth per GPU: 900 GB/s total
  • Links per GPU: Up to 18 NVLink connections
  • Architecture: NVIDIA Hopper (H100 series)
  • Use Cases: Large language models, multi-GPU training, HPC simulations
  • Bandwidth per GPU: 600 GB/s total
  • Links per GPU: Up to 12 NVLink connections
  • Architecture: NVIDIA Ampere (A100, A6000 series)
  • Use Cases: Deep learning, data analytics, scientific computing

The NVLink Switch enables rack-scale GPU interconnection, creating a non-blocking compute fabric for extreme-scale AI systems.

Specifications:

  • Ports: 144 NVLink ports per switch
  • Total Bandwidth: 14.4 TB/s switching capacity
  • Topology: Supports up to 576 fully connected GPUs
  • Latency: Ultra-low latency for inter-node communication
  • Protocol Engines: NVIDIA SHARP for in-network reductions

System Architectures

NVIDIA GB300 NVL72

  • GPU Count: 72 Blackwell GPUs
  • Total Bandwidth: 130 TB/s aggregate GPU bandwidth
  • Compute Power: Up to 1.4 exaFLOPS AI performance
  • Configuration: Fully connected all-to-all communication
  • Form Factor: Single rack solution

Prerequisites

Before checking NVLink status, ensure you have:

  • NVIDIA Drivers Installed: Use the latest drivers for your GPU generation
  • CUDA Toolkit: Required for bandwidth testing and samples
  • Appropriate Hardware: GPUs with NVLink support

Check Physical Topology

Display all GPUs and their interconnect configuration:

nvidia-smi topo -m

This command shows the topology matrix, indicating which GPUs are connected via NVLink, PCIe, or other interfaces.

Check the state and speed of all NVLink connections:

nvidia-smi nvlink -s

Shows the operational status of each link, including active/inactive states and link speeds.

Display detailed connection information for a specific GPU (e.g., GPU 0):

nvidia-smi nvlink -i 0 -c

Display connection information for all GPUs in the system:

nvidia-smi nvlink -c

Interpreting Topology Output

The topology matrix uses symbols to indicate connection types:

  • NV#: NVLink connection (# indicates number of links)
  • SYS: Connection through system bus/memory
  • PHB: PCIe Host Bridge
  • NODE: NUMA node boundary
  • PIX: PCIe switch between GPUs

Bandwidth Testing

Installing CUDA Samples

To thoroughly test NVLink performance, compile and run NVIDIA's CUDA samples:

Clone CUDA Samples Repository

git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples

Checkout Appropriate Version

Select the branch matching your CUDA version:

# For CUDA 12.2
git checkout tags/v12.2
 
# For CUDA 12.6
git checkout tags/v12.6

Install Build Prerequisites

sudo apt -y install freeglut3-dev build-essential libx11-dev \
  libxmu-dev libxi-dev libgl1-mesa-glx libglu1-mesa \
  libglu1-mesa-dev libglfw3-dev libgles2-mesa-dev

Running Bandwidth Tests

Basic Bandwidth Test

Navigate to the bandwidth test directory and compile:

cd Samples/1_Utilities/bandwidthTest
make
./bandwidthTest

Expected Output:

  • Host to Device Bandwidth
  • Device to Host Bandwidth
  • Device to Device Bandwidth (shows NVLink effectiveness)

P2P Bandwidth and Latency Test

The peer-to-peer test provides detailed multi-GPU performance metrics:

cd Samples/5_Domain_Specific/p2pBandwidthLatencyTest
make
./p2pBandwidthLatencyTest

Key Metrics:

  • P2P Connectivity Matrix: Shows which GPUs can directly access each other
  • Unidirectional Bandwidth: One-way data transfer rates
  • Bidirectional Bandwidth: Simultaneous two-way transfer rates
  • Latency Matrix: GPU-to-GPU and CPU-to-GPU latency measurements

Interpreting Bandwidth Results

Without NVLink (PCIe only):

  • Bandwidth: ~6-40 GB/s
  • Latency: 15-30 µs

With NVLink (Generation 2-3):

  • Bandwidth: 50-100 GB/s per direction
  • Latency: 1-3 µs

With NVLink (Generation 4-5):

  • Bandwidth: 100-250+ GB/s per direction
  • Latency: less than 2 µs

Example Output Analysis

Unidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3
     0 589.40  52.75  52.88  52.90
     1  52.88 592.53  52.80  52.85
     2  52.90  52.80 595.32  52.78
     3  52.85  52.88  52.75 593.88

This shows:

  • Diagonal values: Internal GPU memory bandwidth (~590 GB/s)
  • Off-diagonal values: NVLink bandwidth between GPUs (~52 GB/s indicates PCIe or single NVLink connection)

For H100 systems with full NVLink, expect 200-250 GB/s between GPU pairs.

Use Cases and Applications

AI and Machine Learning

Large Language Model Training

NVLink enables efficient training of models with hundreds of billions of parameters by allowing:

  • Model Parallelism: Distribute model layers across multiple GPUs
  • Data Parallelism: Process multiple batches simultaneously
  • Pipeline Parallelism: Stream data through GPU pipeline stages
  • Gradient All-Reduce: Fast synchronization during backpropagation

Inference Optimization

For trillion-parameter models, NVLink provides:

  • Test-Time Reasoning: Real-time inference across distributed models
  • High Throughput: Serve more requests per second
  • Low Latency: Reduced response times for interactive applications

High-Performance Computing

Scientific Simulations

  • Computational Fluid Dynamics: Exchange boundary data between domains
  • Molecular Dynamics: Share particle interaction data
  • Climate Modeling: Distribute atmospheric/oceanic grids
  • Astrophysics: Process massive datasets across GPUs

Data Analytics

  • Graph Analytics: Traverse large-scale graph structures
  • Database Acceleration: Speed up SQL query processing
  • Real-Time Analytics: Process streaming data at scale

Optimization Best Practices

Driver and Software Configuration

Use Latest Drivers: Always install the latest NVIDIA datacenter drivers

Enable Persistence Mode: Reduces latency for GPU initialization

nvidia-smi -pm 1

Set Power Limits: Ensure GPUs run at maximum performance

nvidia-smi -pl 400  # Adjust based on GPU model

Application Optimization

CUDA Programming

  • Use CUDA-Aware MPI: Directly pass GPU pointers between ranks
  • Enable Peer Access: Explicitly enable P2P memory access 
    cudaDeviceEnablePeerAccess(peer_device, 0);
  • Optimize Communication Patterns: Minimize cross-GPU data transfers
  • Use Unified Memory: Leverage automatic memory migration

Framework Configuration

PyTorch:

# Enable NVLink for DataParallel
torch.distributed.init_process_group(backend='nccl')
model = torch.nn.parallel.DistributedDataParallel(model)

TensorFlow:

# Configure multi-GPU strategy
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    model = create_model()

Monitoring and Diagnostics

Real-Time Monitoring

# Monitor GPU utilization and NVLink traffic
nvidia-smi dmon -s u,m,e
 
# Watch NVLink bandwidth utilization
watch -n 1 nvidia-smi nvlink --status

Performance Profiling

Use NVIDIA Nsight Systems for detailed profiling:

nsys profile --stats=true ./your_application

Troubleshooting

Common Issues

Symptoms: nvidia-smi topo -m shows PCIe instead of NVLink

Solutions:

  • Verify physical NVLink bridge installation
  • Check BIOS settings for PCIe bifurcation
  • Update to latest NVIDIA drivers
  • Ensure GPU models support NVLink (not all do)

Reduced Bandwidth

Symptoms: Bandwidth tests show lower than expected throughput

Solutions:

  • Check for GPU throttling: nvidia-smi -q -d CLOCK
  • Verify all NVLink lanes are active: nvidia-smi nvlink -s
  • Review system topology for suboptimal routing
  • Check for background processes consuming GPU resources

Peer Access Failures

Symptoms: CUDA peer access errors or fallback to PCIe

Solutions:

  • Verify P2P capability: Check p2pBandwidthLatencyTest output
  • Ensure IOMMU is properly configured in BIOS
  • Update CUDA toolkit to latest version
  • Check for conflicting GPU virtualization settings

The next evolution enables:

  • Custom Rack-Scale Architectures: Hyperscaler-specific designs
  • Industry-Leading AI Scaling: Exascale AI infrastructure
  • Shared AI Infrastructure: Multi-tenant GPU clusters with NVLink

Upcoming Technologies

  • NVLink-C2C: Chip-to-chip interconnect for CPU-GPU coherence
  • Grace-Hopper Superchips: Integrated CPU-GPU with NVLink-C2C
  • Next-Gen NVLink: Even higher bandwidths for future architectures

Conclusion

NVIDIA NVLink represents a fundamental shift in how GPUs communicate, enabling the massive-scale AI and HPC systems that power today's most demanding workloads. From dual-GPU workstations to 576-GPU supercomputers, NVLink provides the high-bandwidth, low-latency interconnect necessary for breakthrough performance.

Technical Specifications Reference

NVLink GenerationArchitectureMax Links/GPUBandwidth/LinkTotal Bandwidth/GPU
NVLink 3.0Ampere1250 GB/s600 GB/s
NVLink 4.0Hopper1850 GB/s900 GB/s
NVLink 5.0Blackwell18100 GB/s1,800 GB/s

Article last updated: October 28, 2025
Author: flozi00