This Article is a part of
AI Infrastructure & Data Resource Center
AI Networking Infrastructure
Buying more GPUs doesn’t automatically make an AI cluster faster.
At some point, the GPUs spend less time computing and more time waiting for data from other GPUs, storage systems, or network peers. That’s where AI networking infrastructure becomes critical.
Our broader AI Infrastructure and Data Guide covers the complete stack. This article focuses specifically on the network fabric connecting those systems.
Table of Contents
Why Networking Becomes a Bottleneck
The vast majority of enterprise networks only have north-south traffic (i.e., a user makes a request, a server consumes it and then responds).
Distributed AI training is different.
Thousands of accelerators may exchange data repeatedly during training. Combined operations such as All Reduce, All gather and All to All may generate a considerable amount of east west traffic among the GPUs/nodes.
This provides an uncomfortable dilemma as a single, slow or busy communications link could choke the entire process.
The result isn’t necessarily a failed job. It’s often something less obvious — expensive GPUs sitting idle while the network catches up.
That’s why networking has to be designed alongside AI Compute & GPU Infrastructure, rather than treated as ordinary data-center connectivity.
Front-End and Back-End Networking
Large AI environments can use different fabrics for different jobs.
A front-end network handles services such as user requests, management traffic, storage access and general data-center communication.
Back-end or scale-out fabric For a distributed workload, a back-end or scale-out fabric interconnects compute nodes and accelerators.. It must support high throughput, low latency and predictable performance with very heavy east-west traffic.
These networks don’t always have to be physically separate. The architecture depends on the deployment. But isolating or carefully managing different traffic classes can prevent training traffic from interfering with latency-sensitive services.
For instance, AMD explains that its Pensando Salina DPU serves as “a front-end networking component for AI workloads,” and that its Pollara AI NIC is meant for “accelerator-to-accelerator back end comms.”
Scale-Up, Scale-Out and Scale-Across
AI networking becomes easier to understand when you separate three different problems.
Scale-up
Scale-up connects accelerators within a server or rack.
NVIDIA’s NVLink is a prominent example. Its GB200 NVL72 platform connects 72 Blackwell GPUs into a single NVLink domain, with NVIDIA specifying up to 130 TB/s of aggregate NVLink communication bandwidth for the rack-scale system.
This is fundamentally different from connecting separate servers through an Ethernet or InfiniBand network.
Scale-out
Scale-out extends one platform over more and more servers, forming a much bigger cluster of AI servers.
This is where the role of InfiniBand, RoCEv2 Ethernet, for example, becomes crucial.
Effort aim simply to make a high number of dollars or thousands of dollars to behave as well as possible when they have to communicate from machine to machine.
Scale-across
Scale-across connects AI infrastructure across locations.
This can become relevant when organizations build multiple data centers or need to distribute AI capacity geographically. At this level, distance, optical connectivity, latency, reliability and data-governance requirements become increasingly important.
These three layers have different engineering constraints. Treating them as one generic “network” can lead to expensive architectural compromises.
Tail Latency Matters More Than a Peak Bandwidth Number
Here’s where networking benchmarks can become misleading.
A switch might advertise enormous bandwidth, but that doesn’t tell you what happens when hundreds of GPUs simultaneously exchange data.
AI workloads can be particularly sensitive to congestion and tail latency — the slower end of the latency distribution rather than just the average.
A delayed communication from one participant can affect a collective operation involving many others.
That’s why AI-oriented fabrics increasingly use technologies such as:
- RDMA
- adaptive routing
- congestion control
- traffic prioritization
- telemetry
- multipath networking
The Ultra Ethernet Consortium is clearly focused on the bandwidth, latency, tail latency and scale demanded by AI and HPC workloads while construing Ethernet interoperability as the bedrock upon which to build it. Ultra-Ethernet working group has announced the release of specifications UEC Specification 1.0.3 (16 July 2026).
InfiniBand vs. Ethernet for AI
InfiniBand has been the dominant choice of high-performance computing and large AI clusters for many years.
Its strengths include:
- RDMA support
- low latency
- high bandwidth
- mature collective-communication capabilities
- strong integration with HPC and AI software stacks
However, Ethernet has undergone rapid development.
RoCEv2 (RDMA over Converged Ethernet) is a set of standards that provide RDMA over Ethernet; the Ultra Ethernet Consortium is creating a wider EOS architecture based on Ethernet and optimized for both HPC and AI.
That is not to say, however, that Ethernet has completely overtaken InfiniBand.
The appropriate selection is workload dependent but also on size and software ecosystem, operating skills, cost along with a set performance criteria for the cluster.
For example, NVIDIA‘s Spectrum-X platform is tailored specifically for AI-focused Ethernet networking. According to NVIDIA, its Spectrum-X achieves as much as 95% network efficiency at over 100,000 GPU deployments, though these are vendor-reported measurements not representative of an Ethernet benchmark.
So don‘t treat “95%” as meaning every Ethernet network would reach this figure.
The topology and workload matter.
RDMA and RoCEv2
RDMA — Remote Direct Memory Access — allows systems to move data between memory locations with significantly less CPU involvement than conventional networking paths.
That matters when AI servers are already busy feeding and coordinating large accelerator workloads.
Can do RDMA-over-Ethernet (RoCEv2) without the need for an additional infini band fabric.
Each come with but one problem.
High-performance RoCE networks need careful engineering around congestion management, routing, traffic isolation and loss behavior. Simply installing faster Ethernet switches doesn’t automatically produce an AI-optimized network.
DPUs and SmartNICs
Host CPUs have plenty to do in an AI server.
They shouldn’t necessarily have to handle every networking, storage and security operation as well.
That‘s exactly what the DPUs and SmartNICs are good for.
These processors can offload functions such as:
- networking
- security
- storage virtualization
- encryption
- telemetry
- traffic management
- RDMA processing
AMD’s Pensando Salina DPU, for example, supports networking functions alongside security and storage acceleration, including RDMA, RoCEv2 and NVMe-related processing.
AMD also positions its newer AI networking products around separating front-end infrastructure functions from accelerator-to-accelerator back-end communication.
The important point isn’t that every AI server needs a DPU.
It’s that offloading infrastructure work can leave host CPUs available for application and orchestration tasks.
Multipath Networking Is Becoming More Important
As clusters grow, a single network path becomes an increasingly obvious limitation.
OpenAI, in collaboration with AMD, Broadcom, Intel, Microsoft and NVIDIA, created Multipath Reliable Connection (MRC). This networking technology increases the performance and robustness of large AI training clusters.
MRC was made available to the wider industry when OpenAI published MRC to the Open Compute Project on 5 May 2026.
The significance is bigger than one protocol.
Large AI systems increasingly need networking that can:
- use multiple paths efficiently
- respond to congestion
- tolerate failures
- maintain predictable performance
- scale without creating a single communication bottleneck
That’s where multipath designs become valuable.
Where Photonics Fits In
Copper remains extremely important, but increasing bandwidth and data-center scale are putting more pressure on electrical interconnects.
Optical networking can move data over longer distances with different power and signal-integrity characteristics than traditional copper connections.
One emerging approach is co-packaged optics (CPO), where optical engines are integrated closely with switching silicon.
The industry isn’t replacing every copper connection overnight. Instead, optical technologies are gradually becoming more important as AI clusters expand and bandwidth requirements rise.
NVIDIA’s current Spectrum-X roadmap already includes co-packaged optical switching for large-scale AI factories.
How AI Networking Connects to Storage
Networking doesn’t only connect GPUs to GPUs.
It also connects compute nodes to storage.
That’s particularly important for training datasets, checkpoints and distributed AI workloads.
A fast storage system can still underperform if the network between the storage and compute layer becomes saturated.
Our AI Storage Infrastructure guide covers that side of the architecture in greater detail.
The three layers therefore work together:
Compute → Network → Storage
A weakness in any one of them can limit the system.
What Actually Matters When Choosing AI Networking?
Between the two, it‘s a win-win situation nor a clear winner; InfiniBand vs Ethernet.
Instead, evaluate:
- Cluster size — How many GPUs and nodes are involved?
- Workload — Training, inference and mixed workloads have different communication patterns.
- Latency — How sensitive is the workload to communication delays?
- Bandwidth — How much data needs to move between participants?
- Congestion control — Can the fabric maintain predictable performance under load?
- Software support — Do your frameworks, collective libraries and drivers support the chosen fabric properly?
- Operations — Will your team know how to set up and diagnose it?
- Total cost — Also include switches, NICs, cables, optics, power and software and operational overhead.
That’s the part people often miss.
The most expensive network isn’t necessarily the fastest network for your workload.
FAQs
Q1: What is AI Networking Infrastructure?
A: AI Networking Infrastructure The collection of switches, NICs, DPUs, interconnects, protocols and network software that provide the data to and from all of the GPUs, servers, storage and users fast enough to enable AI workloads.
Q2: Is InfiniBand better than Ethernet for AI?
A: Not universally. InfiniBand remains a strong choice for high-performance distributed AI and HPC workloads, while RoCEv2 and newer AI-focused Ethernet technologies can offer high performance with Ethernet’s broader ecosystem and operational familiarity.
Q3: What is RoCEv2?
A: RoCEv2 is RDMA over Converged Ethernet version 2. It means that RDMA communications can be run through routed Ethernet networks and is widely used for AI, HPC high network throughput applications.
Q4: Do AI clusters need DPUs?
A: Not necessarily. DPUs become more useful as infrastructure grows and networking, storage and security processing consume more host resources. Their value depends on the architecture and workload.
Conclusion
AI networking is no longer just about connecting servers.
At large scale, it’s part of the compute architecture itself.
GPUs need fast GPU-to-GPU communication. Storage needs a high-throughput path to compute. Distributed training needs predictable collective communication. And increasingly dense AI systems need networks that can scale without wasting expensive accelerator time.
That’s why the future of AI Networking Infrastructure isn’t simply about buying faster switches.
It’s about building a fabric that keeps the entire AI system moving.
