InfiniBand Routing Engines: Adaptive Routing and Credit Loops

InfiniBand has no distributed routing protocol. The subnet manager computes every forwarding table for every switch and programs them, and the algorithm it uses to do that — the routing engine — is a configuration choice. Pick the wrong one for your topology and the fabric will either underperform or deadlock.

That second failure mode is the one worth understanding first, because it explains why most of the routing engines exist at all.

Credit loops, and why they deadlock#

InfiniBand links use credit-based flow control. A sender may only transmit when the receiver has advertised buffer credits. No credits, no transmission — which is how the fabric guarantees it never drops a packet under congestion.

The guarantee has a cost. Consider four switches where A is waiting on B, B on C, C on D, and D on A. Every one of them holds a packet it cannot send, and every one is waiting for a credit that will only be released when the packet ahead of it moves. Nothing moves. The cycle does not resolve, and it does not time out — this is not congestion that clears, it is a deadlock, and it usually needs intervention.

The condition is a cyclic dependency in the channel dependency graph, and it depends on the routes, not just the cabling. A perfectly ordinary topology, routed carelessly, deadlocks. This is what the routing engines are for: each one guarantees, in a different way and with different costs, that no such cycle exists.

The engines#

Min-hop#

The obvious algorithm: route every destination along the shortest path.

It is optimal for hop count and gives no thought whatsoever to credit loops. It also balances poorly, because when several equal-cost paths exist it has no principled reason to spread traffic across them. Useful as a baseline and on small or irregular fabrics where the alternatives do not apply. Not a production choice for a large cluster.

Up/Down (updn)#

Assigns every switch a rank relative to one or more designated roots, then enforces a single rule: a path may go up, and it may go down, but once it has gone down it may never go up again.

That rule alone makes credit loops impossible. A cycle requires a down-then-up transition somewhere, and the rule forbids it. This is the elegant part — deadlock freedom falls out of one constraint, with no knowledge of the topology required.

The cost is that some shortest paths are now illegal, so traffic takes longer routes and link utilisation is uneven. Up/Down is the general-purpose safe choice: it works on any topology, including irregular ones and ones that became irregular because of a failure, and it is what you fall back to when a topology-specific engine cannot be used.

Root selection matters. The SM can derive roots automatically, but on a fabric of any size you specify them, normally the spine switches, via a root GUID file.

Fat-tree (ftree)#

The engine for a proper fat-tree or leaf-spine topology, and what most GPU clusters should be running.

It understands the tiered structure rather than inferring it, and it uses that to place routes deliberately: destinations are distributed across the available up-links so that traffic to different endpoints uses different spines. The result is far better link utilisation than min-hop, and deadlock freedom comes from the tree structure itself.

The requirement is that the topology really is a fat-tree, cabled the way the engine expects. When it is not — a missing link, an asymmetric tier, a node cabled to the wrong leaf — the engine may refuse to route, or route badly. This is why a cabling error can present as a performance problem rather than a link failure.

Torus-2QoS#

For 2D and 3D torus topologies, which are their own world. A torus is full of cycles by construction, so deadlock freedom has to be engineered explicitly — this engine uses virtual lanes to break the dependency cycles, and provides QoS levels as part of the design. If you are not running a torus, this is not for you.

DragonFly+ (dfp)#

For DragonFly and DragonFly+ topologies: groups of switches densely connected internally, with the groups connected to each other by global links. It scales to very large node counts with a low network diameter and comparatively few long cables, which is why it appears in large HPC installations.

Routing it is harder. The natural path — local, global, local — combined with adaptive rerouting around a busy global link creates exactly the up-down-up pattern that produces cycles. The engine handles this with virtual lane assignment: traffic is moved to a different VL when it takes a non-minimal path, so the dependency graph is split and no cycle can close.

Adaptive routing#

Adaptive routing is a separate axis from the routing engine. The engine decides which paths exist; adaptive routing lets a switch choose among equivalent paths at forwarding time based on observed load, rather than pinning every flow to one path.

The gain is real. Static routing hashes flows to paths, and hash collisions mean two heavy flows can land on the same link while a parallel one sits idle. In an AI training job where the traffic pattern is a small number of very large flows, one collision is enough to slow the whole collective — all-reduce runs at the speed of its slowest participant.

The caution is that adaptive routing is what turns a theoretical credit loop into a real one. A packet rerouted around congestion may take a path the engine’s deadlock-freedom argument did not consider. Engines that support adaptive routing on cyclic topologies handle this with virtual lanes; the constraint is not optional, and it is why you cannot simply enable adaptive routing on top of an arbitrary routing configuration and expect it to be safe.

ibdiagnet2.ar reports the adaptive routing state actually programmed into the fabric, which is the place to check what is really in effect.

Configuring it#

The routing engine is an OpenSM setting:

routing_engine updn
root_guid_file /etc/opensm/root_guid.conf

root_guid.conf lists the GUIDs of the switches to treat as roots, one per line. Get them from:

ibswitches

Restart the subnet manager to apply:

systemctl restart opensmd

Then verify, because a routing engine that failed to apply falls back silently:

ibdiagnet

Check the SM section of the log and ibdiagnet2.fdbs for the forwarding tables actually programmed. Confirm the engine you configured is the engine in effect — a fat-tree engine that could not validate the topology will fall back to min-hop, and the only symptom is that performance is worse than it should be.

Multiple engines can be listed in priority order, so the SM tries the preferred one and falls back if it cannot be applied. That is useful for resilience and makes verification more important, not less.

What I would tell someone starting#

Match the engine to the topology, and verify the match took. Fat-tree for leaf-spine, DragonFly+ for DragonFly, Up/Down when the topology is irregular or you need a safe default. Then check which one is actually running.

Know where your roots are. On Up/Down and fat-tree, root selection determines the shape of every path in the fabric. Leaving it automatic on a large cluster is leaving a significant decision to a heuristic.

Treat “the fabric is slower than it should be” as a routing question, not only a hardware one. A topology-aware engine that silently fell back to min-hop, or a cabling error that broke the engine’s assumptions, produces exactly that symptom with clean counters everywhere.

Understand the credit loop before enabling adaptive routing. Adaptive routing is a large win on AI workloads and it is the feature most likely to turn a latent dependency cycle into a stopped fabric.

References

• rdma-core — OpenSM source and opensm.conf documentation
• NVIDIA networking documentation — subnet manager and routing engine reference

Engine names, configuration keys and file locations follow OpenSM conventions; check opensm.conf on your own installation. Which engines are available, and which support adaptive routing, depends on the OpenSM version and the hardware.