↑↓ navigate · enter open · esc close
Notes from building and operating the network layer of AI infrastructure: InfiniBand fabric design and diagnostics, RDMA transport, lossless Ethernet with RoCE on NVIDIA Spectrum-X, and Cumulus Linux on the switch side.
AI Networking pinned
Four leaves, two vendors, one Ethernet Segment. Five configuration differences between Cumulus and Arista stand between the RFC and a working all-active bond — four get solved here, and none of them are in the RFC.
AI Networking pinned
RA drives dynamic peer discovery in BGP unnumbered — and turning it off tears down established sessions instantly, with no flap and no timeout. Why, and what the ND cache does and does not tell you.
The control plane agrees on the Designated Forwarder and the host still receives every broadcast twice. Multi-point captures, a same-vendor control group, and a packet capture that was quietly lying.
Arista and Cisco NX-OS, an unnumbered eBGP session, an IPv4 route installed with an IPv6 next hop — and no forwarding. The fix is one interface command, and the reason is RFC 8950.
The subnet manager picks the routing engine, and the routing engine decides whether your fabric deadlocks. Min-hop, Up/Down, fat-tree and DragonFly+, and what each one is actually protecting you from.
ibdiagnet is the first command to run on a sick fabric. What its output files contain, which counters actually matter, and how to get from a slow training job to a specific port.
When a fabric link flaps or runs at the wrong width, the answer is usually in the optics. How to read PCIe and fabric port state, per-lane signal quality and cable EEPROM with the NVIDIA firmware tools.
What RoCE actually needs from the network, how PFC and ECN divide the work, and how the lossless and lossy modes are configured and verified on Cumulus Linux.