Link-Layer Troubleshooting with MLXLink and MLXCables

Most fabric problems that survive an afternoon of investigation are physical. A link negotiates at half the expected width, a port flaps under load but not at idle, one node’s collective performance is consistently worse than its neighbours’. The switch says the port is up, the host says the interface is up, and both are telling the truth.

mlxlink and mlxcables, from the NVIDIA Firmware Tools package, are where you go next. They read what the transceiver and the link training logic actually see, rather than the up/down summary everything else reports.

Getting MFT running#

MFT ships as a package for the usual distributions. Once installed, the tools need the mst device layer started before they can reach the adapters:

mst start
mst status -v

mst status -v is the command to run first and the one worth reading carefully. It lists every Mellanox device the host can see and, critically, gives you the device name each tool wants as its -d argument — both the /dev/mst/... device path and the PCI address. It also shows the PCIe slot, the number of ports, and the RDMA device name (mlx5_0 and friends) that perftest and the verbs layer use.

Getting the device identifier from here rather than guessing saves a surprising amount of confusion on hosts with more than one adapter.

The PCIe side#

Before looking at the fabric, rule out the host. A ConnectX adapter that has negotiated a narrower PCIe link or a lower generation than the slot supports will cap throughput no matter how healthy the fabric is, and nothing in the network will indicate why.

mlxlink -d <device> --port_type PCIE

What matters in the output:

• Link speed and width — compare against what the adapter and the slot are rated for. A 400G adapter in a Gen4 x8 slot cannot reach line rate; the arithmetic simply does not work. Seeing x8 where x16 is expected usually means the card is in the wrong slot, is not fully seated, or the slot is bifurcated.
• Link state and any error counters on the PCIe side.

This check takes ten seconds and eliminates an entire category of “the fabric is slow” investigations. Do it before touching anything else.

The fabric side#

mlxlink -d <device> -p <port>

The output falls into three groups, and the third is the one people tend to skip.

Operational state. Physical state, logical state, negotiated speed and width, and the active protocol. The first thing to check is whether the negotiated values match the expected ones — a link that trained to a lower speed or a narrower width is already telling you something, and it does it without ever going down.

Link status and troubleshooting information. mlxlink reports a status opcode with a human-readable message. This is genuinely useful: it distinguishes “no cable detected” from “cable detected but no signal from the far end” from “signal present but training failed” from “link is up but degraded”. Those four states produce identical “port down” reports everywhere else in the stack and have completely different causes.

Physical layer counters and per-lane signal quality, with --show_counters, --show_eye and related flags depending on version:

• Effective and raw BER. Raw BER is what the receiver sees before forward error correction; effective BER is what survives it. Modern high-speed links run with a raw error rate that would be alarming on an older link and is entirely normal here, because FEC is doing its job. The number to watch is effective BER — if errors are getting past FEC, the link is genuinely damaged.
• Per-lane figures. This is the detail that makes the tool worth using. A 4-lane link with one bad lane behaves very differently from a link that is uniformly marginal: the first is a connector, a fibre, or a bent pin, and is usually fixed by reseating; the second is more often a length, loss-budget, or transceiver problem. You cannot distinguish them from aggregate counters.
• Eye margin measurements, where available, quantify how much headroom the receiver has. A link that works today with almost no margin is the one that will flap when the datacentre warms up.

Reading the cable itself#

mlxcables goes one level lower, to the transceiver EEPROM:

mst cable add
mlxcables
mlxcables -d <cable-device> --DDM

mst cable add enumerates cable devices so they can be addressed directly. On InfiniBand fabrics, mst cable add --with_ib includes IB-attached cables.

What it gives you:

• Vendor, part number, serial number, and firmware revision. Mundane and repeatedly decisive. Mixed transceiver models or firmware revisions within a fabric are a real source of intermittent behaviour, and an inventory pulled from the hardware is worth more than the one in the spreadsheet.
• Cable type and length. Passive copper, active copper, or optical, and the actual length. Worth checking against the link budget for the speed you are running.
• Digital diagnostic monitoring — temperature, supply voltage, and per-lane transmit and receive optical power, live.

The DDM readings are where intermittent faults become visible. Low receive power on one lane points at that fibre, that connector, or contamination — clean and reseat before replacing anything. Low receive power on all lanes points at the far-end transmitter, the fibre run, or a patch panel. Temperature at the top of the operating range correlates with links that flap in the afternoon and are fine overnight, which is one of the most annoying failure patterns to chase without this data.

A workflow#

When a link is suspect:

1. mst status -v — get the right device identifier.
2. mlxlink --port_type PCIE — rule out the host slot before blaming the fabric.
3. mlxlink -d <device> -p <port> — is the negotiated speed and width what you expect? What does the troubleshooting message say?
4. Per-lane BER and eye margin — is the problem one lane or all of them?
5. mlxcables --DDM — optical power per lane, temperature, and what the cable actually is.
6. Clean and reseat before replacing. A large share of “bad optics” are dirty connectors, and a fibre inspection scope pays for itself the first time.

Then re-measure. The point of per-lane numbers is that they give you a before and after — “it seems better now” is not a result.

Why this is worth learning properly#

Fabric diagnostics at the InfiniBand layer, ibdiagnet and its counters, will tell you a link has symbol errors. It will not tell you whether that is one dirty connector, a transceiver running hot, or a cable that is simply too long for the speed. Those have different fixes and very different costs.

On a large GPU cluster the physical layer is also where the volume is: thousands of cables, and failures that are marginal rather than binary. A link that is up and passing traffic with no error margin left does not appear in any dashboard until it starts flapping in the middle of a training run. Per-lane BER and DDM readings are the only place that shows up early.

References

• NVIDIA Firmware Tools (MFT) — mst, mlxlink, mlxcables
• NVIDIA networking documentation

Exact flags and output fields vary between MFT versions and adapter generations; check mlxlink --help and mlxcables --help on your own installation. Thresholds for “acceptable” BER and optical power are speed- and media-dependent — read them against the specification for the link you are running, not against a number from another fabric.