From Gray Failure to Root Cause: Autonomous Operations in the Pikes Peak AI Lab

How Dell, Broadcom and Metrum AI correlated compute, network and storage telemetry to identify a silent RoCE back-pressure condition and stage a governed remediation.

Field Note · Autonomous Operations · Pikes Peak AI Lab · Infra & Networking · ~9 min read

Executive Takeaway

AI infrastructure can report healthy status while a workload runs slower than the team expects. Links stay up, drop counters stay at zero, and throughput dashboards look ordinary. The difficulty is not detecting an error. It is connecting a suspected slowdown to the specific host, network interface card (NIC), switch interface and queue behavior behind it.

In the Pikes Peak AI Lab, an agent workflow correlated Broadcom remote direct memory access (RDMA) over Converged Ethernet (RoCE) NIC counters, Dell PowerSwitch Z9864F-ON interface telemetry and Link Layer Discovery Protocol (LLDP) topology data for two hosts attached to a single leaf switch. That correlation localized a suspected receiver-side back-pressure condition and produced a ranked recommendation for human review.

The result is a practical validation of the Dell AI Factory operating model: AI infrastructure is not complete when it is installed; it must also be observable, governed, and ready to operate at scale.

Key Results at a Glance

01 · THE OPERATIONAL CHALLENGE

Large-model training depends on synchronized collective operations. When one receiver cannot drain data quickly enough, other graphics processing units (GPUs) can wait even though links remain up, switch drop counters remain at zero, and throughput dashboards look normal. The operational problem is not simply detecting an error; it is connecting the user-visible slowdown to the specific host, NIC, switch port, and queue behavior that caused it. This is a gray failure: the infrastructure's self-view appears healthy while the workload experiences degraded performance.

GPUs Talk Over a Lossless Fabric: Between Hosts, Not Within Them

Inside a host, GPUs talk over a proprietary intra-host link (NVIDIA NVLink on the H200 systems, AMD Infinity Fabric on the MI300X systems), fast enough that eight GPUs on one baseboard behave like a single large accelerator. This failure does not live there. It lives the moment a collective operation crosses to another host and depends on the Ethernet fabric to carry it. That is what makes this specifically a multi-host problem: a single-host job never touches the fabric and could not produce this failure at all.

Large training jobs run collective operations, AllReduce and All-to-All, where every GPU exchanges gradients with every other GPU in lockstep. The step cannot finish until the slowest exchange finishes. So the network between hosts is not plumbing; it is on the critical path.

The Pikes Peak AI Lab carries that traffic with RoCE version 2 (RoCEv2): one host's Broadcom NIC uses RDMA to write directly into another host's GPU memory with no central processing unit (CPU) in the path, across the Dell PowerSwitch fabric. RoCE performance is highly sensitive to packet loss and congestion: a single drop can force a slow retransmit that stalls the whole collective. Plain Ethernet makes no delivery guarantee, so the fabric leans on two mechanisms to keep loss to a minimum:

• Explicit Congestion Notification (ECN) / Data Center Quantized Congestion Notification (DCQCN), the gentle throttle. A switch marks packets as a queue fills; the receiver tells the sender to slow down.
• Priority Flow Control (PFC), the hard brake. Just before a buffer overflows, a switch sends a pause upstream: stop, don't drop.

A healthy fabric mostly uses ECN and rarely needs PFC. Hold that thought.

Why This Is a "Gray Failure"

A gray failure is a component that looks healthy to its own monitoring while it degrades the real workload. The defining trait, differential observability, is a gap between what the system reports about itself and what the user actually experiences. It fits exactly.

Table 1 | Differential observability

Nothing pages anyone. The bandwidth chart even looks busy. The only evidence lives in control-plane counters, RNR NAK retries on the NIC and PFC pauses on the switch, that no standard dashboard plots.

Why Standard Monitoring Misses It

Most fabric monitoring is single-layer, either the switch or the host, and it watches throughput, link state, and errors. This failure trips none of those. To see it, you have to correlate two layers at once: the host side (RNR NAK retries at the RDMA transport on the Broadcom NIC) and the switch side (PFC pauses on the exact Dell PowerSwitch Z9864F-ON ports those hosts use), and then confirm what isn't there, zero drops and zero physical errors, to rule out the look-alikes.

That correlation needs a piece most clusters don't have: a map from a host's NIC to the switch port it plugs into. Without it, you have two piles of numbers and no way to line them up. This is the gap the Pikes Peak AI Lab set out to close with autonomous agents.

---

02 · THE VALIDATION ENVIRONMENT

The Pikes Peak AI Lab provides a controlled environment for testing and validating AI infrastructure behavior before broader deployment. The environment combines Dell PowerEdge XE9680 servers with AMD Instinct and NVIDIA GPUs, Broadcom BCM57608 (Thor 2) RoCE NICs in the hosts, Dell PowerSwitch Z9864F-ON switches built on Broadcom Tomahawk 5 silicon and running Enterprise SONiC Distribution by Dell Technologies, Dell PowerScale F910 storage, and the Integrated Dell Remote Access Controller (iDRAC) on every server. The lab is intentionally heterogeneous, which makes it useful for validating how compute, networking, storage, and operations work together across realistic multi-host AI environments, rather than in a single-vendor or single-layer test.

On top of that, the lab team runs something most AI clusters still do not have: a team of autonomous agents that read the switch, the NIC, and Dell iDRAC together, root-cause faults across all three, and stage remediation for a human to approve. The agent platform is Metrum AI Ops. This field note walks through one fault those agents caught, explained, and recommended a fix for, end to end.

Figure 1 | Metrum AI Ops Digital Twin view showing the Pikes Peak AI Lab

Table 2 summarizes the lab inventory and the telemetry sources available to the agents.

Table 2 | Lab inventory

---

03 · WHAT THE AGENTS WATCH

Dell and Broadcom hardware already exposes the telemetry that explains a fault of this kind. Dell iDRAC exposes the host. The Broadcom NIC exposes the RDMA transport. The Dell PowerSwitch Z9864F-ON running Dell SONiC exposes the fabric. Metrum AI Ops supplies the layer that reads these sources together and reasons across them.

Table 3 separates platform coverage from the evidence this investigation actually used. An available integration is not proof that the source contributed to this diagnosis.

Table 3 | Telemetry sources and their contribution. The grouping separates the evidence behind this diagnosis from integrations that were available but not exercised by this specific incident.

Agent Roles and the Approval Boundary

The workflow uses two analytical agents and a coordinator. A Level 1 support agent detects the condition and localizes it. An operations manager agent routes the case. A network specialist agent correlates the cross-layer evidence, ranks candidate causes, and drafts the recommendation.

Telemetry collection runs read-only, and the staged output is a recommendation rather than an executable change. A member of the lab team must approve any configuration change before it reaches a device. Policy enforcement, automated rollback, and audit integration are requirements for closed-loop operation; this investigation did not exercise them, and this field note does not present them as demonstrated.

Table 4 shows the ticket record for this incident as it stood at the time of writing: what each agent contributed, and where the case is waiting on human approval.

Table 4 | INC0010398: state at the time of writing

Figure 2 | Metrum AI Ops Command Center showing rack view and multi-agent team view

---

04 · THE INCIDENT AND EVIDENCE CHAIN

A Receiver That Couldn't Keep Up

On hosts r2c8s1 and r2c8s2, the Broadcom NIC telemetry showed a counter almost nobody watches: RNR NAK retries. An RNR NAK is a receiver not ready (RNR) negative acknowledgment (NAK): the RDMA receiver telling the sender "I have no buffer for this yet, resend later." The sender waits out a timer and retransmits. A handful of RNR NAKs is normal.

These two hosts had logged millions: approximately 7.3 million RNR NAK retries, indicating repeated receiver-not-ready conditions, with approximately 4.48% of packets associated with receiver-not-ready retransmission in the captured data.

Table 5 | Receiver back-pressure signature

The corresponding Dell PowerSwitch Z9864F-ON ports recorded substantial PFC activity from the hosts, while queue drops and physical-layer errors remained at zero. Put together, the picture is a slow receiver, one side outrunning the other's ability to take data in. Not a broken cable, not a congested core: an endpoint that could not drain its own NIC fast enough.

The Detection, Shown

With that instrumentation already in place, three of its read-only sources, the Broadcom NIC RDMA exporter, the Dell PowerSwitch Z9864F-ON telemetry from Dell SONiC, and the LLDP topology map, turn the detection into a correlation that all three have to agree on. Figure 3 shows that correlation for hosts r2c8s1 and r2c8s2, both on the same leaf switch, as the agents assembled it from NIC telemetry, switch telemetry, and the LLDP topology map.

Figure 3 | Two hosts, one leaf switch

Note: counter values are cumulative totals read during the investigation. Sampling window, reset history and PFC direction are not established.

Three layers, one story: a receiver back-pressure event, intra-leaf (one hop, no spine), with zero queue drops on the monitored interfaces. PFC held the line, but at the cost of constant pausing.

The agents used the LLDP topology map to connect the two hosts to the same leaf switch and then correlated the NIC and switch evidence. The resulting diagnosis was a receiver-side back-pressure condition: the receiver could not drain incoming traffic at the rate offered by the sender.

The wording above intentionally distinguishes evidence from inference. The counters demonstrate receiver-not-ready behavior and PFC response; they support, but do not by themselves prove, the precise application-level mechanism that caused the receiver to fall behind.

---

05 · RULING OUT LOOK-ALIKES

The same PFC symptom shows up in problems with very different causes, so the differential matters. This is where having the switch, the NIC, and Dell iDRAC in one pipeline pays off, because each look-alike is cleared by a different source:

• Not packet loss: switch queue drops on the Dell PowerSwitch Z9864F-ON were zero.
• Not a bad cable or optic: symbol and link-integrity errors on the NIC and switch ports were zero; the physical layer was clean.
• Not a fabric-wide PFC storm: the pauses stayed at the host-to-leaf edge, never propagated to the spine, and the storm watchdog never tripped.
• Not a host hardware fault: Dell iDRAC on both Dell PowerEdge XE9680 hosts is the source the agents check for thermal, power, and NIC port events, and it gave no reason to suspect the server itself.

What's left is the receiver: too few receive buffers posted, or a NIC the GPU couldn't drain fast enough.

This differential diagnosis is the core proof point. The system did not merely report that "the network was slow"; it identified what the event was not, localized where it was occurring, and ranked the remaining cause.

---

06 · THE GOVERNED REMEDIATION

The Industry Check

This isn't a niche curiosity. In "RDMA over Ethernet for Distributed Training at Meta Scale" (ACM SIGCOMM 2024), Meta describes the same failure modes from running one of the world's largest AI networks: receiver-side bottlenecks, the burstiness of collectives, and load imbalance across paths. The most useful part corrected the lab team's own first instinct about the fix: after finding DCQCN tuning impractical to get right at their scale, Meta's team moved congestion management out of the network layer entirely and into the collective communication library itself, letting the receiver, not a switch knob, decide when the sender is allowed to transmit.

The immediate relief options include reviewing receive-buffer depth, RNR retry behavior, and compute/communication overlap. These are tactical mitigations and should be tested against the workload and software stack.

Meta's deployed solution was receiver-driven traffic admission: the sender may only transmit after the receiver grants a clear-to-send. If the sender can never outrun the receiver, the receiver is never "not ready," and RNR NAKs cannot happen. It fixes the cause at the collective layer, not with a driver knob.

The lab's higher-level recommendation is to evaluate receiver-driven admission or clear-to-send behavior in the collective communication layer, co-tuned with the Ethernet fabric. That recommendation is staged for human approval; it was not applied as part of this investigation.

So the network specialist agent's recommendation, attached to INC0010398, splits into the tiers shown in Table 6.

Table 6 | Remediation tiers

---

07 · WHAT CHANGED OPERATIONALLY

Two operational numbers tell the story. Mean time to detect (MTTD) is how long a fault runs before anyone notices. Mean time to know (MTTK) is how long from noticing to understanding the root cause. In the lab the two map cleanly onto two agents: Level 1 support detects and localizes, then the network specialist root-causes the cross-layer correlation. For a gray failure, both numbers are the whole battle.

The agents detected, localized, and produced a ranked root-cause recommendation for the incident in approximately five minutes. The ~5 minute figure covers the detection and root-cause workflow, including human-approval routing; it does not represent remediation completion or total incident-resolution time. And on the standard-monitoring side, no actionable alert was generated for this condition. The honest way to state the operational improvement is: from no actionable alert in the existing dashboards, to cross-layer diagnosis and a governed next action in approximately five minutes.

Table 7 breaks the ~5 minutes into its detection and root-cause stages.

Table 7 | MTTD / MTTK with the Metrum AI Ops agents

The deterministic core is tiny: the detector runs in about a second and the KB lookup adds ~12 s, both measured. The rest of the ~5 minutes is LLM reasoning spread across the two agents, roughly half in Level 1 detection and localization, half in the specialist's root-cause analysis. The point isn't just a faster alarm: a gray failure has no alarm to speed up, so this is the difference between "never caught" and "caught, localized, and explained in minutes."

These numbers don't tell us how long this particular event ran before the RDMA exporter came online in the lab, and we won't manufacture a false-precision cost estimate for a duration we don't actually know. What they do tell us is what happens to the next one.

Turning "never caught" into "caught, localized, and explained in about five minutes" is the difference that compounds every time this pattern recurs across a fleet this size, not just once.

---

08 · WHY THIS MATTERS FOR DELL AI FACTORY CUSTOMERS

The lab demonstrates that Dell and Broadcom infrastructure can expose the evidence needed to explain difficult AI-fabric failures when telemetry is connected across the stack. For customers, the implication is broader than one RoCE runbook:

• Validate before scale: use a lab to test compute, network, storage, and operations together.
• Operate the full stack: correlate infrastructure health with workload behavior.
• Protect GPU utilization: identify gray failures before they become prolonged training or inference delays.
• Use an open ecosystem: combine Dell infrastructure, Broadcom Ethernet, AMD and NVIDIA accelerators, Dell SONiC, and partner software.
• Govern change: stage remediation with policy, approval, rollback, and audit controls.

This is the Dell AI Factory story in operational form: a validated foundation, an open partner ecosystem, and expert services that help move AI infrastructure from deployment to dependable production operation.

---

09 · SCOPE, MATURITY AND NEXT PHASE

This case documents one lab investigation. It is not a product announcement, benchmark, guarantee, or claim that the same diagnosis will occur in every environment. Coverage maturity varies by runbook. The demonstrated capability is cross-layer detection, localization, root-cause reasoning, and staged remediation. Approved and applied closed-loop remediation remains a controlled next phase.

The next validation areas include ECN hotspots, PFC storms, link flaps, NVIDIA Xid errors and GPU ECC faults, SSD wear, NVMe controller instability, and storage-management-plane health. Each scenario should preserve the same evidence discipline: identify the measured signal, distinguish inference from proof, document the approval boundary, and publish only validated outcomes.

Metrum AI Ops is the platform behind the agents in the Pikes Peak AI Lab: compute, storage, and fabric in one detection pipeline, one agent team, with a human approving every change.

metrum.ai/ops

---

Source and Positioning References

---

Disclaimer

This document is provided for informational purposes only. It describes a field investigation conducted in the Pikes Peak AI Lab, a Dell Technologies and Broadcom facility, using the Metrum AI Ops platform. It is not a product announcement, benchmark, or guarantee of performance, availability, or outcomes in other environments. The remediation described as staged for approval had not been applied at the time of writing; the MTTD/MTTK figures reflect the detection and root-cause workflow only, not incident resolution. Actual results will vary based on workload, configuration, deployment conditions, and other factors.

Autonomous remediation requires customer-defined policy, approval gates, rollback procedures, credential controls, and change-management integration. Production deployments must be evaluated against each organization's safety, security, and compliance requirements.

Dell, PowerEdge, PowerScale, PowerSwitch, and iDRAC are trademarks of Dell Inc. or its subsidiaries. Broadcom and Tomahawk are trademarks of Broadcom Inc. SONiC is a project of the Linux Foundation. NVIDIA, NVLink, NCCL and H200 are trademarks of NVIDIA Corporation. AMD, Instinct, Infinity Fabric and ROCm are trademarks of Advanced Micro Devices, Inc. VMware ESXi is a trademark of Broadcom Inc. METRUM AI is a trademark of Metrum AI, Inc. All other trademarks are the property of their respective owners. Use of these names does not imply endorsement.

Third-party statistics and projections cited herein are attributed to their respective sources and have not been independently verified by Dell Technologies, Broadcom, or Metrum AI.