Fast Notification for tunnel-based lossless RDMA transmission in WAN

For modern AI services such as distributed LLMs training or inference, WAN needs to support the tunneling of RDMA traffic between data centers (DCs). RDMA is a widely used technology in high-performance computing and AI clusters, achieving low latency, reduced CPU overhead, and high network throughput. Currently, mainstream RDMA protocols (e.g., IB, RoCE) operate over best-effort forwarding, where a small number of packet losses can result in a dramatic reduction in the effective throughput. Therefore, WAN requires the FAst Notification for Traffic Engineering and Load balancing to ensure reliable and congestion-free data transfer. points existing TE mechanisms face limitations in responsiveness, coverage, and operational overhead, especially in high-speed, large-scale environments. ECN is a widely deployed congestion control mechanism, which enables a forwarding element to notify the sender for congestion control without having to drop packets. But it still relies on end-to-end signaling, making real-time feedback challenging in long-distance WAN. BFD is designed for rapid fault detection by sending frequent control packets between peers, but higher probe frequency increases CPU and bandwidth usage, make it struggles to balance detection speed with system overhead. is an IETF Problem Statement for Fast Network Notification(FANN), based on the analysis of gaps in current network mechanisms and the operational requirements of modern applications (e.g., AI/ML training), formally defines the scope and core requirements for fast network notifications. Moreover, it futher specifies what information such notifications carry, who the intended recipients are, how they should be delivered, and what kinds of timely actions they may enable. To enable lossless data transmission, some drafts are proposed to support FANN. proposes a proactive notification mechanism ARN for adaptive routing, and describes the information carried in ARN to notify remote nodes for re-routing. describes the mechanisms of delivering ARN message. This document specifies the FANN mechanism for scenarios where service traffic is carried over tunnels in WAN. It first introduces the typical scenarios, then specifies the process of fast notification to achieve key TE areas such as congestion control, load balancing, and failure protection, and finally defines the protocol implementation.

CNP: Congestion Notification Packet ECN: Explicit Congestion Notification FANN: Fast Network Notification PFC: Priority-based Flow Control RoCEv2: RDMA over Converged Ethernet version 2 WAN: Wide Area Network

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

The growth of computing power of a single DC is limited by space and power supply, making it difficult to meet the fast-growing computing resources demands of LLMs training. Therefore, distributed model training across multiple DCs provides a more efficient and cost-effective solution to aggregate computing resources. In this scenario, TB-scale training parameters need to be rapidly synchronized over WAN.

Some customers deploy LLMs by building on-premises AI facilities, but as inference concurrency increases, scaling out these facilities requires significant investment. To address this, distributed model inference between customer on-premise and third-party DC provides a more agile and cost-effective solution. In this scenario, data such as the KV cache and model parameters need to be rapidly synchronized over WAN.

In the above scenarios, a large volume of data between DCs need to be synchronized using RDMA protocol. RDMA traffic generated by LLM training or inference is highly concurrent, bursty, and extremely latency-sensitive. Therefore, operators typically encapsulate it in tunnels over the WAN to enable flexible steering and end-to-end service isolation. In these scenarios, the framework for RDMA traffic transmission over WAN tunnels is as follows:

The AI servers in DC1 sends RDMA traffic to WAN's ingress PE. At the WAN's ingress PE, the RDMA traffic is encapsulated according to the tunnel protocol and forwarded across WAN to egress PE. The WAN's P node(R1-R5) transits the payload from ingress PE to egress PE via tunnels. At the WAN's egress PE, the payload are decapsulated to RDMA packets and transmitted to the AI servers in DC2.

Tunneling technologies include various protocols, such as GRE, VXLAN, MPLS, and SRv6. Moreover, AI workloads are highly sensitive to packet loss, latency and throughput. Network failures, congestion or underutilization can all lead to significant waste of compute resources. When transmittig RDMA traffic over tunnels, WAN should support FANN capability to realize rapid response to network conditions. Specifically, WAN devices should support fast notification mechanism to imporve three key TE scenarios: failure protection, flow control, and load balancing.

For large-scale and dynamic networks, protection mechanisms need to ensure service continuity in case of failures. According to , existing failure handling methods, such as BFD and FRR, lack flexibility and responsiveness in complex typologies. Therefore, WAN should support fast notification for failures, allowing near-instantaneous and dynamic protection responses, minimizing failure impact. Upon network failure, the ingress PE should immediately adapt its forwarding policy to steer traffic away from faulty links or nodes. Therefore, the fast-notification-based failure protection process is as follows:

When a P node detects a local link/node failure, it collects failure information about the affected link or flow. The P node sends notification to ingress PE with failure information (In addition to the identity of the failed link or node, the notification must also include information about the affected traffic). Ingress PE receives the notification and reroutes the traffic based on its content to exclude the failed link or node: *If backup path is available, ingress PE should switch the service traffic to the backup path. *If multiple feasible paths exist, ingress PE should updates its load-balancing policy to utilize all available paths. Ingress PE should send a corresponding notification to the sender and controller.

RDMA traffic is bursty and highly sensitive to packet loss, and WAN require proactive congestion control mechanisms. redefines how the explicit congestion notification (ECN) field of the IP header should be constructed on entry to and exit from any IP-in-IP tunnel, in order to achieve ECN-based congestion control across WANs between DCs. However, analysis that ECN/TCP methods still relies on end-to-end signaling and lacks precise real-time feedback. Currently, PFC is widely used in data centers to prevent data loss due to congestion. PFC uses a step-by-step back-pressure mechanism to control the upstream to stop or continue transmitting traffic. PFC achieves link-layer priority-based traffic control, but still faces problems such as queue head blocking and deadlock due to coarse control granularity. When network congestion occurs, the ingress PE should immediately adapt its forwarding policy to reduce the traffic sent to congested nodes. Therefore, the fast-notification-based congestion control process is as follows:

when a P node detects congestion, it collects congestion information about the congested link or flow. The P node sends notification to ingress PE with congestion information. Ingress PE receives the notification and reroutes the traffic based on its content to exclude the congestion link: *If backup path is available, ingress PE should switch the service traffic to the backup path. *If multiple feasible paths exist, ingress PE should updates its load-balancing policy to utilize all available paths. Ingress PE should reduce the transmission rate of corresponding traffic, and send notification to sender and controller.

Devices and links in WAN often carry multiple services simultaneously. In addition to failure and congestion, dynamic load balancing based on network state changes can effectively improve network resource utilization. When significant changes occur in the network state, the ingress PE should dynamically adjust its forwarding strategy to maximize network resource utilization. Therefore, the fast-notification-based load balancing process is as follows:

When a node detects the network state change, it collects the network state change information, such as link utilization, queue buildup. The node sends fast notification to the ingress PE with information about the network state change. Ingress PE receives the fast notification and updates its load-balancing policy to maximize the utilization of network resources.

Based on the framework analysis of fast notification in key TE areas, a unified protocol implementation for fast notification should be established, with explicit forwarding procedures to realize tunnel-based lossless transmission of RDMA packets in WAN.

The source quench mechanism has been deprecated in ICMPv6 because TCP's built-in congestion avoidance algorithms are more efficient, and source quench may interfere with their normal operation. However, fast network notification is a network-layer mechanism confined to the forwarding plane, designed for event-driven generation and consumption without involving endpoints. This avoids the conflict that led to Source Quench's deprecation, making ICMPv6 suitable as a carrier for fast notifications.

This document specifies a new ICMPv6 message to realize rapid notification in key traffic engineering areas including failure protection, congestion control, and load balancing. This ICMPv6 message consists of a fixed ICMPv6 header, and a variable-length metadata stack controlled by a 32-bit bitmap:

Type (1 byte): ICMPv6 type for FANN. IANA allocation required (suggested value: TBD).
Code (1 byte): 0 for event notification, 1 for recovery notification.
Checksum (2 bytes): Standard ICMPv6 checksum.
Version (1 byte): Set to 1 for this specification.
Reserved (1 byte): Set to 0 on transmission, ignored on reception.
Hop Limit (1 byte): Controls propagation scope. Decremented by each forwarding node; discarded when reaching zero.
Event Type (1 byte): Primary category of the event. 0x01: Link failure; 0x02: Congestion; 0x03: Performance degradation; 0x04: Microburst; 0x05: Signal degradation / link errors; 0x06: Queue buildup; 0x07: Recovery (condition cleared);
Event Sub-Type (1 byte): More granular classification within an event type. For example, for Congestion (Event Type = 0x02): 0x01 for mild ( > 50% utilization), 0x02 for moderate ( > 70%), 0x03 for severe ( > 90%).
Event Identifier (4 bytes): Unique identifier for this event instance, used for deduplication and correlation between multiple notifications.
Timestamp (8 bytes): Time when the event was detected, in microseconds since Unix epoch (UTC).
Originating Node IPv6 Address (16 bytes): IPv6 address of the node that detected the event. This serves as the node identifier, eliminating the need for a separate Node ID in the bitmap.
Bitmap (32 bits): Each bit indicates whether the corresponding metadata is present in the metadata stack.

The 32-bit bitmap field indicates which metadata is present in the metadata stack. Each bit corresponds to a specific metadata type with a fixed length, enabling efficient parsing without TLV overhead. Bits are listed below by their index.

Bit 0: Ingress Port ID (4 bytes) Interface identifier where the event was observed on the ingress side.
Bit 1: Egress Port ID (4 bytes) Interface identifier where the event was observed on the egress side. Together with Ingress Port ID, uniquely identifies the location of the event.
Bit 2: Ingress Timestamp (8 bytes) Time when the event was observed at the ingress interface.
Bit 3: Egress Timestamp (8 bytes) Time when the event was observed at the egress interface.
Bit 4: Egress TX Link Utilization (4 bytes) Real-time bandwidth utilization of the egress interface (e.g., 95% = 0x0000005F).
Bit 5: Packet Loss (4 bytes) Packet loss count or loss rate at the interface.
Bit 6: Latency / Delay (4 bytes) Current latency of the interface or path in microseconds.
Bit 7: Jitter (4 bytes) Latency variation in microseconds.
Bit 8: Queue Occupancy (4 bytes) Current queue depth in bytes.
Bit 9: Buffer Occupancy (4 bytes) Overall buffer occupancy for shared buffer pools.
Bit 9: Buffer Occupancy (4 bytes) Overall buffer occupancy for shared buffer pools.
Bit 10: Signal Degradation / Link Errors (4 bytes) Bit error rate, CRC error count, or signal quality metric (0-100%).
Bit 11: Hard Failure / Link Down (4 bytes) Boolean flag (lower bit = 1 indicates link down).
Bit 12: Microburst Detected (4 bytes) Boolean flag (lower bit = 1 indicates microburst detected).
Bit 13: Flow ID (5-tuple) (37 bytes) Source IPv6 (16) + Destination IPv6 (16) + Source Port (2) + Destination Port (2) + Protocol (1).
Bit 14: Path ID (16 bytes) Identifier of the affected path (e.g., SRv6 SID list or MPLS label stack).
Bits 15-31: Reserved for future use.

Bitmap bits 0 and 1 (Ingress Port ID and Egress Port ID) serve as the location identifier, eliminating the need for a separate Event Location field. The port IDs uniquely indicate where in the network the event occurred, corresponding to "Location of Event: This can be used to indicate the location where the event occurred in the network. Although bits 11 and 12 are logically single-bit flags, they each occupy 4 bytes for alignment purposes, with the upper 31 bits set to zero.

The metadata stack is constructed by concatenating metadata fields in ascending bit order. For each bit that is set to 1 in the bitmap, the corresponding metadata field of fixed length is appended to the stack. Bits set to 0 are skipped. For example, if the bitmap has bits 1 (Egress Port ID), 4 (Egress TX Link Utilization), 6 (Latency), and 10 (Signal Degradation) set to 1, the metadata stack will be: Egress Port ID (4 bytes): 0x00000008 (port 8) Egress TX Link Utilization (4 bytes): 0x0000005A (90% utilization) Latency (4 bytes): 0x000003E8 (1000 microseconds) Signal Degradation (4 bytes): 0x0000000A (10% error rate)

While the preceding sections define the Fast Network Notification (FANN) message format using ICMPv6, the same fixed header and 32-bit bitmap structure can be directly carried over UDP. When encapsulated in UDP, the ICMPv6 Type/Code/Checksum fields are simply replaced by the standard UDP header (source port, destination port, length, checksum), while the FANN fixed fields, the 32-bit bitmap, and the metadata stack remain unchanged. The receiving node identifies the FANN message by a well-known UDP destination port (to be allocated by IANA) and processes it identically to the ICMPv6 variant. This approach is consistent with the principle that the solution may reuse existing protocols where appropriate. Moreover, the use of UDP offers practical deployment advantages in environments where ICMPv6 traffic is filtered or where NAT traversal is required, without compromising the notification's timeliness or forwarding-plane efficiency.

This document specifies Fast Network Notification (FANN) mechanisms for tunnel-based lossless RDMA transmission in WAN, using ICMPv6 and UDP as transport protocols. While these protocols are widely deployed and well-understood, extending them with new notification semantics introduces potential security considerations that must be addressed. Implementations MUST enforce the rate limiting behavior specified in RFC 4443 §2.4 for all ICMPv6 messages carrying FANN information. The TLV parser MUST validate that the sum of all TLV Length fields does not exceed the total ICMPv6 payload length. Any packet failing this check MUST be silently discarded. All FANN notifications MUST be sent from a control-plane interface of the originating node (e.g., a loopback interface configured for management), and MUST NOT originate from data-plane forwarding interfaces (e.g., physical ports carrying customer traffic). This ensures that FANN traffic cannot be injected by compromised customer devices.

TBD