RFC 9599: BCP 89: Guidelines for Adding Congestion Notification to Protocols that Encapsulate IP

1.2. Scope

This document only concerns wire protocol processing of explicit notification of congestion. It makes no changes or recommendations concerning algorithms for congestion marking or congestion response because algorithm issues should be independent of the layer that the algorithm operates in.¶

The default ECN semantics are described in [RFC3168] and updated by [RFC8311]. Also, the guidelines for AQM designers [RFC7567] clarify the semantics of both drop and ECN signals from AQM algorithms. [RFC4774] is the appropriate best current practice specification of how algorithms with alternative semantics for the ECN field can be partitioned from Internet traffic that uses the default ECN semantics. There are two main examples for how alternative ECN semantics have been defined in practice:¶

The aim is that the default rules for encapsulating and decapsulating the ECN field are sufficiently generic that tunnels and subnets will encapsulate and decapsulate packets without regard to how algorithms elsewhere are setting or interpreting the semantics of the ECN field. [RFC6040] updates [RFC4774] to allow alternative encapsulation and decapsulation behaviours to be defined for alternative ECN semantics. However, it reinforces the same point -- it is far preferable to try to fit within the common ECN encapsulation and decapsulation behaviours because expecting all lower-layer technologies and tunnels to be updated is likely to be completely impractical.¶

Alternative semantics for the ECN field can be defined to depend on the traffic class indicated by the Differentiated Services Code Point (DSCP). Therefore, correct propagation of congestion signals could depend on correct propagation of the DSCP between the layers and along the path. For instance, if the meaning of the ECN field depends on the DSCP (as in PCN or VoLTE) and the outer DSCP is stripped on descapsulation, as in the pipe model of [RFC2983], the special semantics of the ECN field would be lost. Similarly, if the DSCP is changed at the boundary between Diffserv domains, the special ECN semantics would also be lost. This is an important implication of the localized scope of most Diffserv arrangements. In this document, correct propagation of traffic class information is assumed while the meaning of 'correct' and how it is achieved is covered elsewhere (e.g., [RFC2983]) and is outside the scope of this document.¶

The guidelines in this document do ensure that common encapsulation and decapsulation rules are sufficiently generic to cover cases where ECT(1) is used instead of ECT(0) to identify alternative ECN semantics (as in L4S [RFC9331]) and where ECN-marking algorithms use ECT(1) to encode three severity levels into the ECN field (e.g., PCN [RFC6660]) rather than the default of two. All these different semantics for the ECN field work because it has been possible to define common default decapsulation rules that allow for all cases [RFC6040].¶

Note that the guidelines in this document do not necessarily require the subnet wire protocol to be changed to add support for congestion notification. For instance, the feed-up-and-forward mode (Section 3.2) and the null mode (Section 3.4) do not. Another way to add congestion notification without consuming header space in the subnet protocol might be to use a parallel control plane protocol.¶

This document focuses on the congestion notification interface between IP and lower-layer or tunnel protocols that can encapsulate IP, where the term 'IP' includes IPv4 or IPv6, unicast, multicast, or anycast. However, it is likely that the guidelines will also be useful when a lower-layer protocol or tunnel encapsulates itself, e.g., Ethernet Media Access Control (MAC) in MAC ([IEEE802.1Q]; previously 802.1ah), or when it encapsulates other protocols. In the feed-backward mode, propagation of congestion signals for multicast and anycast packets is out of scope (because the complexity would make it unlikely to be attempted).¶

3.1. Feed-Forward-and-Up Mode

Like IP and MPLS, many subnet technologies are based on self-contained PDUs or frames sent unreliably. They provide no feedback channel at the subnetwork layer, instead relying on higher layers (e.g., TCP) to feed back loss signals.¶

In these cases, ECN may best be supported by standardising explicit notification of congestion into the lower-layer protocol that carries the data forwards. Then, a specification is needed for how the egress of the lower-layer subnet propagates this explicit signal into the forwarded upper-layer (IP) header. This signal continues forwards until it finally reaches the destination transport (at L4). Typically, the destination will feed this congestion notification back to the source transport using an end-to-end protocol (e.g., TCP). This is the arrangement that has already been used to add ECN to IP-in-IP tunnels [RFC6040], IP-in-MPLS, and MPLS-in-MPLS [RFC5129].¶

This mode is illustrated in Figure 1. Along the middle of the figure, layers 2, 3, and 4 of the protocol stack are shown. One packet is shown along the bottom as it progresses across the network from source to destination, crossing two subnets connected by a router and crossing two switches on the path across each subnet. Congestion at the output of the first switch (shown as *) leads to a congestion marking in the L2 header (shown as C in the illustration of the packet). The chevrons show the progress of the resulting congestion indication. It is propagated from link to link across the subnet in the L2 header. Then, when the router removes the marked L2 header, it propagates the marking up into the L3 (IP) header. The router forwards the marked L3 header into subnet B. The L2 protocol used in subnet B does not support congestion notification, but the signal proceeds across it in the L3 header.¶

Note that there is no implication that each 'C' marking is encoded the same; a different encoding might be used for the 'C' marking in each protocol.¶

Finally, for completeness, we show the L3 marking arriving at the destination, where the host transport protocol (e.g., TCP) feeds it back to the source in the L4 acknowledgement (the 'C' at L4 in the packet at the top of the diagram).¶

                    _ _ _
         /_______  | | |C|  ACK Packet (V)
         \         |_|_|_|
+---+        layer: 2 3 4 header                            +---+
|  <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4
|   |                         +---+                         | ^ |
|   | . . . . . . Packet U. . | >>|>>> Packet U >>>>>>>>>>>>|>^ |L3
|   |     +---+     +---+     | ^ |     +---+     +---+     |   |
|   |     |  *|>>>>>|>>>|>>>>>|>^ |     |   |     |   |     |   |L2
|___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___|
source          subnet A      router       subnet B         dest
   __ _ _ _|   __ _ _ _|  __ _ _|       __ _ _ _|
  |  | | | |  |  | | |C| |  | |C|      |  | |C| |    Data________\
  |__|_|_|_|  |__|_|_|_| |__|_|_|      |__|_|_|_|    Packet (U)  /
layer:4 3 2A      4 3 2A     4 3           4 3 2B
header

Of course, modern networks are rarely as simple as this textbook example, often involving multiple nested layers. For example, a Third Generation Partnership Project (3GPP) mobile network may have two IP-in-IP GTP [GTPv1] tunnels in series and an MPLS backhaul between the base station and the first router. Nonetheless, the example illustrates the general idea of feeding congestion notification forward then upward whenever a header is removed at the egress of a subnet.¶

Note that the Forward Explicit Congestion Notification (FECN) bit in Frame Relay [Buck00] and the Explicit Forward Congestion Indication (EFCI) [ITU-T.I.371] bit in ATM user data cells follow a feed-forward pattern. However, in ATM, this arrangement is only part of a feed-forward-and-backward pattern at the lower layer, not feed-forward-and-up out of the lower layer -- the intention was never to interface with IP-ECN at the subnet egress. To our knowledge, Frame Relay FECN is solely used by network operators to detect where they should provision more capacity.¶

3.2. Feed-Up-and-Forward Mode

Ethernet is particularly difficult to extend incrementally to support congestion notification. One way is to use so-called 'Layer 3 switches'. These are Ethernet switches that dig into the Ethernet payload to find an IP header and manipulate or act on certain IP fields (specifically Diffserv and ECN). For instance, in Data Center TCP [RFC8257], Layer 3 switches are configured to mark the ECN field of the IP header within the Ethernet payload when their output buffer becomes congested. With respect to switching, a Layer 3 switch acts solely on the addresses in the Ethernet header; it does not use IP addresses and it does not decrement the TTL field in the IP header.¶

                    _ _ _
         /_______  | | |C|  ACK packet (V)
         \         |_|_|_|
+---+        layer: 2 3 4 header                            +---+
|  <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4
|   |                         +---+                         | ^ |
|   | . . .  >>>> Packet U >>>|>>>|>>> Packet U >>>>>>>>>>>>|>^ |L3
|   |     +--^+     +---+     |  v|     +---+     +---+     | ^ |
|   |     |  *|     |   |     |  >|>>>>>|>>>|>>>>>|>>>|>>>>>|>^ |L2
|___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___|
source          subnet E      router       subnet F         dest
   __ _ _ _|   __ _ _ _|  __ _ _|       __ _ _ _|
  |  | | | |  |  | |C| | |  | |C|      |  | |C|C|    Data________\
  |__|_|_|_|  |__|_|_|_| |__|_|_|      |__|_|_|_|    Packet (U)  /
layer:4 3 2       4 3 2      4 3           4 3 2
header

By comparing Figure 2 with Figure 1, it can be seen that subnet E (perhaps a subnet of Layer 3 Ethernet switches) works in feed-up-and-forward mode by notifying congestion directly into L3 at the point of congestion, even though the congested switch does not otherwise act at L3. In this example, the technology in subnet F (e.g., MPLS) does support ECN. So, when the router adds the Layer 2 header, it copies the ECN marking from L3 to L2 as well, as shown by the 'C's in both layers.¶

3.3. Feed-Backward Mode

In some Layer 2 technologies, congestion notification has been defined for use internally within the subnet with its own feedback and load regulation but the interface with IP for ECN has not been defined.¶

For instance, the relative rate mechanism was one of the more popular ways to manage traffic for the Available Bit Rate (ABR) service in ATM, and it tended to supersede earlier designs. In this approach, ATM switches send special resource management (RM) cells in both the forward and backward directions to control the ingress rate of user data into a virtual circuit. If a switch buffer is approaching congestion or is congested, it sends an RM cell back towards the ingress with respectively the No Increase (NI) or Congestion Indication (CI) bit set in its message type field [ATM-TM-ABR]. The ingress then holds or decreases its sending bit rate accordingly.¶

                     _ _ _
          /_______  | | |C|  ACK packet (X)
          \         |_|_|_|
 +---+        layer: 2 3 4 header                            +---+
 |  <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet X <<<<<<<<<<<<<|<< |L4
 |   |                         +---+                         | ^ |
 |   |                         |  *|>>> Packet W >>>>>>>>>>>>|>^ |L3
 |   |     +---+     +---+     |   |     +---+     +---+     |   |
 |   |     |   |     |   |     |  <|<<<<<|<<<|<(V)<|<<<|     |   |L2
 |   | . . | . |Packet U | . . | . | . . | . | . . | .*| . . |   |L2
 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___|
 source          subnet G      router       subnet H         dest
     __ _ _ _    __ _ _ _    __ _ _        __ _ _ _   later
    |  | | | |  |  | | | |  |  | | |      |  | |C| |  data________\
    |__|_|_|_|  |__|_|_|_|  |__|_|_|      |__|_|_|_|  packet (W)  /
        4 3 2       4 3 2       4 3           4 3 2
                                        _
                                  /__  |C|  Feedback control
                                  \    |_|  cell/frame (V)
                                        2
     __ _ _ _    __ _ _ _    __ _ _        __ _ _ _   earlier
    |  | | | |  |  | | | |  |  | | |      |  | | | |  data________\
    |__|_|_|_|  |__|_|_|_|  |__|_|_|      |__|_|_|_|  packet (U)  /
layer:  4 3 2       4 3 2       4 3           4 3 2
header

ATM's feed-backward approach does not fit well when layered beneath IP's feed-forward approach unless the initial data source is the same node as the ATM ingress. Figure 3 shows the feed-backward approach being used in subnet H. If the final switch on the path is congested (*), it does not feed forward any congestion indications on the packet (U). Instead, it sends a control cell (V) back to the router at the ATM ingress.¶

However, the backward feedback does not reach the original data source directly because IP does not support backward feedback (and subnet G is independent of subnet H). Instead, the router in the middle throttles down its sending rate, but the original data sources don't reduce their rates. The resulting rate mismatch causes the middle router's buffer at layer 3 to back up until it becomes congested, which it signals forwards on later data packets at layer 3 (e.g., packet W). Note that the forward signal from the middle router is not triggered directly by the backward signal. Rather, it is triggered by congestion resulting from the middle router's mismatched rate response to the backward signal.¶

In response to this later forward signalling, end-to-end feedback at layer 4 finally completes the tortuous path of congestion indications back to the origin data source as before.¶

Quantized Congestion Notification (QCN) [IEEE802.1Q] would suffer from similar problems if extended to multiple subnets. However, QCN was clearly characterized as solely applicable to a single subnet from the start (see Section 6).¶

3.4. Null Mode

Link- and physical-layer resources are often 'non-blocking' by design. Congestion notification may be implemented in these cases, but it does not need to be deployed at the lower layer; ECN in IP would be sufficient.¶

A degenerate example is a point-to-point Ethernet link. Excess loading of the link merely causes the queue from the higher layer to back up, while the lower layer remains immune to congestion. Even a whole meshed subnetwork can be made immune to interior congestion by limiting ingress capacity and sufficient sizing of interior links, e.g., a non-blocking fat-tree network [Leiserson85]. An alternative to fat links near the root is numerous thin links with multi-path routing to ensure even worst-case patterns of load cannot congest any link, e.g., a Clos network [Clos53].¶

4.1. IP-in-IP Tunnels with Shim Headers

A common pattern for many tunnelling protocols is to encapsulate an inner IP header with a shim header(s) then an outer IP header. A shim header is defined as one that is not sufficient alone to forward the packet as an outer header. Another common pattern is for a shim to encapsulate an L2 header, which in turn encapsulates (or might encapsulate) an IP header. [RFC9601] clarifies that [RFC6040] is just as applicable when there are shims and even an L2 header between two IP headers.¶

However, it is not always feasible or necessary to propagate ECN between IP headers when separated by a shim. For instance, it might be too costly to dig to arbitrary depths to find an inner IP header, there may be little or no congestion within the tunnel by design (see null mode in Section 3.4 above), or a legacy implementation might not support ECN. In cases where a tunnel does not support ECN, it is important that the ingress does not copy the ECN field from an inner IP header to an outer. Therefore Section 4 of [RFC9601] requires network operators to configure the ingress of a tunnel that does not support ECN so that it zeros the ECN field in the outer IP header.¶

Nonetheless, in many cases it is feasible to propagate the ECN field between IP headers separated by shim headers and/or an L2 header. Particularly in the typical case when the outer IP header and the shim(s) are added (or removed) as part of the same procedure. Even if a shim encapsulates an L2 header, it is often possible to find an inner IP header within the L2 PDU and propagate ECN between that and the outer IP header. This can be thought of as a special case of the feed-up-and-forward mode (Section 3.2), so the guidelines for this mode apply (Section 5).¶

Numerous shim protocols have been defined for IP tunnelling. More recent ones, e.g., Geneve [RFC8926] and Generic UDP Encapsulation (GUE) [INTAREA-GUE] cite and follow [RFC6040]. Some earlier ones, e.g., CAPWAP [RFC5415] and LISP [RFC9300], cite [RFC3168], which is compatible with [RFC6040].¶

However, as Section 9.3 of [RFC3168] pointed out, ECN support needs to be defined for many earlier shim-based tunnelling protocols, e.g., L2TPv2 [RFC2661], L2TPv3 [RFC3931], GRE [RFC2784], PPTP [RFC2637], GTP [GTPv1] [GTPv1-U] [GTPv2-C], and Teredo [RFC4380], as well as some recent ones, e.g., VXLAN [RFC7348], NVGRE [RFC7637], and NSH [RFC8300].¶

All these IP-based encapsulations can be updated in one shot by simple reference to [RFC6040]. However, it would not be appropriate to update all these protocols from within the present guidance document. Instead, a companion specification [RFC9601] has the appropriate Standards Track status to update Standards Track protocols. For those that are not under IETF change control [RFC9601] can only recommend that the relevant body updates them.¶

4.2. Wire Protocol Design: Indication of ECN Support

This section is intended to guide the redesign of any lower-layer protocol that encapsulates IP to add built-in congestion notification support at the lower layer using feed-forward-and-up mode. It reflects the approaches used in [RFC6040] and in [RFC5129]. Therefore, IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with [RFC6040] or [RFC5129] will already satisfy this guidance.¶

A lower-layer (or subnet) congestion notification system:¶

Note that there is no need for all interior nodes within a subnet to be able to mark congestion explicitly. A mix of drop and explicit congestion signals from different nodes is fine. However, if any interior nodes might generate congestion markings, Guideline 2 above says that all relevant egress nodes SHOULD be able to propagate those markings up to the higher layer.¶

In IP, if the ECN field in each PDU is cleared to the Not ECN-Capable Transport (Not-ECT) codepoint, it indicates that the L4 transport will not understand congestion markings. A congested buffer must not mark these Not-ECT PDUs; therefore, it has to signal congestion by increasingly applying drop instead.¶

The mechanism a lower layer uses to distinguish the ECN capability of PDUs need not mimic that of IP. The above guidelines merely say that the lower-layer system as a whole should achieve the same outcome. For instance, ECN-capable feedback loops might use PDUs that are identified by a particular set of labels or tags. Alternatively, logical-link protocols that use flow state might determine whether a PDU can be congestion marked by checking for ECN support in the flow state. Other protocols might depend on out-of-band control signals.¶

The per-domain checking of ECN support in MPLS [RFC5129] is a good example of a way to avoid sending congestion markings to L4 transports that will not understand them without using any header space in the subnet protocol.¶

In MPLS, header space is extremely limited; therefore, [RFC5129] does not provide a field in the MPLS header to indicate whether the PDU is an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are allowed to set explicit congestion indications without checking whether the PDU is destined for a L4 transport that will understand them. Nonetheless, this is made safe by requiring that the network operator upgrades all decapsulating edges of a whole domain at once as soon as even one switch within the domain is configured to mark rather than drop some PDUs during congestion. Therefore, any edge node that might decapsulate a packet will be capable of checking whether the higher-layer transport is ECN-capable. When decapsulating a CE-marked packet, if the decapsulator discovers that the higher layer (inner header) indicates the transport is not ECN-capable, it drops the packet -- effectively on behalf of the earlier congested node (see Decapsulation Guideline 1 in Section 4.4).¶

It was only appropriate to define such an incremental deployment strategy because MPLS is targeted solely at professional operators who can be expected to ensure that a whole subnetwork is consistently configured. This strategy might not be appropriate for other link technologies targeted at zero-configuration deployment or deployment by the general public (e.g., Ethernet). For such 'plug-and-play' environments, it will be necessary to invent a fail-safe approach that ensures congestion markings will never fall into black holes, no matter how inconsistently a system is put together. Alternatively, congestion notification relying on correct system configuration could be confined to flavours of Ethernet intended only for professional network operators, such as Provider Backbone Bridges (PBB) ([IEEE802.1Q]; previously 802.1ah).¶

ECN support in TRansparent Interconnection of Lots of Links (TRILL) [RFC9600] provides a good example of how to add congestion notification to a lower-layer protocol without relying on careful and consistent operator configuration. TRILL provides an extension header word with space for flags of different categories depending on whether logic to understand the extension is critical. The congestion-experienced marking has been defined as a 'critical ingress-to-egress' flag. So, if a transit RBridge sets this flag on a frame and an egress RBridge does not have any logic to process it, the egress RBridge will drop the frame, which is the desired default action anyway. Therefore, TRILL RBridges can be updated with support for congestion notification in no particular order and, at the egress of the TRILL campus, congestion notification will be propagated to IP as ECN whenever ECN logic has been implemented at the egress, or as drop otherwise.¶

QCN [IEEE802.1Q] is not intended to extend beyond a single subnet or interoperate with IP-ECN. Nonetheless, the way QCN indicates to lower-layer devices that the endpoints will not understand QCN provides another example that a lower-layer protocol designer might be able to mimic for their scenario. An operator can define certain Priority Code Points (PCPs [IEEE802.1Q]; previously 802.1p) to indicate non-QCN frames. Then an ingress bridge has to map each arriving not-QCN-capable IP packet to one of these non-QCN PCPs.¶

When drop for non-ECN traffic is deferred to the egress of a subnet, it cannot necessarily be assumed that one congestion mark is equivalent to one drop, as was originally required by [RFC3168]. [RFC8311] updated [RFC3168] to allow experimentation with congestion markings that are not equivalent to drop, particularly for L4S [RFC9331]. ECN support in TRILL [RFC9600] is a good example of a way to defer drop to the egress of a subnet both when marks are equivalent to drops (as in [RFC3168]) and when they are not (as in L4S). The ECN scheme for MPLS [RFC5129] was defined before L4S, so it only currently supports deferred drop that is equivalent to ECN marking. Nonetheless, in principle, MPLS (and potentially future L2 protocols) could support L4S marking by copying TRILL's approach for determining the drop level of any non-ECN traffic at the subnet egress.¶

4.3. Encapsulation Guidelines

This section is intended to guide the redesign of any node that encapsulates IP with a lower-layer header when adding built-in congestion notification support to the lower-layer protocol using feed-forward-and-up mode. It reflects the approaches used in [RFC6040] and [RFC5129]. Therefore, IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with [RFC6040] or [RFC5129] will already satisfy this guidance.¶

4.4. Decapsulation Guidelines

This section is intended to guide the redesign of any node that decapsulates IP from within a lower-layer header when adding built-in congestion notification support to the lower-layer protocol using feed-forward-and-up mode. It reflects the approaches used in [RFC6040] and in [RFC5129]. Therefore, IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with [RFC6040] or [RFC5129] will already satisfy this guidance.¶

A subnet egress SHOULD NOT simply copy congestion notifications from outer headers to the forwarded header. It SHOULD calculate the outgoing congestion notification field from the inner and outer headers using the following guidelines. If there is any conflict, rules earlier in the list take precedence over rules later in the list.¶

4.5. Sequences of Similar Tunnels or Subnets

In some deployments, particularly in 3GPP networks, an IP packet may traverse two or more IP-in-IP tunnels in sequence that all use identical technology (e.g., GTP).¶

In such cases, it would be sufficient for every encapsulation and decapsulation in the chain to comply with [RFC6040]. Alternatively, as an optimization, a node that decapsulates a packet and immediately re-encapsulates it for the next tunnel MAY copy the incoming outer ECN field directly to the outgoing outer header and the incoming inner ECN field directly to the outgoing inner header. Then, the overall behaviour across the sequence of tunnel segments would still be consistent with [RFC6040].¶

Appendix C of [RFC6040] describes how a tunnel egress can monitor how much congestion has been introduced within a tunnel. A network operator might want to monitor how much congestion had been introduced within a whole sequence of tunnels. Using the technique in Appendix C of [RFC6040] at the final egress, the operator could monitor the whole sequence of tunnels, but only if the above optimization were used consistently along the sequence of tunnels, in order to make it appear as a single tunnel. Therefore, tunnel endpoint implementations SHOULD allow the operator to configure whether this optimization is enabled.¶

When congestion notification support is added to a subnet technology, consideration SHOULD be given to a similar optimization between subnets in sequence if they all use the same technology.¶

4.6. Reframing and Congestion Markings

The guidance in this section is worded in terms of framing boundaries, but it applies equally whether the PDUs are frames, cells, or packets.¶

Where an AQM marks the ECN field of IP packets as they queue into a Layer 2 link, there will be no problem with framing boundaries because the ECN markings would be applied directly to IP packets. The guidance in this section is only applicable where a congestion notification capability is being added to a Layer 2 protocol so that Layer 2 frames can be marked by an AQM at layer 2. This would only be necessary where AQM will be applied at pure Layer 2 nodes (without IP awareness).¶

Where congestion marking has had to be applied at non-IP-aware nodes and framing boundaries do not necessarily align with packet boundaries, the decapsulating IP forwarding node SHOULD propagate congestion markings from Layer 2 frame headers to IP packets that may have different boundaries as a consequence of reframing.¶

Two possible design goals for propagating congestion indications, described in Section 5.3 of [RFC3168] and Section 2.4 of [RFC7141], are:¶

In either case, an implementation SHOULD ensure that any new incoming congestion indication is propagated immediately; not held awaiting the possibility of further congestion indications to be sufficient to indicate congestion on an outgoing PDU [RFC7141]. Nonetheless, to facilitate pipelined implementation, it would be acceptable for congestion marks to propagate to a slightly later IP packet.¶

At decapsulation in either case:¶

The following are two ways that goal 1 might be achieved, but they are not intended to be the only ways:¶

The following gives one way that goal 2 might be achieved, but it is not intended to be the only way:¶

Generally, relative to the number of IP PDUs, the number of L2 frames may be higher (e.g., ATM), roughly the same, or lower (e.g., 802.11 aggregation at an L2-only station). This distinction may influence the choice of mechanism.¶