RFC 9330: Low Latency, Low Loss, and Scalable Throughput (L4S) Internet Service: Architecture

4.1. Protocol Mechanisms

The L4S architecture involves: a) unassignment of the previous use of the identifier; b) reassignment of the same identifier; and c) optional further identifiers:¶

4.2. Network Components

The L4S architecture aims to provide low latency without the need for per-flow operations in network components. Nonetheless, the architecture does not preclude per-flow solutions. The following bullets describe the known arrangements: a) the DualQ Coupled AQM with an L4S AQM in one queue coupled from a Classic AQM in the other; b) per-flow queues with an instance of a Classic and an L4S AQM in each queue; and c) Dual queues with per-flow AQMs but no per-flow queues:¶

4.3. Host Mechanisms

The L4S architecture includes two main mechanisms in the end host that we enumerate next:¶

5.1. Why These Primary Components?

Explicit congestion signalling (protocol):

Explicit congestion signalling is a key part of the L4S approach. In contrast, use of drop as a congestion signal creates tension because drop is both an impairment (less would be better) and a useful signal (more would be better):¶

• Explicit congestion signals can be used many times per round trip to keep tight control without any impairment. Under heavy load, even more explicit signals can be applied so that the queue can be kept short whatever the load. In contrast, Classic AQMs have to introduce very high packet drop at high load to keep the queue short. By using ECN, an L4S congestion control's sawtooth reduction can be smaller and therefore return to the operating point more often, without worrying that more sawteeth will cause more signals. The consequent smaller amplitude sawteeth fit between an empty queue and a very shallow marking threshold (~1 ms in the public Internet), so queue delay variation can be very low, without risk of underutilization.¶
• Explicit congestion signals can be emitted immediately to track fluctuations of the queue. L4S shifts smoothing from the network to the host. The network doesn't know the round-trip times (RTTs) of any of the flows. So if the network is responsible for smoothing (as in the Classic approach), it has to assume a worst case RTT, otherwise long RTT flows would become unstable. This delays Classic congestion signals by 100-200 ms. In contrast, each host knows its own RTT. So, in the L4S approach, the host can smooth each flow over its own RTT, introducing no more smoothing delay than strictly necessary (usually only a few milliseconds). A host can also choose not to introduce any smoothing delay if appropriate, e.g., during flow start-up.¶

Neither of the above are feasible if explicit congestion signalling has to be considered 'equivalent to drop' (as was required with Classic ECN [RFC3168]), because drop is an impairment as well as a signal. So drop cannot be excessively frequent, and drop cannot be immediate; otherwise, too many drops would turn out to have been due to only a transient fluctuation in the queue that would not have warranted dropping a packet in hindsight. Therefore, in an L4S AQM, the L4S queue uses a new L4S variant of ECN that is not equivalent to drop (see Section 5.2 of the L4S ECN spec [RFC9331]), while the Classic queue uses either Classic ECN [RFC3168] or drop, which are still equivalent to each other.¶

Before Classic ECN was standardized, there were various proposals to give an ECN mark a different meaning from drop. However, there was no particular reason to agree on any one of the alternative meanings, so 'equivalent to drop' was the only compromise that could be reached. [RFC3168] contains a statement that:¶

• An environment where all end nodes were ECN-Capable could allow new criteria to be developed for setting the CE codepoint, and new congestion control mechanisms for end-node reaction to CE packets. However, this is a research issue, and as such is not addressed in this document.¶

Latency isolation (network):

L4S congestion controls keep queue delay low, whereas Classic congestion controls need a queue of the order of the RTT to avoid underutilization. One queue cannot have two lengths; therefore, L4S traffic needs to be isolated in a separate queue (e.g., DualQ) or queues (e.g., FQ).¶

Coupled congestion notification:

Coupling the congestion notification between two queues as in the DualQ Coupled AQM is not necessarily essential, but it is a simple way to allow senders to determine their rate packet by packet, rather than be overridden by a network scheduler. An alternative is for a network scheduler to control the rate of each application flow (see the discussion in Section 5.2).¶

L4S packet identifier (protocol):

Once there are at least two treatments in the network, hosts need an identifier at the IP layer to distinguish which treatment they intend to use.¶

Scalable congestion notification:

A Scalable congestion control in the host keeps the signalling frequency from the network high, whatever the flow rate, so that queue delay variations can be small when conditions are stable, and rate can track variations in available capacity as rapidly as possible otherwise.¶

Low loss:

Latency is not the only concern of L4S. The 'Low Loss' part of the name denotes that L4S generally achieves zero congestion loss due to its use of ECN. Otherwise, loss would itself cause delay, particularly for short flows, due to retransmission delay [RFC2884].¶

Scalable throughput:

The 'Scalable throughput' part of the name denotes that the per-flow throughput of Scalable congestion controls should scale indefinitely, avoiding the imminent scaling problems with Reno-friendly congestion control algorithms [RFC3649]. It was known when TCP congestion avoidance was first developed in 1988 that it would not scale to high bandwidth-delay products (see footnote 6 in [TCP-CA]). Today, regular broadband flow rates over WAN distances are already beyond the scaling range of Classic Reno congestion control. So 'less unscalable' CUBIC [RFC8312] and Compound [CTCP] variants of TCP have been successfully deployed. However, these are now approaching their scaling limits.¶

For instance, we will consider a scenario with a maximum RTT of 30 ms at the peak of each sawtooth. As Reno packet rate scales 8 times from 1,250 to 10,000 packet/s (from 15 to 120 Mb/s with 1500 B packets), the time to recover from a congestion event rises proportionately by 8 times as well, from 422 ms to 3.38 s. It is clearly problematic for a congestion control to take multiple seconds to recover from each congestion event. CUBIC [RFC8312] was developed to be less unscalable, but it is approaching its scaling limit; with the same max RTT of 30 ms, at 120 Mb/s, CUBIC is still fully in its Reno-friendly mode, so it takes about 4.3 s to recover. However, once flow rate scales by 8 times again to 960 Mb/s it enters true CUBIC mode, with a recovery time of 12.2 s. From then on, each further scaling by 8 times doubles CUBIC's recovery time (because the cube root of 8 is 2), e.g., at 7.68 Gb/s, the recovery time is 24.3 s. In contrast, a Scalable congestion control like DCTCP or Prague induces 2 congestion signals per round trip on average, which remains invariant for any flow rate, keeping dynamic control very tight.¶

For a feel of where the global average lone-flow download sits on this scale at the time of writing (2021), according to [BDPdata], the global average fixed access capacity was 103 Mb/s in 2020 and the average base RTT to a CDN was 25 to 34 ms in 2019. Averaging of per-country data was weighted by Internet user population (data collected globally is necessarily of variable quality, but the paper does double-check that the outcome compares well against a second source). So a lone CUBIC flow would at best take about 200 round trips (5 s) to recover from each of its sawtooth reductions, if the flow even lasted that long. This is described as 'at best' because it assumes everyone uses an AQM, whereas in reality, most users still have a (probably bloated) tail-drop buffer. In the tail-drop case, the likely average recovery time would be at least 4 times 5 s, if not more, because RTT under load would be at least double that of an AQM, and the recovery time of Reno-friendly flows depends on the square of RTT.¶

Although work on scaling congestion controls tends to start with TCP as the transport, the above is not intended to exclude other transports (e.g., SCTP and QUIC) or less elastic algorithms (e.g., RMCAT), which all tend to adopt the same or similar developments.¶

5.2. What L4S Adds to Existing Approaches

All the following approaches address some part of the same problem space as L4S. In each case, it is shown that L4S complements them or improves on them, rather than being a mutually exclusive alternative:¶

Diffserv:

Diffserv addresses the problem of bandwidth apportionment for important traffic as well as queuing latency for delay-sensitive traffic. Of these, L4S solely addresses the problem of queuing latency. Diffserv will still be necessary where important traffic requires priority (e.g., for commercial reasons or for protection of critical infrastructure traffic) -- see [L4S-DIFFSERV]. Nonetheless, the L4S approach can provide low latency for all traffic within each Diffserv class (including the case where there is only the one default Diffserv class).¶

Also, Diffserv can only provide a latency benefit if a small subset of the traffic on a bottleneck link requests low latency. As already explained, it has no effect when all the applications in use at one time at a single site (e.g., a home, small business, or mobile device) require low latency. In contrast, because L4S works for all traffic, it needs none of the management baggage (traffic policing or traffic contracts) associated with favouring some packets over others. This lack of management baggage ought to give L4S a better chance of end-to-end deployment.¶

In particular, if networks do not trust end systems to identify which packets should be favoured, they assign packets to Diffserv classes themselves. However, the techniques available to such networks, like inspection of flow identifiers or deeper inspection of application signatures, do not always sit well with encryption of the layers above IP [RFC8404]. In these cases, users can have either privacy or Quality of Service (QoS), but not both.¶

As with Diffserv, the L4S identifier is in the IP header. But, in contrast to Diffserv, the L4S identifier does not convey a want or a need for a certain level of quality. Rather, it promises a certain behaviour (Scalable congestion response), which networks can objectively verify if they need to. This is because low delay depends on collective host behaviour, whereas bandwidth priority depends on network behaviour.¶

State-of-the-art AQMs:

AQMs for Classic traffic, such as PIE and FQ-CoDel, give a significant reduction in queuing delay relative to no AQM at all. L4S is intended to complement these AQMs and should not distract from the need to deploy them as widely as possible. Nonetheless, AQMs alone cannot reduce queuing delay too far without significantly reducing link utilization, because the root cause of the problem is on the host -- where Classic congestion controls use large sawtoothing rate variations. The L4S approach resolves this tension between delay and utilization by enabling hosts to minimize the amplitude of their sawteeth. A single-queue Classic AQM is not sufficient to allow hosts to use small sawteeth for two reasons: i) smaller sawteeth would not get lower delay in an AQM designed for larger amplitude Classic sawteeth, because a queue can only have one length at a time and ii) much smaller sawteeth implies much more frequent sawteeth, so L4S flows would drive a Classic AQM into a high level of ECN-marking, which would appear as heavy congestion to Classic flows, which in turn would greatly reduce their rate as a result (see Section 6.4.4).¶

Per-flow queuing or marking:

Similarly, per-flow approaches, such as FQ-CoDel or Approx Fair CoDel [AFCD], are not incompatible with the L4S approach. However, per-flow queuing alone is not enough -- it only isolates the queuing of one flow from others, not from itself. Per-flow implementations need to have support for Scalable congestion control added, which has already been done for FQ-CoDel in Linux (see Section 5.2.7 of [RFC8290] and [FQ_CoDel_Thresh]). Without this simple modification, per-flow AQMs, like FQ-CoDel, would still not be able to support applications that need both very low delay and high bandwidth, e.g., video-based control of remote procedures or interactive cloud-based video (see Note 1 below).¶

Although per-flow techniques are not incompatible with L4S, it is important to have the DualQ alternative. This is because handling end-to-end (layer 4) flows in the network (layer 3 or 2) precludes some important end-to-end functions. For instance:¶

1. Per-flow forms of L4S, like FQ-CoDel, are incompatible with full end-to-end encryption of transport layer identifiers for privacy and confidentiality (e.g., IPsec or encrypted VPN tunnels, as opposed to DTLS over UDP), because they require packet inspection to access the end-to-end transport flow identifiers.¶

   In contrast, the DualQ form of L4S requires no deeper inspection than the IP layer. So as long as operators take the DualQ approach, their users can have both very low queuing delay and full end-to-end encryption [RFC8404].¶

2. With per-flow forms of L4S, the network takes over control of the relative rates of each application flow. Some see it as an advantage that the network will prevent some flows running faster than others. Others consider it an inherent part of the Internet's appeal that applications can control their rate while taking account of the needs of others via congestion signals. They maintain that this has allowed applications with interesting rate behaviours to evolve, for instance: i) a variable bit-rate video that varies around an equal share, rather than being forced to remain equal at every instant or ii) end-to-end scavenger behaviours [RFC6817] that use less than an equal share of capacity [LEDBAT_AQM].¶

   The L4S architecture does not require the IETF to commit to one approach over the other, because it supports both so that the 'market' can decide. Nonetheless, in the spirit of 'Do one thing and do it well' [McIlroy78], the DualQ option provides low delay without prejudging the issue of flow-rate control. Then, flow rate policing can be added separately if desired. In contrast to scheduling, a policer would allow application control up to a point, but the network would still be able to set the point at which it intervened to prevent one flow completely starving another.¶

Note:¶

1. It might seem that self-inflicted queuing delay within a per-flow queue should not be counted, because if the delay wasn't in the network, it would just shift to the sender. However, modern adaptive applications, e.g., HTTP/2 [RFC9113] or some interactive media applications (see Section 6.1), can keep low latency objects at the front of their local send queue by shuffling priorities of other objects dependent on the progress of other transfers (for example, see [lowat]). They cannot shuffle objects once they have released them into the network.¶

Alternative Back-off ECN (ABE):

Here again, L4S is not an alternative to ABE but a complement that introduces much lower queuing delay. ABE [RFC8511] alters the host behaviour in response to ECN marking to utilize a link better and give ECN flows faster throughput. It uses ECT(0) and assumes the network still treats ECN and drop the same. Therefore, ABE exploits any lower queuing delay that AQMs can provide. But, as explained above, AQMs still cannot reduce queuing delay too much without losing link utilization (to allow for other, non-ABE, flows).¶

BBR:

Bottleneck Bandwidth and Round-trip propagation time (BBR) [BBR-CC] controls queuing delay end-to-end without needing any special logic in the network, such as an AQM. So it works pretty much on any path. BBR keeps queuing delay reasonably low, but perhaps not quite as low as with state-of-the-art AQMs, such as PIE or FQ-CoDel, and certainly nowhere near as low as with L4S. Queuing delay is also not consistently low, due to BBR's regular bandwidth probing spikes and its aggressive flow start-up phase.¶

L4S complements BBR. Indeed, BBRv2 can use L4S ECN where available and a Scalable L4S congestion control behaviour in response to any ECN signalling from the path [BBRv2]. The L4S ECN signal complements the delay-based congestion control aspects of BBR with an explicit indication that hosts can use, both to converge on a fair rate and to keep below a shallow queue target set by the network. Without L4S ECN, both these aspects need to be assumed or estimated.¶

6.1. Applications

A transport layer that solves the current latency issues will provide new service, product, and application opportunities.¶

With the L4S approach, the following existing applications also experience significantly better quality of experience under load:¶

The significantly lower queuing latency also enables some interactive application functions to be offloaded to the cloud that would hardly even be usable today, including:¶

The above two applications have been successfully demonstrated with L4S, both running together over a 40 Mb/s broadband access link loaded up with the numerous other latency-sensitive applications in the previous list, as well as numerous downloads, with all sharing the same bottleneck queue simultaneously [L4Sdemo16] [L4Sdemo16-Video]. For the former, a panoramic video of a football stadium could be swiped and pinched so that, on the fly, a proxy in the cloud could generate a sub-window of the match video under the finger-gesture control of each user. For the latter, a virtual reality headset displayed a viewport taken from a 360-degree camera in a racing car. The user's head movements controlled the viewport extracted by a cloud-based proxy. In both cases, with a 7 ms end-to-end base delay, the additional queuing delay of roughly 1 ms was so low that it seemed the video was generated locally.¶

Using a swiping finger gesture or head movement to pan a video are extremely latency-demanding actions -- far more demanding than VoIP -- because human vision can detect extremely low delays of the order of single milliseconds when delay is translated into a visual lag between a video and a reference point (the finger or the orientation of the head sensed by the balance system in the inner ear, i.e., the vestibular system). With an alternative AQM, the video noticeably lagged behind the finger gestures and head movements.¶

Without the low queuing delay of L4S, cloud-based applications like these would not be credible without significantly more access-network bandwidth (to deliver all possible areas of the video that might be viewed) and more local processing, which would increase the weight and power consumption of head-mounted displays. When all interactive processing can be done in the cloud, only the data to be rendered for the end user needs to be sent.¶

Other low latency high bandwidth applications, such as:¶

are not credible at all without very low queuing delay. No amount of extra access bandwidth or local processing can make up for lost time.¶

6.3. Applicability with Specific Link Technologies

Certain link technologies aggregate data from multiple packets into bursts and buffer incoming packets while building each burst. Wi-Fi, PON, and cable all involve such packet aggregation, whereas fixed Ethernet and DSL do not. No sender, whether L4S or not, can do anything to reduce the buffering needed for packet aggregation. So an AQM should not count this buffering as part of the queue that it controls, given no amount of congestion signals will reduce it.¶

Certain link technologies also add buffering for other reasons, specifically:¶

L4S cannot remove the need for all these different forms of buffering. However, by removing 'the longest pole in the tent' (buffering for the large sawteeth of Classic congestion controls), L4S exposes all these 'shorter poles' to greater scrutiny.¶

Until now, the buffering needed for these additional reasons tended to be over-specified -- with the excuse that none were 'the longest pole in the tent'. But having removed the 'longest pole', it becomes worthwhile to minimize them, for instance, reducing packet aggregation burst sizes and MAC scheduling intervals.¶

Also, certain link types, particularly radio-based links, are far more prone to transmission losses. Section 6.4.3 explains how an L4S response to loss has to be as drastic as a Classic response. Nonetheless, research referred to in the same section has demonstrated potential for considerably more effective loss repair at the link layer, due to the relaxed ordering constraints of L4S packets.¶

8.3. Interaction between Rate Policing and L4S

As mentioned in Section 5.2, L4S should remove the need for low latency Diffserv classes. However, those Diffserv classes that give certain applications or users priority over capacity would still be applicable in certain scenarios (e.g., corporate networks). Then, within such Diffserv classes, L4S would often be applicable to give traffic low latency and low loss as well. Within such a Diffserv class, the bandwidth available to a user or application is often limited by a rate policer. Similarly, in the default Diffserv class, rate policers are sometimes used to partition shared capacity.¶

A Classic rate policer drops any packets exceeding a set rate, usually also giving a burst allowance (variants exist where the policer re-marks noncompliant traffic to a discard-eligible Diffserv codepoint, so they can be dropped elsewhere during contention). Whenever L4S traffic encounters one of these rate policers, it will experience drops and the source will have to fall back to a Classic congestion control, thus losing the benefits of L4S (Section 6.4.3). So in networks that already use rate policers and plan to deploy L4S, it will be preferable to redesign these rate policers to be more friendly to the L4S service.¶

L4S-friendly rate policing is currently a research area (note that this is not the same as latency policing). It might be achieved by setting a threshold where ECN marking is introduced, such that it is just under the policed rate or just under the burst allowance where drop is introduced. For instance, the two-rate, three-colour marker [RFC2698] or a PCN threshold and excess-rate marker [RFC5670] could mark ECN at the lower rate and drop at the higher. Or an existing rate policer could have congestion-rate policing added, e.g., using the 'local' (non-ConEx) variant of the ConEx aggregate congestion policer [CONG-POLICING]. It might also be possible to design Scalable congestion controls to respond less catastrophically to loss that has not been preceded by a period of increasing delay.¶

The design of L4S-friendly rate policers will require a separate, dedicated document. For further discussion of the interaction between L4S and Diffserv, see [L4S-DIFFSERV].¶