martinduke / draft-duke-quic-load-balancers Goto Github PK

This is an oversight from removing the protocol.

Feedback in Singapore was strongly in favor of removing the in-band protocol entirely. Given sufficient demand, it could rise again in another document.

@kazuho suggested in Tokyo that his server implementation would require at least 6B for server use, although it's fine if it's inside or outside the crypto.

Existing text:

PCID provides at least 2B, but as @kazuho isn't supporting that, meh.

SCID provides at least 8B for the server, though they must appear random.

For BCID, the load balancer MUST provide at last 2B but SHOULD provide 6.

Kazuho, is that sufficient?

Obviously, if servers and load balancers don't support the same algorithm they will not work with each other. There's not much to do there.

However, if servers have mutually exclusive algorithm sets, there is no way for load balancers to have a common regime.

One way to solve this is to force servers to support everything. Another would be to extend the solution to Issue #8, taking a handful of bits to encode various configuration regimes.

Perhaps inevitably, now that QUIC-LB provides a framework for coordination with trusted middleboxes, other middlebox functions may want to use it.

In New York there was talk of anti-DDoS boxes that would send RETRY on behalf of servers. To make this work, I believe we'd need to standardize the encoding of the original DCID in the token, so that the server can extract it and put it in its TPs.

Anyway, consider this a thread to discuss this possibility and a possible reference for a PR. I will not hold up the spec for this extension.

I've been looking at load balancing more lately and specifically trying to figure out an algorithm to use to statelessly load balance new connections that doesn't expose any type of attack surface.

The closest thing I've come up with is something like this:

int serverId;
if (packet.IsInitial()) {
    serverId = hash(key, packet.destCid)
} else {
    serverId = packet.destCid.ExtractServerId()
}

ExtractServerId will rely on whatever encoding scheme was chosen and get the server ID from that.

My problem with the above pseudocode is post initial Initial packets. The first Initial packet will use the client chosen CID, but after that it uses the server CID. That change in CID will break the above logic. Because of this, I've actually been considering opening a transport issue recommending all Initial packets use the client CID.

Is there another way to achieve this goal? And how much of this kind of stuff should be included in the QUIC-LB spec?

P.S. How would unknown (experimental?) version numbers factor into the above pseudocode? Just go to the else?

We should add some language describing how a multi-tiered low-state load balancer architecture might use QUIC-LB. There should be no normative changes.

QUIC-LB has no mechanism to gracefully change connection parameters, which seems bad. If we used one bit of the connection ID like a key phase bit, to encode the epoch, then it would be easier to gracefully migrate the data center incrementally from one scheme to the next. The load balancer would have to temporarily support both schemes.

Draft-25 changes the retry packet format, so update the retry services section.

There is some disagreements as to whether given attackers the ability to attack a single server would be a feature or a bug. Kazuho suggested that he wants to use QUIC-LB for this, and that he doesn't see the linkability issue as serious.

Blocking these attacks is (in my opinion) a nice side effect of the privacy goal, but we haven't explicitly considered it in the design. Should it be a formal objective of the draft?

I filed this issue so that others could register their feelings on this point.

The draft compares itself to "Packet Number Encryption" because it uses CTR. Although they are related, the QUIC spec now says AES-ECB, so we should probably not confuse people.

quic-transport-22 makes the change. It must reflect in this draft as well.

I don't believe the document specifies which bits carry the server ID. One of the packet diagrams contains a Routing Byte Mask field, but it is not explained.

In QUIC Interim, @MikeBishop suggested that it is sometimes beneficial to block malicious QUIC-like traffic at the edge on an network, so that the network behind the edge does not need to be over-provisioned.

I think that can be done using the capability of QUIC-LB, or by possibly just slightly adjusting the design. Essentially, what we need is a LB that validates the CID and drops invalid ones but respects the 5-tuple in terms of forwarding packets, instead of routing the packets based on the CID. Nodes behind the edge would calculate the MAC value and embed it as part of the CID they issue.

Maybe it'd be worth checking such a use-case can covered by QUIC-LB, as well as clarifying how that can be done.

At the moment, the AES-CTR nonce is (scid_len - sid_len). This should really be 16 Bytes.

There are a number of different configuration values that need to be passed to all back end servers. One of these is the secret used for encrypting 0-RTT tickets. Should QUIC-LB be used for sending these configuration values to the backend servers?

When making QUIC-LB version-neutral, I eliminated the distinction between Initial/0RTT packets and other sorts of long headers.

So, if an Initial packet shows up and has a CID long enough to go through the chosen algorithm, it might end up being obviously not server generated by having an invalid mapping or having some decrypted zero-padding not be zero. If a packet has long headers, it should forward these packets.

The presence of CONN_CLOSE in INITIAL and HANDSHAKE packets (and APP_CLOSE in HANDSHAKE) raises a number of questions for implementers.

1. If I must generate a CONN_CLOSE frame during the handshake, what context should I use? The highest one I have, or the highest that I received a packet for?

2. Relatedly, under what circumstances should I accept a CONN_CLOSE with old keys? I would argue we should toss any Initial Packet with CONN_CLOSE if the peer has already sent us something with Handshake keys, but it's possible to be more strict than that.

3. Should I accept a Handshake CONNECTION_CLOSE when we are using 1-RTT merely because I still have the Handshake keys lying around?

As far as I can tell, other frame types have no impact on the other contexts, except for a minor CRYPTO frame issue I'll handle in a different PR.

This issue is a catch-all for possible attacks on the current ECID algorithm, which uses AES-CTR with a nonce of at least 8B, meaning the CID must have a length of at least (# of bits to encode all the servers) + 64

Intel's Manasi Deval is interested in hardware acceleration of QUIC, certainly including crypto offload.

For this to work, there needs to be some indication of CID length in short headers. Alternatives are not viable.

1. Requiring a single CID length to work with the hardware is not compatible with many CID encoding schemes, unless that number is uncomfortably large (i.e. 18)
2. Specifying a minimum CID length for HW offload would allow disaggregation, but doesn't help find the packet number, which is important for decryption
3. A prefix matching approach (match bytes in succession until you find only one CID in the table that matches) can work, but is uneconomical for large tables.

One possibility is to make it part of the base standard.

However, to minimize that pain and keep CIDs completely opaque in the general case, another approach is to make it part of QUIC-LB. This views the crypto offload as another kind of "trusted middlebox". The encoding scheme could include four bits in plaintext that encodes the length. Given the minimum lengths in QUIC-LB, we could even get away with three bits if we disallowed a couple of otherwise legal values.

We might reduce the cost a bit by putting config rotation in the reserved bits of the first byte (thus unprotecting them), though that's a somewhat separate issue.

Consider this issue a forum to discuss potential attacks against the PCID algorithm.

The routing bits are secret, which is what protects the server mapping. To crack this, the attacker's first task is to discover the routing bits.

The two possible ways to obtain attack information are to (1) passively observe all (or a portion of) all CIDs coming out of a load balancer or (2) Open a connection and get many CIDs from one server, ostensibly for migration purposes.

There are basically two defenses:

• Because the true server ID ("modulus") is combined with an integer multiple of the divisor, there is no readily obvious signature for a single server (e.g.. bits 4, 5, and 7 are always '1' for a given server)
• Given the minimum CID length of 8 bytes, there are roughly 10^18 combinations of routing bits. This is a challenge to attackers trying to create statistics about each possible set of routing bits.

Kazuho expressed a desire to have the following algorithm:

1. The LB requires at least 16B of CID; the server can optionally extend to 18
2. The server encodes its Server ID in the needed number of bits, plus whatever it wants for the remaining bytes (thus often making the 2 additional bytes unnecessary)
3. Those 16B are encrypted using AES-ECB. If you use the free fields the same way, you'll get the same CID, so don't do that.

This is clearly secure, IMO. The only issue is it drives QUIC to 16B+ CIDs, which is kind of unfortunate.

We can absolutely add this algorithm to the draft or make this the ECID algorithm. I hope to understand better, however, the limitations of the current ECID goes away.

Is there support for me to write this up as a third algorithm, before we decide on the existing ECID?

Martin Thomson's new first byte proposal would eliminate any extra QUIC long header codepoints that we are currently using for QUIC-LB.

This is not yet final, but are there other mechanisms we could use? One would be to use the RETRY type: if a server receives one, it must be QUIC_LB. A load balancer would probably be able to disambiguate this as well. But it is kind of gross.

(Copied from the original report in Google groups.)

Section 7.2 states:

The QUIC-LB load balancers send the encoding parameters to servers and periodically retransmit until that server responds with an acknowledgement. Specifics of this retransmission are implementation-dependent.

IIUC, this means that the following scenario can occur:

    LB: send M1
                            Server: receive M1
                            Server: send ACK1 (gets routed to loopy land)
    LB: send M1 again
                            Server: receive M1
                            Server: send ACK2
    LB: receive ACK2
    LB: send M2 (gets lost)
    LB: receive ACK1 (after some odd wandering it's back again)
    LB: stop retransmitting M2

This can probably be solved by adding a simple epoch-like counter to the packets.

The draft states:

When a server needs a new connection ID, it adds an arbitrary integer multiple of the divisor to its modulus, without exceeding the maximum integer value implied by the number of routing bits.

This almost makes it sound like one could run out of connection IDs. I think this number can wrap. More generally, the server is free to choose any valid value, not necessarily via addition.

Subodh wanted a failover mode if QUIC-LB configuration was somehow unavailable. This is a special type of CID that instructs the load balancer to just use the 4-tuple. As we have 4 cod e points for config rotation, we thought we could use one of them to represent this mode.

With no protocol, there is no real need for an interop. Instead, an appendix with test vectors should be sufficient to make sure that the specification is unambiguously implementable.

In Tokyo, people expressed a desire for a literal, plaintext encoding of the SID. The idea is that people perceive "obfuscation" as no better than "unprotected", and would like to make that explicit.

I don't see value in this over PCID, but several team members said I should let the WG decide.

This is not encrypted, so we should discuss the analysis.

Upon further reflection, it may be better for Retry Services to be in a different draft. The main reason is that CID encoding deals with the CID, which is an invariant part of QUIC.

Meanwhile, Retry services are most definitely version-dependent. It would be annoying to have update all of this stuff based on QUIC version revs that changed the Retry semantics.

On the other hand, it would be a bummer to have two separate chunks of configuration, particularly if it was happening in-band.

Thoughts on this? I'm inclined to separate out the drafts before Singapore.

For as many algorithms as possible. @huitema points out that figuring out how to read a CID is pretty gnarly to follow in the text.

Via @dtikhonov: Some load balancers might also serve as NATs, in which case they have some per-connection state.

If this state is per-fourtuple, all the QUIC address change mechanisms will work fine. We should add some text to the security considerations that doing this on a connection ID basis (trying to be QUIC-aware) is actually worse -- it can lead to masking address changes from the server and prevent employment of path verification mechanisms.

There are now three distinct functions QUIC-LB is supporting. A more structured presentation of the configuration schema, like Section 3 of RFC 8446, would make things clearer.

Currently, QUIC-LB packets use a Token to prove authenticity (in both directions). This provides protection from off path attackers (in the backend network) but not man-on-the-side or man-in-the-middle attackers. Both of those attackers can see the Token and then create their own QUIC-LB packets to reconfigure back end servers.

If encryption is used instead, then the packets cannot be tampered with or spoofed. The best attacks that man-on-the-side attackers can do is try to interrupt each individual connection (a lot of work compared to sending 1 QUIC-LB packet to misconfigure a backend server). Man-in-the-middle can still drop any packets they want, but nothing can be done to protect against that.

More over, I think we should explicitly default the key used to a null (all zeros) key. In deployments where the entire backend is completely controlled, and there is no fear of attack, there would be no need to configure the encryption key out of band, and the default would be used.

We need to include the config rotation bits in the list of config parameters. If there are no quic-lb packets, servers still need to know how to set these bits.

The current Retry Services spec mandates that retry services MUST forward Initials with unknown versions to prevent ossification. However, this has its own issues:

1. Although dealing with random-version packets is extremely cheap, it would certainly be nice to drop these earlier. It might be tractable to do something like "Retry services MUST NOT drop packets with unknown versions at a rate higher than it is sending Retry packets in response to Initials with known versions". i.e. you can only drop unknown versions under load.

2. There is the possibility that a retry service could send a Retry packet for a version that the back-end server does not understand. Thus the pattern seen at the client would be a Retry followed by a Version Negotiation. VN validation is an open issue but this is an unusual pattern that is technically legal (I think?) but certainly unexpected. @nibanks suggested we extend the protocol to negotiated supported versions, although if servers are heterogenous it won't work very well.

I am not entirely convinced that these are genuine problems, and am even more skeptical that there are good solutions consistent with the objectives of QUIC. But this issue is a forum to discuss this. If anyone has a solution more creative than "drop all packets with unknown versions", a PR would be helpful.