[API Proposal]: Add support for replay detection and out of sequence detection to NegotiateAuthentication

[mconnew]

Background and motivation

WCF has bindings in .NET Framework which can use SPNEGO to initiate an authentication handshake and establish a secure session between client and server. It does this by directly capping the SSPI api's via P/Invoke. To bring this capability to .NET and make it cross platform, we would adopt the usage of NegotiateAuthentication. Some of the features of SSPI that WCF uses are not exposed by NegotiateAuthentication today.
WCF is able to do authentication in the open (using full Message security, not yet supported on .NET), not protected by TLS/SSL, it's important to prevent attacks that can be done with communication in the open.
SSPI has the ability to detect when payloads are being replayed when signing and/or encrypting the payload. This requires passing the request context flag ISC_REQ_REPLAY_DETECT. A second flag that WCF uses, which also helps against in the open attacks is ISC_REQ_SEQUENCE_DETECT. This detects signed/encrypted payloads from being played out of order.
The behavior of NegotiateAuthentication is to not set either flag.

API Proposal

namespace System.Collections.Generic;

public class NegotiateAuthenticationClientOptions
{
    public bool DetectReplay { get; set; }
    public bool SequenceDetection { get; set; }
}

public class NegotiateAuthenticationServerOptions
{
    public bool DetectReplay { get; set; }
    public bool SequenceDetection { get; set; }
}

API Usage

var options = new NegotiateAuthenticationClientOptions {
    Package = Negotiate,
    TargetName = "host/myserver",
    RequiredProtectionLevel = ProtectionLevel.EncryptAndSign,
    DetectReplay = true,
    SequenceDetection = true
};
var nego = new NegotiateAuthentication(options);

Similar code would be used for the ServerOptions.

Alternative Designs

None that I can think of

Risks

No risks as default behavior would be identical to today. It would only affect those wanting to use the feature.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[wfurt]

Is there any case when we do NOT want them @mconnew. e.g. I would assume that would be always on unless it break some useful scenario. cc: @filipnavara for any more thought. I'm not sure if there is only Unix equivalent.

[mconnew]

I would expect it to use more memory and CPU. It needs to keep extra bookkeeping info to keep track of previous messages for the replay attack. The out of order checks can be done cheaper. WCF's usage is to use the secure connection to then transmit a symmetric session key which is then used for encrypting further communication so we don't end up with a large amount of per-session data in memory.
It would also only provide additional security to the Wrap/Unwrap calls, which Http authentication doesn't use. If you used the proposed flags with NegotiateStream, I would expect you would end up with quite a lot of overhead as you can send gigabytes of data in a short amount of time. Detecting replays on that would consume a lot of memory.

As I'm focusing on the WCF client, I didn't consider the service side. We would also need the corresponding flags added to NegotiateAuthenticationServerOptions. I'll update the title and top level comment with that.