Add Session Pro as a concept to libsession

[Doy-lee]

Proofs that are signed by the Session Pro backend are now present in the library as a concept. It's a class that contains the fields that the Session Pro Backend will produce that is encapsulated in ProProof. Proofs can be verified and validated and embedded into messages. This PR goes ontop of: #61

The bulk of this code is fulfilling the requirement

Clients need a way to append Pro proofs and features to messages which use Pro features.

Libsession should have a function which takes

Message plaintext
Message type
Destination
Pro Features used

Clients will call this function when they want to send any message, and libSession will return the encrypted message (with protobuf wrappers) which the client can then use their networking code or libSession to send to the destination.

When a client specifies to libsession that a message will be using Pro features the resulting encrypted message libsession produces will contain the Pro proof and a bitmask for Pro features used in the message.

This main functionality is in a new file session_protocol.cpp whose intent is to be high level wrappers over the lower level cryptography in session_encrypt.cpp. The main work horse functions are

PRO_FEATURES get_pro_features_for_msg(size_t msg_size, PRO_FEATURES flags);

EncryptedForDestination encrypt_for_destination(
        std::span<const uint8_t> plaintext,
        std::span<const uint8_t> ed25519_privkey,
        const Destination& dest,
        config::Namespace space);

DecryptedEnvelope decrypt_envelope(
        const DecryptEnvelopeKey& keys,
        std::span<const uint8_t> envelope_payload,
        std::chrono::sys_seconds unix_ts,
        const array_uc32& pro_backend_pubkey);

Encrypting and decrypt the envelope was implemented by referencing the following implementations across the platforms:

iOS:

• https://github.com/session-foundation/session-ios/blob/82deef869d0f7389b799295817f42ad14f8a1316/SessionMessagingKit/Sending%20%26%20Receiving/MessageSender.swift#L422-L500
• https://github.com/session-foundation/session-ios/blob/82deef869d0f7389b799295817f42ad14f8a1316/SessionMessagingKit/Sending%20%26%20Receiving/MessageReceiver.swift#L91-L174

Desktop: https://github.com/session-foundation/session-desktop/blob/4f51c28cc0ba999df0b46eb09d83bdfa9b372ab5/ts/session/sending/MessageQueue.ts

Android: https://github.com/session-foundation/session-android/blob/e4c16ae8aa8cae2f75aba565b890082563b07bd2/app/src/main/java/org/session/libsession/messaging/sending_receiving/MessageSender.kt#L398

Updating protobufs to include the new pro metadata, the main outline of those changes:

message Envelope {
+   optional bytes  proSig = 11;
}

message Content {
+   optional ProMessage proMessage = 16;
}

+ message ProProof {
+   optional uint32 version           = 1;
+   optional bytes  genIndexHash      = 2;
+   optional bytes  rotatingPublicKey = 3;
+   optional uint64 expiryUnixTs      = 4;
+   optional bytes  sig               = 5;
+ }
+ 
+ message ProMessage {
+   optional ProProof proof    = 1;
+   optional uint64   features = 2;
+ }

--

The unit tests have examples of how to use the API. The idea here is that the platforms will fill in the pro metadata if they have some into the protobufs. You can then hand over the serialised protobuf plaintext for encryption by configuring the Destination as to where the message is intended to go (1o1, legacy groups or groups v2 essentially) with the necessary keys and call encrypt_for_destination.

You then call decrypt_envelope when you receive a payload from the wire to get the plaintext content as well as the pro metadata that was available in that message.

Libsession will do the encryption and return the final payload suitable for sending on the wire. The reverse is available to decrypt an envelope. It supports decrypting both an envelope that is encrypted (e.g.: A groups v2 envelope) and an unencrypted envelope (e.g.: 1o1 and legacy groups envelope) where the content is instead encrypted with the recipient keys.

Still todo:

• Cleaning up the JSON payload generating functions for talking to the Session Pro backend
• A way to sync the revocation list from the Session Pro backend (potentially BYO networking, BYO storage)
• Having a way to transition open groups and open group inbox messages to being able to bundle the pro metadata (they currently send the protobuf Content on the wire directly) to enable pro features in those message in a way that is forwards and backwards compatible.
• All the additional helper functions required in the Session Pro PRD (e.g. separating out PRO verification functions so you can call it standalone and evaluate current device PRO status, e.t.c)

[mpretty-cyro]

We don't actually support sending messages to legacy closed groups anymore on any of the clients (iOS has removed most of the code to handle receiving these messages and I think Android will be doing the same soon) so you could potentially remove EnvelopeType::ClosedGroupMessage since it was only used for legacy groups

[mpretty-cyro]

Actually it looks like we are still setting this for updated groups 🤔

Confirmed with the Android team - unfortunately they are still using it at the moment so we will need to keep it to avoid breaking old Android versions 🥲

[mpretty-cyro]

I think Android is the only one doing this now - just wondering whether we should properly deprecate this value and rely on the namespace that the message was received from (if a message with the wrong encryption type is stored in the wrong namespace then it should be considered invalid and dropped anyway)

[Doy-lee]

For sending we still need to as above for android but for decrypting on receive, you're saying remove this branch and replace it with:

if (space == config::Namespace::GroupMessages) {
  result.envelop.type = EnvelopeType::GroupMessage;
} else {
  result.envelope.type = EnvelopeType::SessionMessage;
}

Is that right, where the C++ side still wants to keep the envelop type so application code after decryption has all necessary information self-contained in struct Envelope to work with, without requiring any external context. But requiring the type on the protobuf Envelope is no longer needed (and an outdated practice).

[mpretty-cyro]

Yea that was my thinking - since the decryption code is based on the namespace then we could eventually stop sending the type entirely once enough time has passed

Is that right, where the C++ side still wants to keep the envelop type so application code after decryption has all necessary information self-contained in struct Envelope to work with, without requiring any external context.

I get where this is coming from but since the Envelope is encrypted for messages sent to a group the decryption logic is already dependent on knowing the namespace that a message came from which makes having a Envelope.type feel redundant 😕 (ie. at this point we already know that it's a GroupMessage)

[Doy-lee]

Alright. understood. I have removed the parsing of envelope types on decrypt.

[Bilb]

again, legacy groups have been readonly for a while. Sending or receiving a message to/from it should never be done anymore.

[Bilb]

maybe we should make this timestamp_ms, just in case 👀

[Doy-lee]

I have the suffix for the C API but C++'s type safety here makes it pretty hard to misuse the timestamp here in a context that isn't compatible with milliseconds.

[Bilb]

would it make sense to also have pro_features unset if the pro_proof is not Valid ?
It might be the case already, but if yes it would be good to have the comment updated

[Doy-lee]

Better to preserve the information so that you have the option of using diagnostics on the pro features that were requested but don't have a valid proof anymore. Similarly I don't wipe the pro proof data as I envision basically all client code should be branching on the status before applying pro features.

if (pro_status == ProStatus::Valid) {
  // Do sensitive operations w/ pro features,
} else {
  if (pro_features) { /*maybe diagnostics on failure*/ }
}

[Bilb]

I am very confused by this. Is it supposed to map to real prefixes?
If yes this should all be changed a fair bit:

Suggested change

	enum class SessionIDPrefix {
	standard = 0,
	group = 0x3,
	community_blinded_legacy = 0x5,
	community_blinded = 0x15,
	version_blinded = 0x25,
	unblinded = 0x7,
	};
	enum class SessionIDPrefix {
	standard = 0x05,
	group = 0x03,
	community_unblinded = 0x05,
	community_blinded_legacy = 0x15,
	community_blinded = 0x25,
	version_blinded = 0x07,
	};

I guess 0x3 is actually 0x03, but the prefixes/names associated are not correct too

[mpretty-cyro]

The community_unblinded uses a 00 prefix which represents the users ed25519 pubkey (and a signature signed with that) whereas the 05 prefix would be for the users x25519 pubkey (and it's signature)

It's used for the SOGS signature header when blinding is disabled - for reference there is a function to get the string prefix for this enum in internal.hpp:

inline constexpr std::string_view to_string(session::SessionIDPrefix prefix) {
    switch (prefix) {
        case session::SessionIDPrefix::unblinded: return "00"sv;
        case session::SessionIDPrefix::group: return "03"sv;
        case session::SessionIDPrefix::standard: return "05"sv;
        case session::SessionIDPrefix::community_blinded_legacy: return "15"sv;
        case session::SessionIDPrefix::community_blinded: return "25"sv;
        case session::SessionIDPrefix::version_blinded: return "07"sv;
    }

    return "05"sv;  // Fallback to standard, shouldn't occur
};

[Bilb]

to me, this should throw

[Doy-lee]

I need confirmation that this branch can't get hit: https://github.com/session-foundation/session-ios/blob/82deef869d0f7389b799295817f42ad14f8a1316/SessionMessagingKit/Sending%20%26%20Receiving/MessageSender.swift#L425

[mpretty-cyro]

The branch you've linked is the encryption for updated groups, legacy groups would fall into this case: https://github.com/session-foundation/session-ios/blob/82deef869d0f7389b799295817f42ad14f8a1316/SessionMessagingKit/Sending%20%26%20Receiving/MessageSender.swift#L449

The code can be hit by a modified client at the moment, but most of the legacy groups logic has been removed so while a VisibleMessage would probably work, anything needed to update a legacy group state would fail - on a non-modified client legacy groups are in a read-only state so there shouldn't be a way to send a message (ie. throwing in this case should be fine)

[Doy-lee]

Alright, updated, throws if you try and hit that codepath.

[Bilb]

I do not think this is anything used by any platforms, but to verify.

[Bilb]

We can't do this, yet. This would throw for any messages already stored in the swarm, and until everyone has upgraded their app, right?

[Doy-lee]

This is correct, I have fixed it by making it optional now. Added a note that we can enforce it in a year, but for now we can't otherwise it'd break all the old clients as you have mentioned.

[jagerman]

Appears unused (aside from a static assert, which could simply be changed to sizeof(array_uc32) == ... or array_uc32{}.size() == ...)

[Doy-lee]

Updated the static assert and I've added a comment as to what its purpose is. It's meant to be set to the actual Session Pro backend key so that eventually users of the library have the key handy in the interface so as to when they go and verify the proofs they can just pull it from the header.

For libsession I've made the decrypt and verify function parameterise the Session Pro backend public key that it accepts so that you can use it against any arbitrary key (so that it's useful for tests) instead of baking in the key from the header. But the idea is that you can use the key in this header officially for that purpose.

[SessionHero01]

Should we leave the expired state out of this status? Whether a pro has expired depends on current time, which I find it the best to leave it to the caller to decide - so that:

1. They can supply their own clock
2. They know when to recheck for validity.

For this reason I suggest we do these states instead:

Suggested change

	enum class ProStatus {
	// Proof not set
	Nil = PRO_STATUS_NIL,
	// Proof set; pro proof sig was not produced by the Pro backend key
	InvalidProBackendSig = PRO_STATUS_INVALID_PRO_BACKEND_SIG,
	// Proof set; envelope pro sig was not produced by the Rotating key
	InvalidUserSig = PRO_STATUS_INVALID_USER_SIG,
	// Proof set, is verified; has not expired
	Valid = PRO_STATUS_VALID,
	// Proof set, is verified; has expired
	Expired = PRO_STATUS_EXPIRED,
	};
	enum class ProStatus {
	// Proof not set
	Nil = PRO_STATUS_NIL,
	// Proof set; pro proof sig was not produced by the Pro backend key
	InvalidProBackendSig = PRO_STATUS_INVALID_PRO_BACKEND_SIG,
	// Proof set; envelope pro sig was not produced by the Rotating key
	InvalidUserSig = PRO_STATUS_INVALID_USER_SIG,
	// Proof set, is verified; whether it's expired you should check the expiration time in pro_proof
	Verified = PRO_STATUS_VERIFIED,
	};

Thoughts?

[Doy-lee]

Yeah this is currently the case that the caller is in control of the clock.

/// - `unix_ts` -- pass in the current system time in seconds which is used to determine, whether or
///   not the Session Pro proof has expired or not if it is in the payload. Ignored if there's no
///   proof in the message.

DecryptedEnvelope decrypt_envelope(
        const DecryptEnvelopeKey& keys,
        std::span<const uint8_t> envelope_payload,
        std::chrono::sys_seconds unix_ts,
        const array_uc32& pro_backend_pubkey);

Pro verification in regards to extracting a pro status from the proof is currently embedded in the decryption function but I intend to pull that out to some utility functions some point later so you can call it standalone. I plan to add those after if we're generally happy with the approach here before writing all the helper utilities that go ontop.

Is that sufficient for your use case? Or would you still want to separate the expired-ness from PRO_STATUS.

[SessionHero01]

The use case is when decrypt_envelop is called, the user hasn't expired yet, let's say 10s later it expires, we won't be calling the decrypt_envelop to determine the validity again, we (the application code) will have to arrange its own timer to check the expire timestamp - hence it's best not for the libsession to say whether the pro has expired as it will be tempting to think that as long as the result you get is valid, it's valid forever. Just pushing the expiration check's responsibility to the user forces them to handle it correctly.

This is more about making the API hard to misuse, as I believe at the current form, all the data the application needs is right there. In terms of extracting a pro status from the proof - I think I have some other ideas on the DecryptedEnvelop so let's see how you think of that suggestion instead.

[SessionHero01]

After a thought I think it doesn't make sense to not have libsession validating the expired status, "expiration" after all, is a criteria of validity. So forget about this one

[Doy-lee]

Sure, yes I think so we can move expiry out and not merge it into the status field.

[Doy-lee]

This has been handled out-of-band, we've decided to keep expired in the status, but, reaffirmed that it's useful to have that separated as the clients envision having to check expiry themselves as well independently from decryption at later points.

[SessionHero01]

Should we have a union type for these instead? So you won't have to worry "only set if status was not Nil":

struct Nil {};
struct InvalidSig {
  PRO_FEATURES pro_features_requested;
};

struct InvalidBackendSig : InvalidSig {};
struct InvalidUserSig: InvalidSig {};
struct Valid {
  config::ProProof pro_proof;
  PRO_FEATURES pro_features;
};

typedef std::variant<Nil, InvalidBackendSig, InvalidUserSig, Valid> ProStatus;

This ProStatus will then be the only property you need on the DecryptedEnvelop, and you won't be able to get to any invalid state.

[Doy-lee]

Handled out-of-band, the solution is to embed the PRO_STATUS into a std::optional<ProProofWrapper>-esque structure so that the presence of the optional determines if pro is nil or non-nil.

User code then just needs to branch on the optional to know that there's some, maybe invalid, pro data to work with

std::optional<ProProofWrapper> pro_proof_wrapper;
if (pro_proof_wrapper) {
  assert(pro_proof_wrapper->pro_status != NIL); // Cannot be nil by definition because the optional is present. So nil state will be removed from the set of possibilities
}

So that removes ProStatus::Nil from the enum and solves the having to worry about a nil status. That is implicit in the presence of the optional