nip-17: support seen events.

[kehiy]

a simple example that i made for testing: https://github.com/kehiy/-/blob/main/research/nostr-seen/main.go

this feature is used on most of chat apps like telegram and simplex chat. it helps the client to show the seen messages with a double check so the sender can be aware of this. its not trust less since we can receive the message and don't add it to the bloom filter. same thing can be done in telegram for example using unofficial versions.

it would deliver a better user experience.

i believe the most important difference between email or similar protocols to modern real-time chat protocol which makes then feel better and more live is seen/(online/offline)/typing statuses.

[vitorpamplona]

You have to specify exacly how the bloom filter MUST be coded in the client and how its byte array should be encoded. Take a look at #1497 as an example.

[kehiy]

@vitorpamplona thanks a lot. ill take a look. you can check my example code in link above as well. but ill try to add details same as what you suggest now.

[vitorpamplona]

Yeah, the code is way too big. The encoding + choices on the filter like what is the hashing function being used should be crystal clear.

Keep in mind that each language has a bloom filter, and they all work with different hashing functions and encoders. We can't rely on them alone to spec this.

[kehiy]

yes, i realized that after checking your example. can i use some stuff from your example?

[vitorpamplona]

Sure! feel free to use anything you want.

[kehiy]

@vitorpamplona i think you can check it again. I've updated the bloom specs. 🫡

[vitorpamplona]

Major privacy issue here is that it exposes who you are talking to in the d-tag. Take a loot at the Private relationship status d-tag from here: #761

[vitorpamplona]

Also, keep in mind that NIP-17 has support for groups. The D tab must support multiple pubkeys at the same time as well.

[kehiy]

what is your suggestion for privacy issue? can we gift wrap the seen list?

is there any id for groups? so we can use group ids in "d" tag instead of multiple pubkeys...

[vitorpamplona]

#761 proposes: sha256(hkdf(private_key, salt: 'nip17') || "<pubkey>") as d-tag with hkdf defined in NIP-44.

For groups, we can just concatenate the sorted pubkeys in hex of all members, like sha256(hkdf(private_key, salt: 'nip17') || "<pubkey1> || <pubkey2> || <pubkey3>"). Logged in user can be excluded from the list.

[kehiy]

that's a great idea. for nip-29 we can do the same but with id. or a specific kind for it. but i can see people still talking about it...

[vitorpamplona]

Or maybe do a new PR with the timestamp idea, so that this one stays live as well.

[kehiy]

no this pr don't care about taking missed events for sender. the goal is to be aware of which messages have been seen by other participant.

im ok with both id and timestamp idea.

[kehiy]

i think the purposes of this pr was a little misunderstanding. i mean this feature by seen status:

[kehiy]

in this situation i know my friend seen my message.

in nostr my friend can put the timestamp of 23:52 on an event which is encrypted for me so i can know he saw any message until then and my client can show this check mark.

or just put the event id of this your welcome message there so i can know any message before this one including itself is saw by him.

[kehiy]

for groups i shows a list of people in a drop down. i don't have examples yet.

[erskingardner]

@kehiy definitely like the idea of read receipts or seen events. I agree it would improve UX. A few questions/thoughts on this:

Why use the bloom filters and a replaceable event for this instead of just sending very simple read receipt events? It seems very overly complicated for something that could be very simple.

Before coming across your PR here, I was thinking about this and my initial naïve thought was that we should use a simple event (1-9999) that can be encrypted to show that a specific user has read a specific a or e tagged event.

However, my use case is within the White Noise client which uses MLS to encrypt data and events within an outer wrapper so I don't have to worry about leaking metadata on the read receipt itself because it will be embedded in another event. It's encrypted.

This doesn't necessarily allow for clients to easily aggregate these read receipts, but I feel that since read receipts are mostly important over a short time window, it's not really necessary for clients to go way back into the past to find the read receipts and the exact time that things were read.

[vitorpamplona]

@staab made an earlier attempt to log each seen event ID in a list. The main issue is that those lists become MASSIVE very quickly. Having individual events to mark as read makes the db even bigger. So, the bloom filter here, as a replaceable event, (clients discard the past) could be a big improvement.

[vitorpamplona]

As an FYI, I will start using bloom filters for many things on Amethyst, including logging which events have been seen by the user across all screens and saving it on the user's Private Outbox relay. I might just have one big bloom filter for everything, including likes and zap events of each post.

[erskingardner]

No I don't want a list of seen events, I don't even think we should care about whether these are stored on relays for the long term. TBH, I think putting a short expiration on these types of events would be fine. Reconstituting the read receipts for a chat conversation, especially one that is private, isn't important.

[erskingardner]

As an FYI, I will start using bloom filters for many things on Amethyst, including logging which events have been seen by the user across all screens and saving it on the user's Private Outbox relay. I might just have one big bloom filter for everything, including likes and zap events of each post.

Why wouldn't you just use a local database?

[vitorpamplona]

Why wouldn't you just use a local database?

Way too slow.

[vitorpamplona]

And things like Tiktok feeds need to know which posts users have seen so that they can show only new content

[Giszmo]

You might want to look into Cockoo filters for being more compact and allowing deletion of inserted elements.

[staab]

For anyone interested in the previous attempts, see #1405

The summary is that timestamp-based misses events that get discovered out of order that you might still want to notify your user of (which would imply tracking when an event was first seen in the client so you can promote it).

The solution was enumerating ids. But that resulted in huge lists. So I tried combining it with timestamps, and keeping track of the last few ids. But that actually resulted in events that were in fact seen in other instances of the client to be marked as unread. So I went back to timestamps.

I don't think bloom filters will really solve anything, since you can get false negatives, resulting in sticky events that can't be marked as read without enumeration or a timestamp.

It's a weirdly hard problem.

[vitorpamplona]

since you can get false negatives, resulting in sticky events that can't be marked as read without enumeration or a timestamp.

Wait, a Bloom filter can only produce false positives; it cannot have false negatives. It will never incorrectly say an element is not in the set when it actually is. This means that we can have items that are marked as seen but were never seen but not the opposite.

Or am I misunderstanding something?

[kehiy]

i think the best way for read status (classifying: not in clients internal system to know what events is seen by user. for chat apps to show the seen status to other party. same as telegram example i sent) is sharing the timestamp of last event we seen from them. they consider all messages with created at <= to that timestamp seen and the rest as ubseen. what do you think?

[Giszmo]

Use a low FPS CUCKOO filter. FPR of 10E-6 with up to 3k elements would be 11kB regardless of how full it is.
To make it possible to roll over, use a counter and fill the filter to 3k events before switching to the next.
The current one goes into the replaceable event and the old one goes into non-replaceable events.

[Giszmo]

@kehiy purely time-based is not great for two reasons:

• Nostr is not consistent. Events might get published or reach the recipient with huge delay and out of order
• On a busy group chat you might jumpt down thousands of comments and want to catch up with older comments later

[staab]

Wait, a Bloom filter can only produce false positives; it cannot have false negatives.

My mistake, I always get this backwards. Bloom filters might be a good solution in that case.

[kehiy]

@kehiy purely time-based is not great for two reasons:

• Nostr is not consistent. Events might get published or reach the recipient with huge delay and out of order
• On a busy group chat you might jumpt down thousands of comments and want to catch up with older comments later

what if you use the id of last event saw by us?

@Giszmo

[kehiy]

I think in similar ways we have to support: isTyping, SelectingSticker/GIF/Image or RecordingVoice, Online, Offline, Lastseen X min ago statuses. But the difference is that these statuses are temp and seen status is permanent once it marked as seen.<

(Some apps support 3 message status: sent, received, seen)

[kehiy]

Consider that these statuses are not related to users general status like playing game, listening to music and ...

These are chat specific statuses. One per each chat.

[kehiy]

I'll open final standard after reaching a point at #2002