Propagate pin requests between 3box pinning nodes

[msterle]

Because the pinning nodes are not peered with each other, pubsub messages sent to one don't propagate to the others. An ipfs peer bootstrapping against the exposed hostname/ip is being directed by the load balancer to one of the pinning nodes. Because of this, it will only receive a subset of pin request messages. For the purposes of an external pinning node, we'd like to receive all pinning messages. To do this:

• Forward pin request ipfs pubsub messages over redis (similarly to messageBroker implementation)
• Forward pin requests received on redis over ipfs pubsub
• Do not handle these pin requests unless the pinning node is the original recipient

The eventual goal is to use libp2p messaging instead of redis, but this will provide similar functionality in the meantime while minimizing changes to the architecture.

[oed]

Still doubt this is the best solution. Brain dump below:

Propagating pin requests won't help much if client isn't directly connected to pinning node that wants to pin data. Since each orbitdb database has a separate pubsub topic in which updates are shared, even if the pinning-node that is not directly connected to the client gets the pin request, it wont be able to pin any data since its not directly connected to the peer that has the data. So propagating the messages in this case will just be a waste of resources.

Propagating pin requests when needed. We can start using ipfs-pubsub-room, with it we can better detect when other peers come online, it also allows to send direct messages without sending to the entire room. With it we can detect when a peer comes online and always share the pin request with the new peer directly. Now this requires some reengineering on both 3box-js and 3box-pinning-node side, and potentially specing out exactly how this would function. I realize that this might be out of scope, but it seems like something closer to the correct answer.

[zachferland]

yeah but this iteration assumes that everyone still connects our pinning node and that they are redundant to it, so data is always available there. But yeah it is definitely not a general solution. But other cases when this is not true i assume for some time clients will be able directly connect to the nodes that they want to pin data with

need to think through the second part, but yeah think we will want to change how we deal with pin messages overall

[oed]

yeah but this iteration assumes that everyone still connects our pinning node and that they are redundant to it, so data is always available there.

I don't think that's true, consider the following:

1. We have pinning node A and B
2. There is a pinning node C outside of our cluster
3. C is connected to B
4. Client connects to A, pin req routed to C
5. C opens db of client, but client not connected to C
6. C is also not connected to A which opens the DB for the user
7. C won't replicate any of the data of the user

[zachferland]

yeah! thanks for the clear steps, its simple to relay orbit updates through nodes which the dbs are not open, but yeah would still not do initial handshake on new peer where existing heads are sent in 1on1 channel, dont have a follow up yet

[zachferland]

would still fix very specific short term problem, since could assume that c connects to client eventually (current imp, although not a reliable assumption), but c at least doesn't miss pin db message on client open, since relayed before, then in brief time will get heads. But yeah not wholistic and can be weighed against longer term solutions vs time for quick fix.

[oed]

Yeah, eventually they might connect. I guess the whole things relies on this anyhow.

[michaelsena]

closing, pursuing another solution

[msterle]

I am suggesting we reopen this ticket and apply it next.

---

After the discussion this week, it seems that we won't, in fact, be immediately pursuing ipfs nodes with unique peer ids, and in fact whether we want to do it at all is still an open question and a consensus unlikely to be reached soon given the current information.

For the below reasons, I suggest we prioritise this ticket over 3box/3box#943 and 3box/3box#944, which depended on this decision to reach production. Without a concrete plan on how to make ipfs HA and scalable, they would be of low value at this time, as a single ipfs node would not be deployed in production.

Reasons for pursuing this now:

1. The ipfs deployment question delaying the planned work above is difficult to resolve because both options rely heavily on assumptions, and I believe testing those assumptions is critical in guiding our decision-making, as well as helping us gain a deeper understanding of ipfs, libp2p and orbitdb. One assumption we are making is that it is relatively simple to replace ipfs/libp2p functionality with distributed equivalents. This ticket will allow us to test this assumption if we first clearly update the scope of work and acceptance criteria, as well as estimate it, and once the work is done, explore any unexpected issues that arose, and determine if more are likely (@zachferland @oed suggest we do the first part on Monday)

2. If we decide to proceed with a single ipfs id, this will have set up the first version of the pubsub forwarding layer, and would allow a relatively short path for the two tickets being deprioritised (split ipfs and pinning, with pinning workers)

3. If successful, this would allow us to show immediate value to our partners who want to run their own pinning nodes as part of the 3box network.