Networking Topology

[wittjosiah]

Problem

P2P networking is difficult, particularly in the browser where the only option is WebRTC, which isn't always reliable.

Notes

The following are my own notes which resulted from discussions we've had around both these issues:

• browsers defer networking to a native container with stronger networking capabilities (the team has been referring this model as "hub & spoke")
  • this could be a server running in the cloud, a self-hosted server or a native desktop app
  • native apps could also defer to a remote agent in order to operate that node directly or if p2p connections fail to establish (in this way p2p can be seen as an upgrade over hub & spoke connection)
• the browser is a full peer (e.g. it has an authoritative copy of the database like everyone else) but it uses an assist for networking because of browser limitations
• mobile devices (or at least iOS) have less ability to used shared storage and have a power management constraint where apps cannot run in the background to sync data, this means that when offline the scenario of sharing data between apps does not work very well because it requires another peer to be an intermediary
  • In this scenario offline interop between apps can still work but another device on the same network is required to help
• p2p networking tools to test further when stepping out of the browser
  • https://socketsupply.co/
  • https://libp2p.io/

Edit: forking discussion around vault architecture to https://github.com/orgs/dxos/discussions/4685

[jmatsushita]

How coupled are the storage partition problem and the p2p networking problem in practice ? Also have you collected a list of the webrtc problems you've ran into?

[wittjosiah]

How coupled are the storage partition problem and the p2p networking problem in practice ?

I don't think the problems are really coupled generally, they just both have implications for how & where we're running different client services. Given that, at a high level they both affect conversations around the architecture of the system, but likely the specific details of solutions to each will be not directly related.

[wittjosiah]

I've forked the discussion on the vault architecture to https://github.com/orgs/dxos/discussions/4685

[jmatsushita]

Also have you collected a list of the webrtc problems you've ran into?

Spotted this meta issue which answers a lot of my questions wrt network performance #3802

[jmatsushita]

Something to add to a list of considerations would be the webrtc transport layer of libp2p. All their examples didn't seem to make it in a recent refactor, but I've tried this a few months ago and it was working well in the lab https://github.com/libp2p/js-libp2p/tree/c6db2104dbe4257e14e66cda75bd8daf6a27f3c5/examples/libp2p-in-the-browser/webrtc/browser-to-browser

[nferch]

Do you have a perspective on how "stringently" p2p a p2p implementation must be? Given the current state of browsers and networking, we have to rely on outside services for peer discovery and NAT traversal, it looks like the libp2p needed a relay server; I'm curious whether this was just to broker the connection (similar to how our signalling works) or if it was a relay once communication was established as well.

WebTransport looks promising, with the caveat that it remains to be seen whether it'll reach the level of adoption and ubiquity we have with WebRTC, so if we were to roll our own I'd be more inclined to use a more established and boring protocol like Websockets.

Do you have any scenarios in mind where "full" browser p2p would be necessary or a differentiator?

[jmatsushita]

it looks like the libp2p needed a relay server; I'm curious whether this was just to broker the connection (similar to how our signalling works) or if it was a relay once communication was established as well.

AFAIU the relay acts as a signalling channel and try to holepunch if needed to establish a direct connection. I think that if that fails the circuit relay is able to act as a TURN relay as a fallback (apparently it's inspired by TURN).

There are some more details here:

• https://docs.libp2p.io/concepts/transports/webrtc/#how-private-to-private-connectivity-works
• https://connectivity.libp2p.io/#hole%20punching?tab=obtaining-a-direct-connection
• https://github.com/libp2p/specs/blob/master/webrtc/webrtc.md

I think WebTransport only works for client to server scenarios though, so I think for browser to browser, we're still stuck with webrtc.

Do you have any scenarios in mind where "full" browser p2p would be necessary or a differentiator?

What I'd personally like to see is "best effort" to establish direct connection with WebRTC and the various tricks in the book for hole punching, and resonable fallback options like public relays (without availability guarantees), and hub and spoke options (with dedicated agents that can be self hosted or part of a commercial offering with SLAs).

[jmatsushita]

Note that the spec says:

Behavior for transferring data on a relayed connection, in the case where the direct connection failed, is out of scope for this specification and dependent on the application.

Maybe @mxinden could help confirm whether the fall back is implemented in js-libp2p and in the example?

EDIT: Seems that asking here makes more sense libp2p/js-libp2p#2230

[jmatsushita]

This is prior art on a hub (node) and spoke (client) approach https://p2panda.org/learn/networks