Increase peer participation and Hyperdrive availability

[da2x]

Is your feature request related to a problem? Please describe.
The current implementation relies on peers manually deciding to host drives they really like and remember to host. This runs into the same mental transaction/decision cost problem that has halted the adoption of micropayments. Is this worth it for me to click my mouse two times and donate bandwidth towards? Having to make the decision is too tiresome and the mental cost is too high.

People also ‘like’ a whole bunch of things they wouldn’t want to commit to hosting for ever. You enjoy videos, like reading articles, laugh of memes, and derive value from hyperdrives large and small. But you don’t necessarily like it enough to take time out of your day to help host it. Even temporarily.

Without peer participation, the concept of a P2P-distributed web can’t be realized. There will only be a network of super-peers and clients that can download from these super peers. Hashbase comes to mind here.

Describe the solution you'd like

Random host-duty assignments. A random one† in peerCount chance you’ll automatically host loaded/visited hyperdrives for 1 day. Only 1 other peer for the drive you just loaded? 100% chance of auto-hosting it. But only a 0.1% chance to auto-host if there are 1k peers. The decision is made once per browser session per drive. Every drive there is a moderate interest in will get a bunch of participating peers. Popular/high-availability drives still gain more peers but at a slower rate than less popular drives with lower availability.

†Maybe scale this with a factor based on the percentage of the drive you’ve downloaded. If you’ve downloaded 80% of the drive you get a 8.0 chance as you’re a more valuable peer than someone who’ve downloaded 2.2% of the drive (0.22 chance).

Users can still opt to host things forever manually, but Beaker itself would call upon users to participate in maintaining the availability of hyperdrives and the network.

Describe alternatives you've considered

• Keep hosting drives for 60 minutes after you close the tab. Benefits popular drives more because the peer is more likely to be called on to host for others in that short time span.
• Altruistic hosting mode. Automatically host every archive you visit for up to 14 days or until you’ve uploaded the same amount of data you’ve downloaded. Did you download 1 MB then stop hosting once you reach 1 MB upload. A megabyte up for a megabyte down. (Maybe a slider for configuring the level of altruism to 1x, 2x, 3x, 5x, 10x?) Enabled by default at 1x. This is inspired by — but really a hybrid of — altruistic mode and seed-ratios/timeouts in BitTorrent clients. Adds accounting complexity.
• Rolling cache approach. Host all drives until you reach a disk cache limit then push out the drives with the most number of peers. Also discard 30 days after visit. One large video drive could consume the entire cache, though. High cache invalidation complexity.

[akavel]

One aspect that sounds risky to me in case of "auto-sharing" is the legal one. E.g. in my country AFAIU downloading some stuff is legal while sharing it is not (though IANAL). So the current explicitness as to what I decide to host/share or not sounds kinda an important feature to me.

edit: How about marketing the co-hosting feature in a kinda "archive" direction? That would sound psychologically appealing & tempting to my inner hoarder... though I'm not sure how to keep the explicit sharing/hosting aspect still clearly conveyed in case of such a rename.

[da2x]

One aspect that sounds risky to me in case of "auto-sharing" is the legal one. E.g. in my country AFAIU downloading some stuff is legal while sharing it is not (though IANAL).

<disclaimer type="not-legal-advice">
You’re thinking of pirated content and BitTorrent, right? The I-only-download-not-upload line was repeated a lot in the early 2000s. I believe its the consumption of — and not the distribution method for — pirated materials that is prohibited. Downloading pirated content has the same legal implications as uploading in the EU. There are tougher penalties for the original uploader, though. There are pan-European regulations for this. You’d need to speak to a copyright lawyer for the details.
</disclaimer>

https://www.youtube.com/watch?v=1Jwo5qc78QU

If you accidentally visit a neo-nazi hyperdrive then yes — under the above proposed scheme — you’d have a random chance to help distribute the same content as you downloaded for a limited time. The more peers the less likely it is that you’ll host it and the less likely that any peer would depend on you for the download. For less popular stuff — then that’s probably really obscure stuff and you’d be out of the swarm before anyone took notice of it.

Remember that you can’t know the contents of a hyerdrive by monitoring network traffic. Everything is encrypted. You can’t even get the cryptographic public key needed to request a copy of the data from the network by observing network traffic. (DHT uses a separate key derived from the public key. Public keys are displayed in the URL bar, yes, but its not sent over the network in that form.)

If you’re afraid of government surveillance than sticking around in the swarm for a day wouldn’t get you into any more trouble than having accessed it once. You’d not be fingered as the ring-leader of a neo-nazi group based on a one-time access to a website or hyperdrive. The same goes for a 24-hour period if the software was known to seed stuff by default.

Private browsing mode is the key to visits sites you don’t want to be associated with. Beaker should reduce interactions with a hyperdrive swarm to an absolute minimum in private browsing mode. If everyone leaches by default then Hypercore becomes a decentralized network with a couple of super-node/servers instead of a distributed one where everyone participates in the distribution of data.

So the current explicitness as to what I decide to host/share or not sounds kinda an important feature to me.

Metered bandwidth connections and battery conditions are probably the most important factors that control whether you’d want to explicitly enable or disable sharing. The user shouldn’t have to worry about or manage that manually, though. See the run conditions mentioned in #1665.

[akavel]

You're right I was loosely thinking about torrents. But your mention of avoiding neo-nazi sounds even more applicable IMO. Then there's also stuff like, I'm sorry, child porn, that I absolutely wouldn't want to touch even with a super long wooden stick. And I can well imagine a "for the lolz" internet "prankster" submitting something similarly toxic on some kinda well meaning reddit-like forum or something under a title of "fluffy cats gif". I personally honestly just don't want to share a single bit of such stuff even for 1ms. Assuming I never know who might post it where, should I be using a proposed Private Mode for 100% of the time? If that becomes popular recommendation, it's back to square one, no?

[pfrazee]

Legal liability would be complex for even lawyers to comment on, so I don't intend to speculate on it. I agree with @akavel that "pranksters" will look for ways to screw with people, which I see as a personal moral issue as much as anything else.

@da2x while I appreciate your position and agree with your motives, I don't personally believe that an even topological distribution is a key factor to this technology. Rather I think there are two important attributes when it comes to the distribution model:

1. Topological flexibility, that is the ability to transfer hosting of data between one or multiple nodes without disruption, therefore removing any lockin to a particular topology (ie a host), and
2. Cost-reduction through bandwidth-sharing, as this enables publishers (especially software publishers) to distribute their work without assuming the costs of scaling.

Our goal is mass open computation, and I think a key premise of open computation is that you control what your device does. I worry that automatically co-hosting content without user approval would run counter to that.

[almereyda]

Congratulations to and now after the launch of Beaker 1, are there any foreseeable options to have Hashbase support rehosting hyper:// sites, as suggested above? This is possibly related to beakerbrowser/hashbase#121, in case it is hyperswarm that is responsible for the hyper:// prefix. Its documentation would lack out on this detail.

The federated wiki client https://github.com/paul90/wiki-client-dat-variant is hosted on a single developer's computer, which diminishes the accessibility of that single entry point to the wiki federation on dat.

Or do we know of other headless options for rehosting of Beaker sites? Can that be achieved with hyperspace and hyperdrive seed maybe? Or is https://github.com/geut/permanent-seeder/ the way to go?

[paul90]

@almereyda - I have been using a hyperdrive daemon to act as a persistent peer for the hyperdrives associated with the federated wiki client. Maybe that is not working either reliably or in the way I had thought.

[pfrazee]

@almereyda cheers! So here's the current situation.

• There are various tools in the community like you listed which I haven't evaluated
• https://github.com/beakerbrowser/homebase is another candidate which I built
• We're putting together a "swiss army knife" cli called hyp which will include some hosting/seeding tools as well

Regarding Hashbase, we're still figuring out the exact strategy but I think we're going to leave it alone and build another self-deployable service for hyper://. We'll keep yall updated, this is one of the areas I'm actively working on.

[da2x]

Small update from my side on thr original issue. Brave has now shipped a beta release with IPFS. It hosts all content accessed over IPFS for one hour.

[almereyda]

@pfrazee Thanks, that's what I needed to know. I remember that Hashbase deviated from DatHTTPD, but weren't aware of the homebase, yet. Maybe I'll make a round through the docs and try to spot places, where this information would be most useful.