-
-
Notifications
You must be signed in to change notification settings - Fork 872
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Co-hosting communities across instances (e.g. "sharding") #3100
Comments
Why not just subscribe to /c/games on multiple instances? Having a single point of truth threatens to remove one of the joys of federation. |
I can edit my post to clarify that combining communities was just one possible side effect. The main issue is am addressing is the vulnerability of communities at the mercy of an instance going offline. My point with that bullet item was that people should be making duplicate communities because they are interested, not out of fear that instances might drop at any moment. |
I think a blockchain type model that is reminiscent of the way bitcoin and other cryptocurrencies are structured may be applicable here. A shared source of truth that all instances can access that are synced between them, but remains completely independent of/between all of them. Sharding across the chain would probably be more complex than the current way that blockchain is implemented via bitcoin since other federated servers can willfully drop the instance from federating and appearing in their feed, but as a base model I think it may make sense. Thoughts? |
FEP-2100 has a proposal along these lines. Apparently it is implemented in a GNU social plugin (the link is not working for me). However it would take a lot of work to make this production ready for Lemmy. |
If this was something where communities opt-into, you could organize the content storage with a DHT. I do not really know how lemmy looks like under the hood, but this could in theory at least allow to leverage the resources of multiple instances. Of course, nodes could in addition always replicate the stored items for redundancy, but they wouldn't have to serve it. Whether this would be worth it depends at the end on how much the bandwidth / processing posts and comments make up of the total bandwidth / processing and if the overhead of this would be worth it, because every node still needs a complete index. So maybe this could be interesting for instances and communities that want to host pictures themselves? |
My understanding is this already happens when an instance subscribes to a remote community. If that's true, it seems we already have a good foundation for next steps. |
I think that FEP is exactly what we want. I think it would relieve a lot of user's concerns about instances shutting down. |
I've started using Lemmy recently, and I've noticed that some communities in popular instances were created a while ago, but a number of them eventually grew enough and became whole instances. Or changed from a "general" instance to a more "themed" one (like for technology, or for a specific country). The current model makes a change in instance to be a very difficult process, and that isn't even mentioning the actual issue of instances shutting down. Co-hosting or syncing would fix a bunch of those small issues. |
oh hey we were just talking about this https://programming.dev/comment/11585424 would definitely be a cool feature! |
FEP-ef61 also covers nomadic identity. Based on this description it is already implemented in streams, so it should also work for Lemmy if someone has the time to work on it. |
Describe the feature request below
There is some uncertainty from Reddit users coming to Lemmy that "my community could vanish without notice!" if an instance goes offline or gets blocked from federating, with some expressed desire to be able to migrate communities across instances #3057. There is also some concern about federation scaling eventually becoming burdensome as traffic increases #3062 .
I propose a form of co-hosting of a single community across multiple instances, similar to sharding a database. I'm not even sure this is possible, but since Federation already replicates data across instances, I don't think its too big a stretch. I'm picturing one instance designated as the "primary" instance, but then other instances in cooperation can be designated as co-hosts that will step up to handle requests while the owner is down, and perhaps eventually a cohost might be promoted to the new primary with permission from the original community mods.
it seems like it might provide a few benefits:
Encourage or facilitate combining fractured communities. -- sinceEdit: this is just another possible side effect. Let's not fixate on it as it's not the main point./games
on one instance can be the same community as/games
on another, users won't feel the pressure to make duplicates "in case one goes down".The text was updated successfully, but these errors were encountered: