In practice most projects seem to use Yjs rather than Automerge. Is there an up-to-date comparison of the two? Has anyone here chosen Automerge over Yjs?
I’m quite familiar with both, having spent some time building a crdt library of my own. The authors of both projects are lovely humans. There are quite a lot of small differences that might matter to some people:
- Yjs is mostly made by a single author (Kevin Jahns). It does not store the full document history, but it does support arbitrarily many checkpoints which you can rewind a document to. Yjs is written in JavaScript. There’s a rust rewrite (Yrs) but it’s significantly slower than the JavaScript version for some reason. (5-10x slower last I checked).
- Automerge was started by Martin Kleppmann, Cambridge professor and author of Designing Data Intensive Applications. They have some funding now and as I understand it there are people working on it full time. To me it feels a bit more researchy - for example the team has been working on Byzantine fault tolerance features, rich text and other interesting but novel stuff. These days it’s written in rust, with wasm builds for the web. Automerge stores the entire history of a document, so unlike Yjs, deleted items are stored forever - with the costs and benefits that brings. Automerge is also significantly slower and less memory efficient than Yjs for large text documents. (It takes ~10 seconds & 200mb of ram to load a 100 page document in my tests.) I’m assured the team is working on optimisations; which is good because I would very much like to see more attention in that area.
They’re both good projects, but honestly both could use a lot of love. I’d love to have a “SQLite of local first software”. I think we’re close, but not quite there yet.
(There are some much faster test based CRDTs around if that’s your jam. Aside from my own work, Cola is also a very impressive and clean - and orders of magnitude faster than Yjs and automerge.)
>> In technical terms: cr-sqlite adds multi-master replication and partition tolerance to SQLite via conflict free replicated data types (CRDTs) and/or causally ordered event logs
I helped coauthor some of the early drafts of Braid. Braid isn’t an attempt to make a local first, crdt based eventually consistent data store. It’s just a protocol.
Braid aims to make it easy for such systems, as they’re built, to be able to talk to each other.
Oh amazing - it looks like the GP commenter is right. Yrs is significantly faster now than it was when I benchmarked it a few months ago. I'd update my comment above, but its too late.
For example, in one of my tests I'm seeing these times:
Yjs: 74ms
Yrs: 9.5ms
That's exceptionally fast.
This speedup seems to be consistent throughout my testing data. For comparison, automerge takes 1100ms to load the same editing history from disk, using its own file format. I'd really love to see automerge be competitive here.
(The test is loading / merging a saved text editing trace from a yjs file, recorded from a single user typing about 100 pages of text).
Sooo we’re building “SQLite for local-first development”, it’s here! Uses CRDTs, can be a partially replicated db, peer to peer networking and discovery.
Bruinen.co
Shoot me a note if you want an early build! Or if interested in building with us :)
Making the app work without an internet connection is step one. Making it reparable without an internet connection is step two.
Step two is blocked if you can't keep the code for all of the app's dependencies near enough at hand such that its still accessible after the network partitions.
I use yjs myself and you can choose to serialise anything you like, most solutions allow saving snapshots of the document state. You can also store any incoming changes too for more fine grained undos and redos etc. AFAIK the state in typical solutions is a binary representation.
Many projects use Yjs for its collaborative rich-text editing (e.g. Linear: https://x.com/artman/status/1733419888654291102). Yjs makes this easy by providing "bindings" to various rich-text editor GUIs, which sync between the editor's internal data structures and Yjs - something that involves a lot of detail work. Automerge's rich-text support is more recent (~last year), and so far they only have one editor binding (ProseMirror), so Yjs is naturally more popular here.
For non-text collaboration, there is a more crowded "market", because it is an easier problem to solve - at least when your app has a central server. Tools range from hosted platforms like Firebase RTDB to DIY solutions like Figma's (https://www.figma.com/blog/how-figmas-multiplayer-technology...). Meanwhile, Automerge's target niche is decentralized collaborative apps, which are rarer.
It was by far the most developer-friendly experience I've had trying to implement collaborative editing. The one thing it didn't have that YJS did was built-in undo/redo.
Interesting, hadn't heard of PartyKit. FWIW Reflect seems to be pivoting sometime over the next 6 months, and they're going to open source their code, along with instructions on how to self-host. So if you're looking for long-term cost reduction, that might not be a bad choice.
yJS has the webrtc adapter and appears to still win out in the edit benchmarks. I’ve used yJS in two projects: once just for presence chat and syncing menus for a coaching site and once in a overlay graphics app for a livestream
Biggest problem with yJS for me has been the ergonomics when I use it with React. There’s a third party project called synced store that I used for the stream overlay but it has some strange behavior.
With first party support with React in automerge I think it’s worth a shot for my rewrite
The big yjs problem for me is the documentation. A few of the most important sections are just “todo” placeholders, like the “how to write a provider” section.
It’s great at first, but woefully underdocumented if you want to use it in a way that doesn’t have off the shelf support, and the code is tough to parse (for me at least). Same with subdocuments.
I wanted to use it with Lexical a while back, and yjs plugin was too tightly coupled too a single data model and was too complicated to DIY.
In my case, my problem was wanting to sync key A’s data to/from Alice, and key B’s to/from both Alice and Bob. It was unclear to me where and when to apply which updates involved in the yjs protocols (which i couldn’t find any documentation on).
I am a big fan of local-first design principles. For most enthusiasts, it's the performance benefits. However, for me, it's the potential it has for privacy in web clients.
If you have a web client that only does on-client data storage, you're not dependent on centralized server-side storage for persistence (which is the main privacy-loss hazard).
The problem is that the app host still has all the technological freedom to not honor their privacy agreement, and there don't seem to be backstops to that behavior in browsers.
Uses OPFS + WASM to run SQLite and host files. I've had a pretty healthy response to it from HN so I plan to make a desktop app + add PeerJS for file sharing.
I love CRDTs, but it really seems like the type of thing where the idea of them is the real impact (many systems are built with structures that enable CRDT-like operations without actually using CRDTs), and the implementations never quite hit product market fit.
I can't picture how that would work. While collaborative features require that some objects are shared and synchronized, efficiency and programmer sanity rely on the fact that some objects are not. If synchronization is opt-in, how would a language integrate it any more fundamentally than a library can?
Also, CRDT's don't provide synchronization for free. They ensure that all concurrent modifications will be merged somehow. If the data being synchronized has any structure, it requires careful CRDT-aware data model design to ensure the merging is semantically reasonable (or that, in the worst case, incompatible changes produce a detectably broken state).
Think about things like browser profiles or password managers that are kind of predicated on multi device and how none of that stuff is what you’d call zero setup. Does every app that wants to do this stuff need a key base or drop box backend? Is apps joining an ad hoc k8s network mesh or relying on a preexisting central redis or postgres or something really necessary for something like sharing data structures?
There’s definitely some room for interesting work here and language level support could be cool.
Elixir interpreter clustering and otp is maybe the closest existing thing, which is awesome but only for existing erlang fans.
It'd be great, but I think that we are not ready for this yet. Think on how long we needed to make the idea of having good tooling, cross-compiling support, testing or debugging support a requirement.
A library or even built-in language support for distributed data structures will take a decade or two to get to the point of proving a set of features to be truly helpful and good/necessary to have as a library or maybe even as a built-in feature.
BTW, we don't even have quickcheck yet, since the generation and proof reduction quality isn't great across implementations :/
Why a language and not a framework? I mean some langs even implement half their functionality in std libraries rather than part of the lang (as a preference actually).
I see their being overhead and what not that would make it very domain specific and less appealing for anything that doesn't need collaboration.
Why would you even want to start an entire new language with no tools, no debugging, no libraries, no ides and no optimizations just so you could make some data structures that could be made with C++ classes?
Construction industry is a good one: often you are working in a remote area or have cut power (so no WiFi) and having access to your data is key. That’s why there is so much printing still going on- paper works offline.
security is a big reason to go local first in general (not necessarily with crdts but they are a strong option), less data over the wire to central servers, you can avoid even needing to know about user data if it stays on their device(s)
collaboration software benefits a lot from this approach because people can just use it and know it won’t get out of sync
Also, anything where you might need data offline at a specific moment. Like if the internet goes down but you still need the thing to show to the guy at the place, you’ll be glad if they used crdts because you’ll automatically have a copy
Also, cheapskates who don’t want to pay cloud bills
but will you ever be able to trust the automatic merge?
Imagine a complex document with a few collaborators which work in parallel offline and make significant changes (that is the use case of automerge?).
You will have to proofread the entire document after every non trivial automatic merge, or how good does this work in practice? Would it be easier to just wait for a wifi connection and do the changes in real time?
now you can compare it to pull requests in software development, but in software development we have compilers, linters and tests which point out merge bugs.
Not quite true, you can utilize peer to peer (p2p) storage options, where only the individuals that are allowed to edit and view the document/media/file do storage. Really you only need one peer that has an IP address to do NAT traversals, that peer need not do any storage, but just passes transactions along to the end users.
With IPV6 and Torr onion routing NAT traversals is almost becoming a non issue.
