There are two levels on which this argument feels weak:
* The author is confusing memory safety with other kinds of safety. This is evident from the fact that they say you can write unsafe code in GC languages like python and javascript. unsafe != memory unsafe. Rust only gives you memory safety, it won't magically fix all your bugs.
* The slippery slope trick. I've seen this so often, people say because Rust has unsafe keyword it's the same as c/c++. The reason it's not is because in c/c++ you don't have any idea where to look for undefined behaviour. In Rust at least the code points you to look at the unsafe blocks. The difference is of degree which for practial purposes makes a huge difference.
A lot of what you say is explained in detail in Martin Kleppmann's article[0]. As you said, there's no guarantee about when the lock will expire. The proper solution for this is a fencing token. The idea is similar to how people have used optimistic locking when updating data in a db to avoid two users overwriting other's work.
Agree, Rust and Postgres and a perfect match. It feels so much more productive to write Postgres tooling in Rust. E.g. we already have extensions like pg_graphql[0], pg_jsonschema[1] and wrappers[2] which use pgrx. We don't have plrust on the platform yet though.
For the past few months, as part of my job at Supabase, I have been working on pg_replicate. pg_replicate lets you very easily build applications which can copy data (full table copies and cdc) from Postgres to any other data system. Around six months back I was figuring out what can be built by tailing Postgres' WAL. pg_replicate grew organically out of that effort. Many similar tools, like Debezium, exist already which do a great job, but pg_replicate is much simpler and focussed only on Postgres. Rust was used in the project because I am most comfortable with it. pg_replicate abstracts over the Postgres logical replication protocol[0] and lets you work with higher level concepts. There are three main concepts to understand pg_replicate: source, sink and pipeline.
1/ A source is a Postgres db from which data is to be copied.
2/ A sink is a data system into which data will be copied.
3/ A pipeline connects a source to a sink.
Currently pg_replicate supports BigQuery, DuckDb local file and, MotherDuck as sinks. More sinks will be added in future. To support a new data system, you just need to implement the BatchSink trait (older Sink trait will be deprecated soon).
pg_replicate is still under heavy development and is a little thin on documentation. Performance is another area which hasn't received much attention. We are releasing this to get feedback from the community and are still evaluating how (or if) we can integrate it with the Supabase platform. Comments and feedback are welcome.
I've recently been playing with the logical replication protocol, and it enables all kinds of interesting usages. one really cool thing is that you see the transactional boundaries, so not only can you write a cache, you can do so in a way thats internally consistent.
its also inherently much nicer than listen/notify, since you don't have to go back and figure out what data was associated with the event
Yes, logical replication complements very nicely the normal way of interacting with the database via queries. This inverted flow makes those apps possible which were hard/impossible to build with just queries.
This is so cool! I appreciate Debezium for its wide DB support (no part of me wants to know the inner workings of MSSQL's replication protocol) but it's finnicky to run. Great to have an alternative, at least for postgres.
I’m curious in your experience how many clients can run pg_replicate at once?
With MySQL I saw the interesting use-case of the black hole storage engine to scale out replication logs but ultimately the only usage I’m aware of was for scaling other mysql read replicas.
The idea of scaling an application by tailing logs from a database sounds very interesting to me, and I’m curious if you’ve explored this at all. There’s of course things like Kafka (and then things like Debezium), but it’s hard to beat direct!
> I’m curious in your experience how many clients can run pg_replicate at once?
I'd expect more clients to put more pressure on the resource usage on the db. But it's not clear whether the relationship between number of clients and resource usage is linear, quadratic or something else since I haven't done benchmarking yet.
> The idea of scaling an application by tailing logs from a database sounds very interesting to me, and I’m curious if you’ve explored this at all. There’s of course things like Kafka (and then things like Debezium), but it’s hard to beat direct!
It doesn't exist yet, but I was thinking of creating a sink which exposes Postgres WAL via websockets. This way the number of clients might scale much better.
* The author is confusing memory safety with other kinds of safety. This is evident from the fact that they say you can write unsafe code in GC languages like python and javascript. unsafe != memory unsafe. Rust only gives you memory safety, it won't magically fix all your bugs.
* The slippery slope trick. I've seen this so often, people say because Rust has unsafe keyword it's the same as c/c++. The reason it's not is because in c/c++ you don't have any idea where to look for undefined behaviour. In Rust at least the code points you to look at the unsafe blocks. The difference is of degree which for practial purposes makes a huge difference.
reply