In the long run I'd like to get rid of the .json files, while git repos should still be the main datastore, there are certain things that should be private or are inherently relational so they're better suited for a proper database. While SQLite would be the obvious first-choice, I might just give Turso a try, seems like it has a lot of velocity and I do like that it's written in Rust (simplifies compliation) and the async/multi-writer support could be quite nice for RubHub.
Constraints would be quite important here, since that seems like one of the main benefits to use a db here, otherwise I'd have to handle all the consistency checks myself
On thinking some more, I'm thinking of instead using NDJSON with an append-only log file with regular compaction. The idea is to spawn a separate thread with an MPSC channel then send messages from the axum handlers, we then just append lines and update an in-memory datastructure (which we read from disk on startup), example:
For fast lookups I'm thinking of using redb, the idea is to keep all authoritative state in git/auth.ndjson so we can always recreate the redb db. Then use redb to for example figure out which email belongs to which user, cache repo state and so on.
In the long run I'd like to get rid of the .json files, while git repos should still be the main datastore, there are certain things that should be private or are inherently relational so they're better suited for a proper database. While SQLite would be the obvious first-choice, I might just give Turso a try, seems like it has a lot of velocity and I do like that it's written in Rust (simplifies compliation) and the async/multi-writer support could be quite nice for RubHub.
What would benefit from a database:
User
User E-Mails
User sessions
Constraints would be quite important here, since that seems like one of the main benefits to use a db here, otherwise I'd have to handle all the consistency checks myself
On thinking some more, I'm thinking of instead using NDJSON with an append-only log file with regular compaction. The idea is to spawn a separate thread with an MPSC channel then send messages from the axum handlers, we then just append lines and update an in-memory datastructure (which we read from disk on startup), example:
Then store this as
auth.ndjsonFor fast lookups I'm thinking of using redb, the idea is to keep all authoritative state in git/auth.ndjson so we can always recreate the redb db. Then use redb to for example figure out which email belongs to which user, cache repo state and so on.