Ben / RubHub

Now, this is a big one, we will need some sort of CI pipeline, but to be completely honest I'm not that happy with the way CI is currently implemented. I'd like to go back to first principles and figure out how things should actually be structured, so I'm going to keep track of things I'd like to accomplish using some sort of CI system and then hopefully figure out a nice way to accomplish this.

Run Tests / Typecheckers / See whether things compile

Depending on the language this will differ somewhat, but basically the idea is that for every commit/PR we want to run some static checks of some sort, be it unit tests, integration tests, type checkers, linters or what have you. This is mainly just a workaround because some people forget to run the test suite before commiting, although depending on the project running the entire testsuite might take a while, additionally a dev generally only tests on 1, maybe 2 platforms and I've seen good looking changes suddenly break some more niche operating systems (generally Windows, although NetBSD/OpenBSD also have some subtle difference that aren't always caught before commiting).

Build the software / Compile cross-platform releases

This is another separate job handled in CI which I think is quite different from the prior one, this might not happen on every commit / branch but having and automated / reliable way of building releases is quite important, especially since depending on the language it might be non-trivial to build for all supported platforms (C/C++ come to mind as being quite problematic, while Go/Zig seem rather simple in that regard).

One important difference is that here we actually generate artifacts that we want to publish in some fashion, it might just be a nightly build that few people use, but it makes sense to reuse that same pipeline for doing actual releases. We might also have some secrets here, though I'd discourage that but some people might embed some secrets in their executable or use secrets to gather data they'd like to include. In the beginning at least I'd try and avoid this or maybe even disallow this entirely.

Deploy the software

This one generally depends on the prior 2 processes succeeding, since generally you don't want to deploy/release something with failing tests. A pretty big issue is also that this phase generally requires secrets of some sorts, although we can probably circumvent that a little if we only produce a release.

General thoughts

I feel like most CI pipelines have it the wrong way around, they seem like shell scripts written in YAML but I feel like for the most part I'd instead like a more make based approach, I mean why do we run unit tests multiple times for the same commit just because they were commited on different branches? Why do we re-run them if only the README.md changed? Why do we have to rebuild every crate we depend on for every CI run? One problem is of course that the dependency graph is a little hard to specify and differs quite a bit depending on the language. So, since CI is generally not latency-sensitive we trade latency for reliability.