Now, this is a big one, we will need some sort of CI pipeline, but to be completely honest I'm not that happy with the way CI is currently implemented. I'd like to go back to first principles and figure out how things should actually be structured, so I'm going to keep track of things I'd like to accomplish using some sort of CI system and then hopefully figure out a nice way to accomplish this.
Run Tests / Typecheckers / See whether things compile
Depending on the language this will differ somewhat, but basically the idea is that for every commit/PR we want to run some static checks of some sort, be it unit tests, integration tests, type checkers, linters or what have you. This is mainly just a workaround because some people forget to run the test suite before commiting, although depending on the project running the entire testsuite might take a while, additionally a dev generally only tests on 1, maybe 2 platforms and I've seen good looking changes suddenly break some more niche operating systems (generally Windows, although NetBSD/OpenBSD also have some subtle difference that aren't always caught before commiting).
Build the software / Compile cross-platform releases
This is another separate job handled in CI which I think is quite different from the prior one, this might not happen on every commit / branch but having and automated / reliable way of building releases is quite important, especially since depending on the language it might be non-trivial to build for all supported platforms (C/C++ come to mind as being quite problematic, while Go/Zig seem rather simple in that regard).
One important difference is that here we actually generate artifacts that we want to publish in some fashion, it might just be a nightly build that few people use, but it makes sense to reuse that same pipeline for doing actual releases. We might also have some secrets here, though I'd discourage that but some people might embed some secrets in their executable or use secrets to gather data they'd like to include. In the beginning at least I'd try and avoid this or maybe even disallow this entirely.
Deploy the software
This one generally depends on the prior 2 processes succeeding, since generally you don't want to deploy/release something with failing tests. A pretty big issue is also that this phase generally requires secrets of some sorts, although we can probably circumvent that a little if we only produce a release.
General thoughts
I feel like most CI pipelines have it the wrong way around, they seem like shell scripts written in YAML but I feel like for the most part I'd instead like a more make based approach, I mean why do we run unit tests multiple times for the same commit just because they were commited on different branches? Why do we re-run them if only the README.md changed? Why do we have to rebuild every crate we depend on for every CI run? One problem is of course that the dependency graph is a little hard to specify and differs quite a bit depending on the language. So, since CI is generally not latency-sensitive we trade latency for reliability.
Additionally, although this is a usecase I'm not too familiar with: some CI jobs run on a schedule similar to cron, suppose this makes sense for something like a nightly release pipeline, or we might automatically try and update dependencies, and if all checks pass we auto-commit things (which actually sounds somewhat reasonable).
Just Linux is not enough, I'd say Windows/MacOS/Linux are the bare-minimum, adding ARM64 variants would also be sensible followed by the BSDs. Other architectures like RISC-V, ARMv7, Loonarch would also be nice. Maybe add something like community runners where people that have access to interesting/exotic hardware can run jobs? Since I suppose here it's mostly about running tests rather than building releases making security not that important (the code is public anyways).
We can't expect the CI Runner to have a public IP, instead I'd just reuse the SSE event stream I'd like for other parts already to signify to the runner that a new workflow is available, which it'll then try to acquite.
This of course lies in conflict with the goal of supporting multiple architectures, but since I'd like people to easily self-host this a runner must be able to work as a Docker container somehow, this is mainly because I (and probably a lot of others) run things on cheap VMs and nested virtualization is apparently quite tricky (should investigate this a little further though, since if it's possible to properly nest VMs then that would simplify things a little).
But due to those 2 conflicting requirements the CI system needs to support multiple backends since depending on the platform chosen some backend just won't be available, so we need to abstract over that.
Regarding upstream synchronization, it might also be nice if we wait for CI to succeed before pushing things upstream, could result in a powerful workflow since it works as a sort of forced pre commit hook
Thinking some more about it I think it might be a good idea to structure things roughly like make, as in we specify what we will build, dependencies (can be files/dirs from the repo or other targets) and a recipe for building it from scratch. To make this reliable each recipe only gets the files it depends on, and also it will be cached automatically (hash of all deps/recipe/runner version)
Now, this is a big one, we will need some sort of CI pipeline, but to be completely honest I'm not that happy with the way CI is currently implemented. I'd like to go back to first principles and figure out how things should actually be structured, so I'm going to keep track of things I'd like to accomplish using some sort of CI system and then hopefully figure out a nice way to accomplish this.
Run Tests / Typecheckers / See whether things compile
Depending on the language this will differ somewhat, but basically the idea is that for every commit/PR we want to run some static checks of some sort, be it unit tests, integration tests, type checkers, linters or what have you. This is mainly just a workaround because some people forget to run the test suite before commiting, although depending on the project running the entire testsuite might take a while, additionally a dev generally only tests on 1, maybe 2 platforms and I've seen good looking changes suddenly break some more niche operating systems (generally Windows, although NetBSD/OpenBSD also have some subtle difference that aren't always caught before commiting).
Build the software / Compile cross-platform releases
This is another separate job handled in CI which I think is quite different from the prior one, this might not happen on every commit / branch but having and automated / reliable way of building releases is quite important, especially since depending on the language it might be non-trivial to build for all supported platforms (C/C++ come to mind as being quite problematic, while Go/Zig seem rather simple in that regard).
One important difference is that here we actually generate artifacts that we want to publish in some fashion, it might just be a nightly build that few people use, but it makes sense to reuse that same pipeline for doing actual releases. We might also have some secrets here, though I'd discourage that but some people might embed some secrets in their executable or use secrets to gather data they'd like to include. In the beginning at least I'd try and avoid this or maybe even disallow this entirely.
Deploy the software
This one generally depends on the prior 2 processes succeeding, since generally you don't want to deploy/release something with failing tests. A pretty big issue is also that this phase generally requires secrets of some sorts, although we can probably circumvent that a little if we only produce a release.
General thoughts
I feel like most CI pipelines have it the wrong way around, they seem like shell scripts written in YAML but I feel like for the most part I'd instead like a more
makebased approach, I mean why do we run unit tests multiple times for the same commit just because they were commited on different branches? Why do we re-run them if only the README.md changed? Why do we have to rebuild every crate we depend on for every CI run? One problem is of course that the dependency graph is a little hard to specify and differs quite a bit depending on the language. So, since CI is generally not latency-sensitive we trade latency for reliability.Additionally, although this is a usecase I'm not too familiar with: some CI jobs run on a schedule similar to cron, suppose this makes sense for something like a nightly release pipeline, or we might automatically try and update dependencies, and if all checks pass we auto-commit things (which actually sounds somewhat reasonable).
Reasearch / Reading list
Must run differents OSs/Architectures
Just Linux is not enough, I'd say Windows/MacOS/Linux are the bare-minimum, adding ARM64 variants would also be sensible followed by the BSDs. Other architectures like RISC-V, ARMv7, Loonarch would also be nice. Maybe add something like community runners where people that have access to interesting/exotic hardware can run jobs? Since I suppose here it's mostly about running tests rather than building releases making security not that important (the code is public anyways).
Pull-based
We can't expect the CI Runner to have a public IP, instead I'd just reuse the SSE event stream I'd like for other parts already to signify to the runner that a new workflow is available, which it'll then try to acquite.
Must run without virtualization / within Docker
This of course lies in conflict with the goal of supporting multiple architectures, but since I'd like people to easily self-host this a runner must be able to work as a Docker container somehow, this is mainly because I (and probably a lot of others) run things on cheap VMs and nested virtualization is apparently quite tricky (should investigate this a little further though, since if it's possible to properly nest VMs then that would simplify things a little).
But due to those 2 conflicting requirements the CI system needs to support multiple backends since depending on the platform chosen some backend just won't be available, so we need to abstract over that.
Regarding upstream synchronization, it might also be nice if we wait for CI to succeed before pushing things upstream, could result in a powerful workflow since it works as a sort of forced pre commit hook
Thinking some more about it I think it might be a good idea to structure things roughly like make, as in we specify what we will build, dependencies (can be files/dirs from the repo or other targets) and a recipe for building it from scratch. To make this reliable each recipe only gets the files it depends on, and also it will be cached automatically (hash of all deps/recipe/runner version)