Programs with custom services, virtual environments, config files in different locations, programs creating datas in different location...
I know today a lot of stuff runs in docker, but how does a sysadmin remember what has done on its system? Is it all about documenting and keeping your docs updated? Is there any other way?
(Eg. For installing calibre-web I had to create a python venv, the venv is owned by root in /opt, but the service starting calibre web in /etc/systemd/system needs to be executed with the User=<user> specifier because calibre web wants to write in a user home directory, at the same time the database folder needs to be owned by www-data because I want to r/w it from nextcloud... So calibreweb is installed as a custom root(?) program, running in a virtual env, can access a folder owned by someone else, but still needs to be executed by another user to store its data there... )
Despite my current confusion in understanding if all of this is right in terms of security, syntax and ownership, No fucking way I will remember all this stuff in a week from now.. So... What do you use to do, if you do something? Do you use flowcharts? Simple text documents? Both?
Essentially, how do you keep track?
"Infrastructure as code" is what the strategy is typically called. You use one of the many tools for orchestrating configuration of hosts (Ansible, OpenTofu, Puppet, Saltstack, Chef, etc.). These allow you to provide configuration files and code for setting up your hosts in a central place. This place is typically a Git repo, allowing you to keep track of when which change was made.
Depending on the tool you use, you trigger applying the configuration on your dev PC, or there's a hosted CI/CD server which automatically rolls out the changes when a new commit is pushed.
Yes. Documentation. Documentation aaaalll the way.
You are right. In two months you wont remember the shit you had to enable/disable to make things work.
Doing things that arent a reocurring doing should be documented. Not crazy. A basic how to set up is enough.
Common/reocurring errors/situations? Document 'em
Got a semi permanent fix for problem, so that it will most likely never come up again, but possibly in 5 years? Document it fella.
You'll kiss your past self on the head and say thanks when you have an critical ticket in 5 years and remember nothing about the doing itself but that you wrote some documentation.
It will save your ass and possibly you might come out as the hero of the day for having a solution right away for a super nieche problem.
I've making a private hosted documentation for stuff, tricks and problems i learn at work.
I've had plenty of situatuons where i remembered that i already encountered such a situation yeeeaars ago at my previois employer and that i've written somtehting down in my personal documentation. Bam and just by a few mins I've got either a really good or at least a shittysysadmin-style solution that works.
Yep, and don't just state the what, but the why in your docs.
The why really helps with knowing if a step is still important, or if it no longer applies. This is especially important with anything cloud based, as I've seen weird workarounds become no longer needed due to updates, and I would never have caught it without my notes on why we had the weird workaround to begin with.
Declarative configuration fixes this problem. You don't really have to write down how to setup something because the configuration is the description.
I use NixOS so in my case all the stuff you described would be defined in a Nix code in a separate Calibre module. I can enable and disable such module at will with a single option in my main config file.
I really recommend looking into immutable, declarative systems. I think NixOS is the most complete solution but there are some other too. I have no experience with them though.
I use a lot of comments in config files, and in the past I've also used bookstack to make documentation (something I should probably do again). You're right that docker (especially docker compose) has helped with this immensely.
