Doh! Forgot earlier in the night, so here you are... technically Saturday.
CPU:
*removed externally hosted image*
The lemmy devs have made some major strides in improving performance recently, as you can see by the overall reduced CPU load.
Memory:
*removed externally hosted image*
I need to figure out why swap is continuing to be used, when there is cache/buffer available to be used. But as you can see, the upgrade to 8GB of RAM is being put to good use.
Network:
*removed externally hosted image*
The two large spikes here are from some backups being uploaded to object storage. Apart from that, traffic levels are fine.
Storage:
*removed externally hosted image*
A HUGE win here this week, turns out a huge portion of the database is data we don't need, and can be safely deleted pretty much any time. The large drop in storage on the 9th was from me manually deleting all but the most recent ~100k rows in the guilty table. Devs are aware of this issue, and are actively working on making DB storage more efficient. While a better fix is being worked on, I have a cronjob running every hour to delete all but the most recent 200k rows.
Cloudflare caching:
*removed externally hosted image*
Cloudflare still saving us substantial egress traffic from the VPS, though no 14MB "icons" being grabbed thousands of times this week 😀
Summary:
All things considered, we're in a much better place today than a week ago. Storage is much less of a concern, and all other server resources are doing well... though I need to investigate swap usage.
Longer term it still looks as though storage will become the trigger for further upgrades. However storage growth will be much more slow and under our control. The recent upward trend is predominantly from locally cached images from object storage, which can be deleted at any time as required.
As usual, feel free to ask questions.
without memory pressure only inactive and pages without write activity pages are moving to swap. It would be interesting to see memory stats for Lemmy itself. I expect that rest of the stack should be pretty much memory-leak free.
I don't think I've ever seen a host exhaust swap before RAM is full..
*removed externally hosted image*
Looks to be postgres related. I've changed a couple of settings, lets see if it happens again.
A much more normal memory usage profile now
*removed externally hosted image*