Many Rust libs are not mature enough for a small team web server to be a good idea, nor is the query building/orm side without strict limitations
Can you go into detail on this? What rust libs are lacking and why? What are the limitations? Are they due to rust's borrow checker?
What features are exclusive to Hexbear but not upstream in Lemmy? Why does this fork have to deviate so much from the upstream project?
I'm primarily a JS/Python dev who wants to learn Rust and convert my existing JS projects to TS. I've been waiting for some free time to contribute to Lemmy or Hexbear. Being a fork, why should I contribute downstream and duplicate effort?
Sure, I'll keep this a bit short though (well, i tried) as some of these are probably worth being a post topic in their own right.
Actix Web - The web server framework that Lemmy chose to use kind of sacrifices a lot for the sake of raw speed (or "cheats") and has undergone a change of hands after some drama awhile back. The design patterns of the framework also feel kind of janky for the purpose of a web api (they lean on the actor pattern iirc and a custom implementation for async)- But for us one of the problems was they allow customizing the thread pool but dont really state this obviously anywhere. One of our devs found this by looking in the source and it's something that is rather critical for handling a large number of concurrent requests. It also doesn't handle middleware very well, but I believe part of that is Lemmy is using an older version and the newer ones have significantly changed some of the apis. I don't think Actix is wholly lacking here but it's far more difficult to use well than anything in JS land and IMO I'm not a big fan of the mind set that goes into creating "the fastest framework ever" over just making something that works with clearly documented behavior.
Diesel - The ORM that Lemmy uses is the bigger offender here imo. As an ORM it does not support: self-joins, group bys, having clauses, window functions, custom sql aliasing and a few other things. Part of this is due to Diesel being young-ish, but the devs themselves state on their repo that some of these are "pointless". I primarily work in databases and all of this functionality are incredibly basic feature sets that any ORM should support. As a result- Lemmy was forced to create "views" as a workaround and some of the biggest architectural problems in the codebase are due to these Diesel workarounds. I can confidently say it would be easier to write raw sql for the entire project than use this, but Diesel also intentionally limits the "escape hatch" options to do so. To me, this lib feels more like some devs trying to "gamify" Rust's type checking system instead of making a real ORM and its by far the biggest slowdown to adding features. This is one of the bigger drivers for the rewrite, if we remove Diesel we're effectively restarting from zero, and there is so much basic functionality missing here that I would consider it a "non-starter".
Features - The mod/report system was exclusive until very recently, 2fa auth and server side token validation/revocation, a lot of smaller FE tweaks are as well..(user tags too? some other devs can flesh this out) Some of the bigger things would be we significantly rewrote/redid some of the database handling because the original implementation was consistently deadlocking itself and doing entire table scans at high frequency to obtain trivial data. For example- one of the views Lemmy uses performs a cross-join on data sets and turns something like 20k comments into a "9 million row" result set that is then filtered out. None of that is necessary at all and we removed it or fixed it, otherwise the debate/election threads wouldn't have been just slow, the site would've been down. Effectively today, the Chapo backend uses an entirely different set of SQL than Lemmy- so some of those and security changes are the big reasons, and these are things we relayed to upstream early on but were not prioritized over federation.. so here we are with what is functionally a fork in all but name.
Contribution - That's difficult for me to answer because I feel if this came down to duplicating efforts we wouldn't be where we are today and there is plenty of space in the Lemmy design for shared logic if they choose to architect the code to do so. For ex, the act of handling apub requests and translating the objects could be an upstream rust library that we use with typescript, but they would have to make that possible and start decoupling a lot of core logic. In the end, apub is an open spec and Lemmy is a specific implementation/usage of it. I feel what you could be asking instead is, would you rather contribute to one consumer client, or another? I kind of see it more like matrix/mastadon, where theres a "core spec" but plenty of different and good client implementations of it.
Can you go into detail on this? What rust libs are lacking and why? What are the limitations? Are they due to rust's borrow checker?
What features are exclusive to Hexbear but not upstream in Lemmy? Why does this fork have to deviate so much from the upstream project?
I'm primarily a JS/Python dev who wants to learn Rust and convert my existing JS projects to TS. I've been waiting for some free time to contribute to Lemmy or Hexbear. Being a fork, why should I contribute downstream and duplicate effort?
Sure, I'll keep this a bit short though (well, i tried) as some of these are probably worth being a post topic in their own right.
Actix Web - The web server framework that Lemmy chose to use kind of sacrifices a lot for the sake of raw speed (or "cheats") and has undergone a change of hands after some drama awhile back. The design patterns of the framework also feel kind of janky for the purpose of a web api (they lean on the actor pattern iirc and a custom implementation for async)- But for us one of the problems was they allow customizing the thread pool but dont really state this obviously anywhere. One of our devs found this by looking in the source and it's something that is rather critical for handling a large number of concurrent requests. It also doesn't handle middleware very well, but I believe part of that is Lemmy is using an older version and the newer ones have significantly changed some of the apis. I don't think Actix is wholly lacking here but it's far more difficult to use well than anything in JS land and IMO I'm not a big fan of the mind set that goes into creating "the fastest framework ever" over just making something that works with clearly documented behavior.
Diesel - The ORM that Lemmy uses is the bigger offender here imo. As an ORM it does not support: self-joins, group bys, having clauses, window functions, custom sql aliasing and a few other things. Part of this is due to Diesel being young-ish, but the devs themselves state on their repo that some of these are "pointless". I primarily work in databases and all of this functionality are incredibly basic feature sets that any ORM should support. As a result- Lemmy was forced to create "views" as a workaround and some of the biggest architectural problems in the codebase are due to these Diesel workarounds. I can confidently say it would be easier to write raw sql for the entire project than use this, but Diesel also intentionally limits the "escape hatch" options to do so. To me, this lib feels more like some devs trying to "gamify" Rust's type checking system instead of making a real ORM and its by far the biggest slowdown to adding features. This is one of the bigger drivers for the rewrite, if we remove Diesel we're effectively restarting from zero, and there is so much basic functionality missing here that I would consider it a "non-starter".
Features - The mod/report system was exclusive until very recently, 2fa auth and server side token validation/revocation, a lot of smaller FE tweaks are as well..(user tags too? some other devs can flesh this out) Some of the bigger things would be we significantly rewrote/redid some of the database handling because the original implementation was consistently deadlocking itself and doing entire table scans at high frequency to obtain trivial data. For example- one of the views Lemmy uses performs a cross-join on data sets and turns something like 20k comments into a "9 million row" result set that is then filtered out. None of that is necessary at all and we removed it or fixed it, otherwise the debate/election threads wouldn't have been just slow, the site would've been down. Effectively today, the Chapo backend uses an entirely different set of SQL than Lemmy- so some of those and security changes are the big reasons, and these are things we relayed to upstream early on but were not prioritized over federation.. so here we are with what is functionally a fork in all but name.
Contribution - That's difficult for me to answer because I feel if this came down to duplicating efforts we wouldn't be where we are today and there is plenty of space in the Lemmy design for shared logic if they choose to architect the code to do so. For ex, the act of handling apub requests and translating the objects could be an upstream rust library that we use with typescript, but they would have to make that possible and start decoupling a lot of core logic. In the end, apub is an open spec and Lemmy is a specific implementation/usage of it. I feel what you could be asking instead is, would you rather contribute to one consumer client, or another? I kind of see it more like matrix/mastadon, where theres a "core spec" but plenty of different and good client implementations of it.
deleted by creator