Finding the Right Architecture for Lemmy

In planning the architecture for the Lemmy App, there were a few things that were top of the priority list before we ever began talking about different features to include. Some notables were:

#1 Lemmy should be cross-platform

Every team has institutional knowledge, and we don't want Lemmy to put artificial operating system restrictions on which type of team can take advantage of the unique knowledge cultivation tools Lemmy provides.

#2 Lemmy should keep itself up to date

There are so many amazing features on the roadmap for Lemmy and it's a pain for users to have to go install an update every time we put one out just to take advantage of our developments. Instead, Lemmy should keep itself up to date¹ at all times so users get the benefits of feature improvements without thinking about it.

#2.1 Lemmy updates should endeavor to be as small as possible

Similar to Visual Studio Code and Slack, Lemmy is an application built using the Electron application framework, which means the bulk of what a user interacts with is written with web technologies. Just like on the web, we want to make sure that updates to this code are as small as possible so that when Lemmy updates itself that update doesn't break the data-cap bank.

#3 Developing Lemmy should be possible without deep Electron knowledge

Much of what Lemmy does could be done strictly in the browser, but there are a few exceptions such as registering global keyboard bindings, adding a tray icon, and launching at startup which are (today) not possible in the browser. For that reason, Lemmy needs the Electron shell to provide a bridge to these desirable native capabilities.

Respecting that constraint, it would still be great if developing Lemmy didn't require venturing too far into the deep end with Electron except when strictly required.

With those goals in mind, the Lemmy Team has spent a lot of time reading the code for other Electron desktop apps trying to discover the different ways that the community solves for these ideals. Predictably there are lots of opinions and methods employed, but until recently none felt right for Lemmy... until we found HTTP Toolkit, an open source developer tool for debugging and testing HTTP.

This application is built in a way that leans completely into and embraces the foundational decision to use Electron as the application framework — a typical web browser based application (think Spotify or GitHub) will have the user interface portion of their application be at least somewhat agnostic to things that take place strictly server side such as interacting with the database, handling authentication flows, etc. There is a pretty clear dividing line for the concerns, even in monolithic applications built with Rails or Laravel or similar server-side web frameworks.

This makes it very convenient to mentally partition development work into "front end" and "back end" boxes and provides a very nice mental model which works well to explain the high level architecture of pretty much every site out there. What HTTP Toolkit did is the exact same thing, but within Electron. Instead of mixing the "front end" (UI and presentational concerns) in with the "Electron back end" (RPC messaging and invoking native API bindings), they literally start up a server (via Electron) on the host machine to act as a go-between for the UI and the Native needs of the application. The UI doesn't ever directly invoke Electron APIs, and instead it reaches out to the locally running server to ask it to invoke them, just like how a typical website would reach out to some server to ask for privileged actions to be performed like querying a database or authenticating a user.

The beauty of this is that it means that the Electron shell can be almost completely agnostic to the business functions and features of the app. All it needs to care about is managing the lifecycle of the locally running server process and keeping a handle on the different windows/views for the app. Everything else, from the perspective of the developer, looks just like typical web development, aka a UI which talks to a server — brilliant!

This is definitely not the only architecture that could work well for Lemmy, but this way of partitioning the concerns really clicked and finally felt right. Lemmy is actively being built following the mental model provided by HTTP Toolkit. Of course, Lemmy has pretty different technical needs from their application, and therefore much of what they do won't necessarily work for Lemmy, but the insight was critical to finding the right architecture for Lemmy.

So what about that priority list?

The reason this way of segmenting the concerns for Lemmy felt so right is that it directly speaks to how to solve for each of the three top priorities we talked about at the beginning:

1. ✅️ We're still using Electron, so cross-platform comes "out of the box".
2. ✅️ The server application can keep itself up to date via update checks against a release repository containing only server code. The UI application can keep itself up to date with service workers and conventional web update mechanisms.
   
   Best of all, these updates are targeted to only the portion of the app which has updated code. If the server doesn't need to change to update the UI, then you'll only get an update that's the size of a website - tiny! If the UI doesn't need to change but we have to update server logic, you'll get an update which is the size of a compiled and minified Node application - tiny!
   
   The only time you'll have to get a "full update" is if we had to change how the Electron shell works (or if we change Electron versions), which is the only time when the download is of a notable size. Luckily, thanks to how "dumb" the Electron shell needs to be, this should be rare.
3. ✅️ Since we can separate the concerns of the app, it means that development only has to be aware of the segments they need for what they're working on - it's possible to work on the UI portion without running the server, it's possible to work on the server without running Electron. This provides maximal flexibility during development and really alleviates some of the pain points around working on Electron applications.

Finally landing on an architecture that feels right is really exciting, and we're really looking forward to the improvements this will make to our development speed. Stay tuned for more!

— Cheers!