news

68 posts with the tag “news”

A post we never want to have to make!

May 3, 2026

NomadicCore

Comms

Ermo

Co-founder

An image showing the word "Sorry" with three exclamation marks

This is the post we never want to make, where we made errors with our previous ISO release and are having to take steps to fix the issue, including releasing an updated ISO.

Over the course of April, the team worked on systemd-presets as a workstream we didn’t actually talk about in our April blog post. We thought that all the work had been completed in such a fashion that users wouldn’t see any discernible difference whilst allowing us to move closer to “best-practice” approaches.

Unfortunately, due to the way systemd functions, it doesn’t natively support a global preset-all approach on first boot, one of the preset approaches we had recently adopted. In combination, our lichen installer was creating an /etc/machine-id file in new installs, which is what systemd uses to detect whether a machine is in its first boot or not.

As such, users on existing systems did not have an issue, but new installs could be broken and we did not detect this as part of our ISO testing process.

Thanks to user feedback, we were able to narrow down, identify and resolve the issue within 24 hours of the ISO release, and we had an updated ISO up for testing within our server by the end of the first day of the original release. The fix came via two commits:

Lichen: Fix first boot
Systemd: Preset-all user services on first-boot

Following additional testing with a wider audience on our Zulip server, we have updated the links on our download page to our new ISO so any users downloading the latest ISO won’t see these issues.

How we view the project and our roles in creating it

Since taking over stewardship of AerynOS last year, we have consciously reset our own expectations of what we aim to deliver to our users / early adopters. It is a deliberate decision to keep the distro at an alpha tag, as there are certain expectations / deliverables for our core tooling we have not yet met.

We know we haven’t publicly laid out our roadmap, another deliberate decision, but this is all in service of delivering a product that early adopters (and eventually users) can rely on without having to worry about “what the hell might go wrong”.

There is an ethos within the team that we take seriously our craft, that being to create new and modern tooling that will make delivering and maintaining a Linux distribution significantly easier and more ergonomic.

In retrospect, we’re glad that in the year that we have collectively had stewardship of the project, this is the first and only time we have had to rush out a new ISO to fix an issue. We hope to not have this occur again in the future.

NVIDIA driver issues for certain GPUs

In the background, we have been offering a “best effort” approach to supporting NVIDIA GPUs. This is primarily because nobody on the core team are actually using NVIDIA GPUs, and because NVIDIA’s approach to open source leaves a lot to be desired from a package- and distro-maintenance point of view.

For an alpha tag distribution that is primarily focused on dogfooding itself, NVIDIA GPU support has been — and remains — fairly low priority.

That said, Reilly identified an issue with our build ordering that caused the NVIDIA module to fail to work for GPUs that require GSP firmware. With this knowledge, we have implemented a manual fix for now. The underlying issue was already known to the team, we just hadn’t caught that it presented an issue for this particular case. Fixing that issue in our infrastructure tooling is therefore moving up on our list of priorities.

Why have monthly ISOs anyway?!

The issue with new installs only presented because of our new ISO release. Had users installed AerynOS from any of our previous ISOs, the distribution would have installed without issue. Last year, we made a decision to move to a monthly ISO cadence as part of a wider “hearts and minds” effort, to demonstrate that AerynOS is in good hands, and that we are able to consistently deliver progress at a time when there was uncertainty of whether the project would be able to survive during Ikey’s initial (and, as it turned out, eventually permanent) absence from the project.

However, as a rolling release distro with a net-installer, it doesn’t strictly matter which ISO you boot to install AerynOS on your system (or in a VM), given that Lichen will always install from the latest unstable stream version of AerynOS. As such, we are reviewing our release cadence and will likely align releases around a couple of key factors:

Major Linux kernel versions for new hardware support
Updates to our installer

which will also have a benefit of reducing bandwidth consumption. This won’t however affect the frequency of our blog posts, as we have found it helpful to communicate often with those following along with the project.

We actually delivered quite a lot in the last couple of days!

Package / stack updates for this iteration include:

Updates:

linux stable & gaming 7.0.3
linux LTS 6.18.26
thunderbird 150.0.1
asciinema 3.2.0
enchant 2.8.16
faugus-launcher 1.18.10
flatpak 1.16.6
glib2 2.88.1
gtk-4 4.22.4
inetutils 2.8
libvirt 12.3.0
rssguard 5.1.0
wine 11.8

Fixes:

boulder: Ensure that failing to set thread priority to SCHED_BATCH does not cause a panic w/ backtrace.
nm-connection-editor: It’s now a separate package and can be installed without networkmanager-applet
strawberry: Fixed not supporting common audio formats

Added:

envision 3.2.0 (VR gaming)
gitui: A graphical git client
hexyl: A terminal based hex viewer with coloured output
pkgset-oxidize: A set of pkgsets for different WM environments designed to work with the oxidize theme tool
oxidize: A tool for atomically changing themes in supported WMs
yt-dlp 2026.03.17

oxidize Window Manager theming

One of our community members, Christian Bendiksen, has been working on automated theming of window managers over the last couple of months with his oxidize tool.

His work is now ready to be included in AerynOS, and can be used to automate theme setup for the four window manager options we have within AerynOS. There are many popular themes already included, and Christian has taken the initiative to play around with our new brand colour palette to make a new Aeryn theme as well!

This work builds on top of the great work Christian and a number of other dedicated contributors have been putting in to build out AerynOS’ Window Manager credentials with the inclusion of packages to bolster our offering in this area. Taking this further, Christian has also created a package set around oxidize. This package set will help simplify the process of getting set up with a great Window Manager configuration without having to go through all of the steps to get there. The next step is to take this further by utilizing our system-model approach to further develop this approach with the goal being to eventually have a simple preconfigured Window Manager option available out of the box.

Of course, for those wishing to configure their Window Manager experience from scratch, this option is of course available to you as well.

ISO refresh

With this blog post, we already have a new 2026.05.2 ISO available on our download page.

It incorporates all the changes and package updates highlighted above and as usual, serves as a vessel for you to use lichen to install AerynOS onto your system or into a virtual machine.

Next Steps

The primary focus on the development side is (still) to attempt to get the Versioned Repos, phase2 feature over the finish line as soon as feasible.

Frankly, we have been so focused on getting the Versioned Repos, phase2 feature right, that we’ve scarcely had the mental bandwidth to focus on anything else from the perspective of our larger development arc.

That said — and assuming we succeed in landing the Versioned Repos, phase2 feature soon — we will then spend some time on sketching out the details of the upcoming avenues of development that will open up as a result.

In parallel to that, we hope to spend some time getting our systemd-preset story straight from a packaging perspective, which will give us the ability to enable services as a packaging operation. This will be especially useful when leveraged via our declarative system-model capabilities.

Supporting the project

Outside of financial donations through Stripe and Ko-fi mentioned above, we are always looking for people to get involved with development and packaging efforts and welcome anyone curious about AerynOS to join us in our Zulip server!

If any hardware vendors are interested in sponsoring the project either financially or through hardware sponsorship, this would be warmly received.

Sponsor AerynOS

If you wish to discuss other sponsorship opportunities, such as hosting or hardware sponsorship, please reach out to us at contact@aerynos.com.

Thank You!

We are very grateful for your support, be it financial or via project contributions in the form of carefully written bug reports, code contributions, design contributions, documentation updates, general feedback, package updates and overall enthusiasm around the project.

We hope that you will continue showing enthusiasm for our project, and that you will want to get involved in whichever way, shape, or form works for you!

Rebranding, Upgrading, and Wallpapering: AerynOS’ April glow-up!

Apr 30, 2026

NomadicCore

Comms

Ermo

Co-founder

Image captured by a drone camera of wintery snowy fields with a river meandering through the middle

No, this isn’t a suspicious phishing attempt! AerynOS has officially gone through a rebrand 🎉

Over the course of April, we rolled out a new logo and refreshed colour palette across the entire project. It took a bit of time to thread this through every corner of the OS and our web presence, but we’re happy to say it’s now fully in place.

Alongside the new branding, we’ve also been collaborating with community member ziegenmelker5, licensing a selection of his photography for use as wallpapers (and the title image above). A handful of these have already landed in our artwork repository and will show up on user systems after the next sync. On top of that, the team has created an abstract wallpaper inspired by the new logo, so there’s a bit of something for everyone.

On the software development side, a big chunk of April was spent improving our core tooling, especially moss and boulder:

Boulder:

Added boulder cache size to show cache size of both boulder and moss.
Added boulder cache clean to free up space on system by deleting the cache.
Updated boulder recipe update to use ent to check for recipe updates and appropriately update the stone.yaml accordingly.

Moss:

moss state prune is now faster and shows a progress bar when removing states.

In addition, we’ve continued our reuse compliance work, extending it from our recipes repository into our os-tools repository. This brings us closer to full compliance across the board.

Lastly, we have expanded our kernel configurations and now offer an LTS, stable and gaming kernel, though switching away from the stable kernel isn’t yet a simple process.

What’s new in the distro

Package / stack updates for this iteration include:

COSMIC DE 1.0.11
GNOME 50.1
KDE Frameworks 6.25.0
KDE Gear 26.04.0
KDE Plasma 6.6.4
dankmaterialshell 1.4.6
dash 0.5.13.3
docker 29.4.1
eza 0.23.4
firefox 150.0.1
fresh 0.2.23
jujutsu 0.40.0
kitty 0.46.2
kmscon 9.3.5
linux-gaming 7.0.2
linux-lts 6.18.25
linux-stable 7.0.2
llvm 22.1.4
ly 1.3.2
maven 3.9.15
mesa 26.0.6
niri 26.4
ntpd-rs 1.7.2
openvpn 2.7.2
pipewire 1.6.4
qemu 11.0.0
python 3.14.4
rust 1.95.0
thunderbird 150.0
uutils-coreutils 0.8.0
wine 11.7
zig 0.16.0
zls 0.16.0
zed 1.0.0

… along with sundry additions and updates.

Infrastructure and Tooling Updates

boulder cache subcommands with clean and size options

Screenshot showing boulder cache size command with output

This month, Joey has added additional subcommands for boulder to calculate cache sizes (both for boulder and moss). Additionally, a boulder cache clean command was added that will delete the cache to help free up space on a user’s system.

This will be particularly helpful for our packagers who after building packages will have increasing amounts of space taken up by the boulder cache. The team itself frequently moves between our unstable and volatile repositories so having the ability to delete the moss cache will also be helpful to ensure predictable outcomes.

boulder recipe automation

Screenshot of a terminal session after running boulder recipe update on the yq package recipe, resulting in the automatic update of the corresponding stone.yaml recipe file

Using our self built ent tool (which integrates with Anitya for release monitoring), we’ve taken another step toward automating package maintenance by combining it with our boulder recipe update command. When invoking this command, boulder will:

Check for updates using ent
Resolve the source URL of the latest update
Download the latest version source archive
Appropriately update the stone.yaml recipe

…all in one go.

There’s also JSON output support, which will be useful as we expand our automation tooling further down the line.

For simple package updates, this helps consolidate some of our mundane workflow steps and help make packaging a more streamlined affair on AerynOS, letting packagers focus on logic and problem solving, not administrative minutiae.

Faster `moss state remove`

Screenshot of a terminal session after running sudo moss state remove 56-77, resulting in a confirmation prompt and log of the deleted state directories

State removal is now faster thanks to parallelization, and it finally provides real-time feedback via a progress bar.

Behind the scenes, moss intelligently handles our deduplicated CAS storage, which is why removal order might look a little unusual, it’s working out which files are safe to delete across shared states.

Improved `moss search`

A new contributor, otherJL0, has made some great improvements to our moss search command:

Support for using provider syntax in search, e.g. sysbinary(...)
Smarter grouping of results based on name or summary
Highlights matched substrings in output by name or summary

This is especially useful for packaging workflows as our packagers can get a better understanding of which packages provide a given binary or library.

Continuation of our Versioned Repository feature set

We’ve continued work on phase 2 of our Versioned Repositories feature, and a draft PR with the basic workflow and new configuration format has been opened for moss. The vessel repository manager companion work has been mapped out, but no PR has yet been opened for this.

As mentioned in previous posts, the key goal of phase 2 is to enable moss to upgrade itself seamlessly, which in turn enables us to add support for new repository features and .stone format features without manual intervention.

In practical terms, this moves us closer to a true install-once, update-forever model, where we can evolve the capabilities of the system over time, without leaving users on older moss versions behind.

We hope to land the phase 2 related feature PRs in the near future.

Wider Project Updates

New Branding: A Fresh Look for AerynOS

April was a big month for us here at AerynOS, as we made some exciting strides in our rebranding efforts. We’re thrilled to finally roll out a brand-new logomark, a version of the triquetra that was originally proposed by community member Petru Jenach, and later refined by community member platlas when AerynOS was rebranded from SerpentOS last year.

Working closely with platlas and the wider community, we’ve spent the last few months refining the design and selecting a brand new colour palette. Special thanks to sammypanda, who suggested the colour scheme that we ended up using.

So, what’s behind the triquetra? This symbol has deep meaning across cultures, but we’ve chosen it particularly for its Irish roots, as a nod to AerynOS’ founder, Ikey Doherty. The triquetra represents themes of life, death, and rebirth, which felt fitting for a project that’s always evolving. It also symbolises unity and commitment, values that align with our community driven approach.

Design-wise, we’ve divided the three points of the triquetra into two colours: green and orange. The green is all about nature, while the orange ties directly to Rust, the programming language at the heart of our project. We love how the orange section can be interpreted as an ‘O’ and the green section as an ‘S’, coming together to form “OS”. It’s subtle, but we think it’s a nice touch.

We’re really excited about this fresh new look, and the fact that it’s something the whole community helped us shape. We’ve spent the month rolling it out across the operating system and our web presence, and we’re happy with how it all ties together.

New Wallpapers: Bringing Nature to Your Desktop

Screenshot of a newly installed virtual machine instance of AerynOS, Gnome edition showing the about this system page with the new default AerynOS wallpaper

Next to the new logomark, we’ve also worked with community member Gabriel Janich (aka ziegenmelker5) to bring some stunning new wallpapers to AerynOS. Gabriel has an amazing collection of photos, and choosing just a handful to feature was no easy task!

In the end, we selected seven that we felt best represented the spirit of AerynOS. You’ll see two of them as the new default wallpapers for Cosmic and GNOME (we have some work to do to properly customise our KDE Plasma defaults). Each one has its own vibe, but all of them bring a touch of nature and tranquility to your desktop.

We hope these new wallpapers, paired with the fresh logo, help make your AerynOS experience even more enjoyable as you dive into the project in the coming days and months.

Screenshot of a newly installed virtual machine instance of AerynOS, Cosmic edition showing the about this system page with the new default AerynOS wallpaper

Python stack upgrade

We mentioned last month that we planned a Python stack upgrade. Due to diligent prior preparation work by Reilly, this stack upgrade landed in a fairly seamless manner with only minor fixes required over the course of a day. The process saw Python being upgraded from 3.11 to 3.14.4 as of this writing.

Our python stack isn’t currently very large (only around 200 packages) which also played a role in the fairly seamless nature of this update. From what we can tell, the updated has enabled our early adopters to continue using Python packages without any regressions.

Kernel updates

We have packaged two alternative kernel options, though we have yet to finish the supporting work required to let users easily switch between them:

linux-lts (6.18): Latest LTS release for our most stable offering.
linux-stable (7.0): Follows the latest stable release with very few optimisations for a current stable offering.
linux-gaming (7.0): Also follows the latest stable release, but with patches for handheld gaming and miscellaneous performance optimisations.

Switching away from our linux-stable kernel to either of the other two alternative options isn’t a smooth process just yet. Improving that experience is on our roadmap.

ISO refresh

We are releasing our newest Alpha ISO, AerynOS 2026.05, which includes the updates we’ve worked on since the start of April, and which features the 7.0.2 linux-stable kernel.

As usual, this is a Live GNOME ISO that merely serves as a delivery vehicle for our Alpha/PoC lichen installer. Hence, installing AerynOS requires a network connection over which the latest pkgsets can be downloaded and subsequently installed onto a hard drive.

Please note that for now, Ventoy cannot be used to install AerynOS ISOs for which we have submitted a bug report upstream. Not to worry, multiple other options work such as Etcher, DD and GNOME Disks.

The link for our 2026.05 ISO can be found on our download page.

Next Steps

The primary focus on the development side is to attempt to get the Versioned Repos, phase2 feature over the finish line as soon as feasible.

Supporting the project

Over the last year, the project has been through a significant period of change. As detailed in our October 2025 blog post, we had to update our sponsorship accounts to receive future sponsorship funds once it became clear our previous project leader had permanently stepped away from the project.

This left us in a position where we had to build up our sponsor income from scratch having lost previous sponsors. We are very grateful that many sponsors (old and new) have joined or stayed with us on this journey and our income is again able to cover our fixed project costs with a little surplus each month.

As of this month, we are now in a net neutral position having borne the project costs for a year whilst receiving sponsorship income for 6 months. We are very appreciative of all who have ever sponsored the project, we wouldn’t be here without your support! ❤️

Ideally we would like to grow our monthly income (and therefore surplus). Doing so would allow us to:

Support our staff who currently work without compensation (apart from occasional hardware donation)
Scale and/or upgrade infrastructure over time
Consider purchasing hardware for compatibility testing
Fund future initiatives for the betterment of the project

Sponsor AerynOS

If you wish to discuss other sponsorship opportunities, such as hosting or hardware sponsorship, please reach out to us at contact@aerynos.com.

Thank You!

We hope that you will continue showing enthusiasm for our project, and that you will want to get involved in whichever way, shape, or form works for you!

March 2026 project update

Mar 31, 2026

NomadicCore

Comms

Ermo

Co-founder

AerynOS: March 2026 project update

Climbing

Another month brings another project update. In some respects, March has felt somewhat quieter than usual for the project. However, in the background, things have been fairly busy on the development front, as we prepare for larger and more visible tooling updates to land in the months to come.

On the packaging front, notable package & stack updates include GNOME 50, KDE Plasma 6.6.3, LLVM 22.1.1, Wayland 1.25.0, QT6.11.0, FFmpeg 8.1 and mesa 26.0.3 which have kept our builders busy with a significant number of package builds and rebuilds. The team mentioned last month that there is a soft-freeze in place for our repository, in terms of accepting new package recipes. This is not a full freeze as packages with little or no reverse dependency rebuild chains are still being accepted where we believe they will prove useful.

The development work this month has centered around the features we offer in terms of automating recipe updates and how we can better support bootstrap builds on the boulder side, along with the usual clean-up and refactoring work across the moss and boulder code bases.

Lastly, we want to take a moment to thank Framework for their on-going hardware sponsorship of our project. This month, they have provided the project with an AMD AI 300 based Framework 16 that Joey Riches is now using as a dedicated device for AerynOS development.

What’s new in the distro

Package / stack updates for this iteration include:

GNOME 50.0
KDE Frameworks 6.24.0
KDE Gear 25.12.3
KDE Plasma 6.6.3
awww 0.12.0
dankmaterialshell 1.4.4
firefox 149
fish 4.6.0
fresh 0.2.20
jujutsu 0.39.0
linux 6.18.19
mangowm 0.12.7
mesa 26.0.2
pipewire 1.6.2
qemu 10.2.2
riftbar 0.1.8
thunderbird 149.0.1
wine 11.5
dash 0.5.13.2
wlroots 0.19.3
wayland 1.25.0
maven 3.9.14
gparted 1.8.1
kitty 0.46.1
systemd 257.13
zls 0.15.1
sudo-rs 0.2.13
llvm 22.1.1
Qt6 6.11.0

… along with sundry additions and updates.

Desktop Updates

GNOME

Gnome Install

This month sees a significant update to the GNOME stack with the delivery of GNOME 50. We have not received any reports of any issues but if you do find any bugs, please report them back to us via GitHub Issues or through our Zulip community.

Some key updates include:

Improved parental controls
VRR and Fractional Scaling are now stable and enabled by default
Wayland only, coming in line with AerynOS’ default of having no X11 session

Plus many more features and fixes.

KDE Plasma

KDE Install

KDE Plasma has been updated to 6.6.3, KDE Frameworks to 6.24.0 and KDE Gear to 25.12.3.

The latest KDE Plasma update brings:

KWin’s screencasting feature has become robust when using PipeWire 1.6 or newer
Reduced CPU and GPU load for full-screen windows for screens using more fractional scale factors
Improve support for mice with high-resolution scroll wheels in the built-in remote desktop server

Wayland Compositor Environments

Our packaging community has really taken to the Wayland Compositor Environment choice we have in AerynOS. With new packagers getting involved with AerynOS, we are seeing faster package updates and submissions for packages that specifically enhance people’s ability to rice their environments just the way they like them.

A few key updates this month include:

mangowm 0.12.7
dankmaterialshell 1.4.4
elephant 2.20.3
bluetui 0.8.1
impala 0.7.4
awww 0.12.0
riftbar 0.1.8

On top of this, several packages have been rebuilt for better performance, support or defaults. The default terminal in our Sway package set has been swapped from alacritty to foot as this is the default terminal proposed by Sway itself. Elephant has had upstream support for moss included and this version has now been added to our own repository. Niri has seen a few upstream improvements backported into our repository also.

Infrastructure and Tooling Updates

Use CDN for the default volatile stream

As part of our content delivery strategy, we had already moved to using a CDN for delivering updates through moss on our Unstable repository. This month, we have made the transition to also using the CDN for our Volatile repository as well.

The Volatile repository is primary used for those who do their own packaging and/or submit packages to our repository. This change helps speed up downloads for those users as they disproportionately engage with our repositories for packaging purposes.

We have updated our documentation accordingly, so if you do any packaging work on AerynOS, please refer to the updated documentation on our dotdev site.

moss: Search for binaries

Moss searching has been updated to add the ability to search by binary provider.

Currently, the command line to do so is moss search --provides some-binary

Example:

$ moss search --provides cat
uutils-coreutils   Cross-platform Rust rewrite of the GNU coreutils

The current implementation is somewhat limited, and work is ongoing to improve and generalize this functionality.

moss: Enable removal of ranges of states

Moss works by utilizing a Content Addressable Storage architecture. This means that all package related files are stored in a deduplicated manner in the CAS and then hardlinked to transaction numbers /usr trees. This has the benefit of allowing for all historic states of a system to be kept without significant overhead of duplicating the majority of all data, as would happen with simple snapshots.

Space is not infinite on our systems however, so moss was designed to be able to delete transactions. This could be done, either individually, or by automatically deleting all but the last 10 transactions. Any unique files in the CAS related to these transactions would be deleting, help free up space on a users system.

Thanks to a contribution by one of our long time trusted contributors, AnonAlly, this moss state removal command now has the ability to either delete multiple states by state number or by ranges of states. This is a nice usability improvement as otherwise, users had to individually delete states one at a time. Given that some of our systems are now reaching into the hundreds of states, this is a nice time saving QoL improvement.

Example:

$ sudo moss state remove 2 5-14 17 23-28

boulder: Add support for control files

After a feature request from Reilly, ermo and tarkah designed a flexible control file format for boulder, which is initially intended to help support bootstrapping efforts for major stack updates.

During these updates, the tests within our recipes can fail though this is expected behavior. In part to conveniently address this failure mode, the control file format enables packagers to override, prepend or append phases within our build recipes.

Initially, we have created a shared control file that outright disables tests, which can be activated by symlinking it in next to recipes whilst bootstrapping them. This will enable initial bootstrapping builds to succeed before final builds are completed with the control file removed and tests therefore re-enabled.

The benefit of the shared control file approach in this instance, is that it does away with the need to go in and manually edit each recipe to disable (and subsequently re-enable) the check phase for the stack that is being bootstrapped.

We have built the control file using the KDL format as part of our wider transition to this file format within our tooling.

To those wondering why we didn’t just add a boulder build flag for this, the reason is that we have a few other future use cases in mind for this feature that are also projected to benefit from a control file, hence settling on the current design.

Wider Project Updates

Recipe repository REUSE compliance

Unless otherwise stated, all packaging recipes in our recipes repository are available under the terms of the MPL-2.0 license. This was managed both at the repository level in the readme and as a header on individual files.

Over time, the application of our header text has varied, specifically in relation to our move from the SerpentOS name to AerynOS. Fabio has reviewed the best practices for ensuring REUSE compliance and updated all files in our recipe repository to standardize the header text to these best practices.

For anyone doing local packaging work, you will need to ensure you update your local forks of the recipe repository and update boulder to ensure that you are working from the latest version of the files, and that new recipe files are created with the correct header text. You may also need to rebase existing PRs to ensure compliance to the new standard.

In addition, a new CI process has been created to check for this REUSE compliance for package recipes. This will ensure compliance with our licensing standards into the future rather than letting the files deviate over time again.

Framework hardware sponsorship

Framework

Back in August 2024, Framework provided the project (then SerpentOS) with an AMD Ryzen 7840u based Framework 13. At the time, this was helpful for Ikey (our project founder and former project lead) to have a separate and dedicated laptop for working on the project following the appropriate hardware enablement.

Fast forward to 2026, the Framework team have provided additional hardware sponsorship in the form of an AMD AI 300 based Framework 16 laptop which is being utilized by Joey Riches. Our project, whilst still in Alpha has matured significantly since 2024 so once Joey disabled secure boot, the installation occurred without a hitch without any further hardware enablement required.

Joey is coming from a desktop Ryzen 3000 based system so coming to this new Framework 16 has provided a nice upgrade in performance whilst also acting as a dedicated device for this work on AerynOS.

Unfortunately, when Ikey stepped away from the project, we lost access to the Framework 13. However, I (NomadicCore) also personally own an AMD 7840u based Framework 13 so can ensure continued hardware support for the device.

As a team, we like the repairable nature of Framework hardware and appreciate them supporting an up-and-coming distro such as ours!

ISO refresh

We are releasing our newest Alpha ISO, AerynOS 2026.03, which includes the updates we’ve worked on since the start of March, and which features the 6.18.20 kernel.

We did notice an issue in our 2026.02 ISO whereby it would not boot from Ventoy USB sticks. The issue reoccured with our 2026.03 ISO and we believe we have identified the issue. We have raised an issue with Ventoy and hopefully will have this resolved soon.

The link for our 2026.03 ISO can be found on our download page.

Next Steps

Python upgrade and repository rebuild

Next month, we are aiming to upgrade our Python stack (which is a notoriously invasive undertaking), along with doing a full repository rebuild across our ~1500 recipes. This effort deliberately coincides with us having landed LLVM 22 this month.

Our last full repository rebuild was completed in June 2025, following the then recent transition to our Rust-based infrastructure. With almost a year of infrastructure and tooling updates, it will be interesting to see how the rebuild process pans out this time.

The AerynOS build infrastructure currently runs with four permanent builders. In addition, it is trivial for us to add temporary builders to help manage peak demand periods.

For this rebuild trek, we will likely add one or two temporary builders to help keep the queue flowing past the larger builds that would otherwise clog up the queue.

Full repository rebuilds like this are important for a couple of reasons:

They test our infrastructure and help highlight any deficiencies that we need to address
They ensure full repository ABI compliance, to help minimize the risk of packages not working due to incompatible dependencies
They help ensure that our recipes and our build artefacts are all valid and up to date
They demonstrate that we can rebuild our repository if we need to for whatever reason

The current iteration of our infrastructure is at ~2750 builds, and is no longer a frustration point for the team when submitting packages for build.

This stands in stark contrast to our original Proof of Concept infrastructure that, in the background, became quite fickle and had increasingly become a source of frustration until it was replaced.

Versioned Repositories, phase2

This month we have been steadily building towards phase2 of our Versioned Repos feature set. We highlighted this in last month’s blog post, and the aim remains to be to teach moss how to seamlessly upgrade itself on user systems, in a way that automagically enables support for new repository and .stone format features.

This will set us up nicely from an “install once, update forever” perspective, as we will always be able to add new features to our tooling without requiring fresh installs or manual upgrade interventions on user systems.

Supporting the project

If any hardware vendors are interested in sponsoring the project either financially or through hardware sponsorship, this would be warmly received.

If you wish to discuss other sponsorship details, please reach out to us at contact@aerynos.com.

Sponsor AerynOS

Thank You!

In that vein, we would also like to (in this case, once again) give Framework Computers a shout out for their generous support in the form of hardware sponsorship for project members now and in the past.

February 2026 project update

Feb 28, 2026

NomadicCore

Comms

Ermo

Co-founder

AerynOS: February 2026 project update

Being Forged

February has been a busy month for the project with a lot of activity around our tooling and infrastructure. We have merged a number of smaller improvements, which have both led to an increase in useful features, correctness and maintainability. Moss has become significantly quicker in usage, boulder has seen improvements to help automate recipe creation & updates to a set standard, and our summit dashboard has seen improvements to better represent build queues live and dynamically in graphical form.

The website re-design is progressing nicely. Last month we shared that we are using the Hextra theme. Over the last month, we have coordinated with the Hextra developer around conformance to WCAG 2.2 Level AA (Web Content Accessibility Guidelines) and have been really impressed with their rate of development in this area.

Documentation has seen continued focus. We have reworked our FAQ section, the installation section has been re-written to make it clearer, and we have conducted a site-wide documentation spelling review to conform to American English.

We have also seen packagers becoming more active in helping flesh out the recipes repository and helping keep existing packages up-to-date.

During the last week of February, we have shared with our packaging community that we will likely be more focused on maintaining our existing set of packages and becoming more selective on adding new packages in the coming months, whilst our tooling capabilities mature to where they need to be for efficiently managing long term growth.

What’s new in the distro

Package / stack updates for this iteration include:

COSMIC 1.0.8
GNOME 49.4
KDE Frameworks 6.23.0
KDE Gear 25.12.2
KDE Plasma 6.6.1
awww 0.11.2+git.2c86d41
bazaar 0.7.10
dankmaterialshell 1.4.3
docker 29.2.1
dracut 110
firefox 148
fish 4.5.0
fresh 0.2.4
ironbar 0.18.0
jujutsu 0.38.0
libinput 1.31.0
linux 6.18.15
lucien 0.1.0+git.ee18358
mangowc 0.12.4
mesa 26.0.1
pipewire 1.6.0
qemu 10.2.1
riftbar 0.1.4
thunderbird 148.0
waybar 0.15.0
wine 11.3
yazi 26.1.22
zoxide 0.9.9

… along with sundry additions and updates.

Desktop Updates

Cosmic

Cosmic Install

With Cosmic Desktop landing frequent updates, the AerynOS team have landed on a nice rhythm for packaging and landing these point releases into our own repository shortly thereafter.

We are currently on 1.0.8 with all the updates that the System76 team have landed over the last month. The team are not aware of any specific AerynOS regressions or bugs on the Cosmic Desktop Environment, so you should have a pleasant experience using it.

Some key updates include:

Addresses possible VLC freezing with COSMIC Applets
Removal of unsupported actions from the Recents section in File Manager
Copy the current path by pressing Shift in File Manager

If you do become aware of any issues, as with any other DE, you can report these in our recipes repository issue tracker.

Gnome

Gnome Install

Similarly with the GNOME stack, the team are in a nice rhythm of packaging updates as upstream updates come through.

This is an area where the team could use some support. If you are able and interested, we are looking for packagers who would be interested in helping maintain the GNOME stack.

This month we have updated to Gnome 49.4 which is a stable bug fix release with updates across the Gnome stack.

Some key updates include:

Fix tab focus behavior in the Quick Settings menu
Prevent the recreation of the default folders after they are removed
Fix screen sharing of monitors with no frame rate

Plus many more fixes.

KDE Plasma

KDE Install

KDE Plasma has been updated to 6.6.1, KDE Frameworks to 6.23.0 and KDE Gear to 25.12.2. With this update, Plasma Login Manager has been promoted to become the default KDE install option in lichen, with SDDM being the backup alternative.

The latest KDE Plasma update brings:

Introduction of Plasma setup as a new first-run wizard
Spectacle now supports OCR which allows you to pull text from screenshots
Accessibility features such as a new gray scale filter inside the Color Blindness Correction settings in System Settings

Wayland Compositor Environments

Last month we shared how, through wider packager interest, we were able to add several Wayland compositor environments to the AerynOS repository. Over the last month, we have seen continued interest in using Wayland compositors on AerynOS with examples of different configurations being shared in the “Show and Tell” channel on our Zulip server.

One area of feedback we received was that the “Console-only” only installer option did not have the necessary packages installed to ensure wifi was available and working following a reboot out of the AerynOS live environment into the actually installed Console-only environment. This was a problem for some users who don’t have an Ethernet connection to their device and rely on wifi.

In response, we have created an additional “Console-only” installation option and included it in the latest ISO release this month. As a result, we now offer both the existing “Headless Install” option which is primarily intended for a server type scenario where Ethernet connectivity is expected to be present, and the aforementioned “Console-only” option which will also include the necessary packages for wifi.

In time, we would like to use our system-model approach to allow people to share their configurations with other users. This will become a straight-forward way of being able to test pre-riced versions of AerynOS with Wayland Compositor environments, whilst also allowing users to conveniently switch back to more traditional Desktop Environments without requiring full system reinstalls.

Infrastructure and Tooling Updates

Boulder and recipe updates

The team have made a number of updates to our package build tool, boulder, as part of our wider efforts to make packaging easier for all involved.

Due to design decisions in YAML, the format we currently use for our recipes, the field version: 0.010 would be parsed into version: 0.01, which is not ideal. This is particularly important for our ent version checking tool as it would incorrectly assume packages were out of date.

To solve this, we have batch converted all recipe version fields from floats to strings, and updated boulder (specifically the boulder recipe new or boulder recipe update commands) to automatically quote version fields as well. Our CI tooling now also checks that versions fields are quoted.

Finally, Boulder now also caches upstreams fetched with boulder recipe new or boulder recipe update. This is important as the prior workflow meant that packagers were wasting time and bandwidth downloading upstream sources twice.

Summit queue visualization

The team have added new visualization functionality to our Summit dashboard to show the dependency graph of the current build queue. If builds are blocked, the queue visualization will help show what blocks the queue, which will in turn give the team greater insight on where they need to look to fix the issue and unblock the queue.

This dependency graph is live and dynamic so will update itself as it works through the package queue. This looks pretty cool when our builders are flying through large stack updates such as the monthly KDE point releases.

Boulder `--verify` flag for manifest checks

Boulder has gained the ability to verify newly rebuilt manifests against existing repo manifests. This enables us to fail builds where the rebuilt manifest does not match the existing manifest, indicating the recipe and its associated .stone package artefacts are out of date with respect to the current infrastructure or tooling.

This will come in handy when doing larger-scale rebuilds.

Moss blitting speeds

A combination of recent updates within the AerynOS project, alongside what we expect to be improvements within the Linux Kernel, have led to a drastic improvement in blitting speeds as part of the AerynOS atomic update process.

We ran a fresh battery of tests on a test system to represent how blitting performance is impacted based on different drive types and filesystems.

CPU: Intel i7-13700T
Memory: Samsung DDR05 16gb 4800 MT/s (2x 8gb sticks)
HDD: Seagate ST1000LM035-1RK172 1tb (Sata 6Gb/s)
SSD: SanDisk SDSSDH32000G 2tb (Sata 6Gb/s)
NVMe: Western Digital SN 810 512gb (Gen4 NVMe)
Operating System: AerynOS 2026.01
DE: Cosmic 1.0.6
Kernel: 6.18.9-126.desktop
Date: 16/02/2026

Type	F2FS	ext4	xfs
HDD SATA3 Cold	0.4k/s	0.3k/s	3.1k/s
HDD SATA3 Hot	13.8k/s	33.4k/s	155.1k/s
SSD SATA3 Cold	34.0k/s	66.7k/s	387.2k/s
SSD SATA3 Hot	69.9k/s	805.1k/s	809.3k/s
Gen4 NVMe Cold	104.8k/s	176.4k/s	428.4k/s
Gen4 NVMe Hot	258.7k/s	765.0k/s	683.1k/s

A brand new install of AerynOS has around 65k files meaning the blitting performance of any measure in the table above with a performance of ~65k/s will be performed in roughly 1 second after a “cold” boot on all SSD drives.

The “cold” stats are where the system has just been turned on and has not yet performed a transaction yet. The process of creating a new transaction caches disk inodes and dentries into the kernel vfs cache in memory, meaning any subsequent transactions will be significantly faster.

We call these subsequent transactions “hot” and as you can see, the hot transactions are much faster than the “cold” transactions on the same system and the same drive.

For reference, when Serpent OS hit pre-alpha in August 2024, the Gen4 NVMe drive with XFS was hitting around 70k on “hot” transactions so there has been a significant speed up since then.

It is important to stress that the numbers above specifically relate to how quickly each filesystem can update hardlinks in dentries when using moss to generate hardlinked filesystem /usr transactions. The numbers should not be taken to indicate any general measure of performance.

Trigger speeds

Hidden from this table above is how long it takes for our triggers to complete. On the slowest variation above with the SATA3 HDD, the trigger step could take up to 7 minutes, however on the Gen4 NVMe it would be done in about 2 seconds.

The team is looking at how to improve trigger performance, as any improvements in this area are the last real hurdle to having moss updates feel subjectively fast and smooth across a variety of less than ideal scenarios, such as older hardware and/or in virtual machines.

Our performance enthusiast, Joey Riches, is already testing various approaches, including parallelizing and/or coalescing the trigger actions as a way of improving the overall trigger wall clock run time. The real world impact of this will vary on a system by system basis, depending on how many cores/threads a system has available.

Wider Project Updates

Website redesign

The team have been working on the new project website re-design over the last month as a background activity. This will continue as a background task as and when the team have time, however progress is still being made.

The team aligned on the Hextra website theme last month, however there were a few features we would have liked to see included within the team (to avoid us having to use custom CSS to deliver what we needed).

Fortunately, the developer of Hextra is very active and a number of these features had already been requested and delivered in the main git branch of the theme.

One area we felt is important is accessibility. We fed back to the Hextra developer and he delivered a whole suite of updates to the theme to better conform to the WCAG 2.2 Level AA standard in the space of two weeks!

We see this as the benefit of open source projects and as part of our commitment to championing great open source projects, we would say, if you want to build a website, the Hextra theme is really good and the developer is active and engaged!

If you are interested or experienced in website design and want to help us out, join our Zulip server and get to know the team.

Documentation

Work is continuing on the documentation site with a focus this month on our FAQ section.

The team have split out the FAQ section into sub-sections and also reviewed the content and updated it based on the latest position of the project.

Notable changes include:

Split the FAQ section into sub-sections
Reviewed the documentation site for spelling mistakes and standardized around American English
Tweaks to existing content to ensure it is up-to-date
Added a clarification of why we don’t have a Discord server
Formalised our LLM policy on the documentation site

Donation methods

Over the last 3 months, we have been promoting the use of Ko-fi as our primary method of accepting donations. This month, the team reviewed its options, and subsequently created donation links directly through Stripe. Going forward, we are requesting that people donate through Stripe as the primary option, though we will still maintain Ko-fi as a secondary option.

The project already had a Stripe account as this is required for the easiest payment process from Ko-fi into our bank account. NomadicCore did a comparison of fees on a €5 donation (converted to local currency) from both Ko-fi and Stripe and found the following:

Donation method	Ko-fi	Stripe	Difference
Payout	37.34	37.35	0%
Stripe conversion fees	-0.75	-0.75	0%
Stripe processing fees	-3.01	-2.36	-1.74%
Ko-fi fees	-1.87	0	-5%
Voluntary Stripe Climate contribution	-0.37	-0.37	0%
Total	31.34	33.87	-6.77%

Note that the total payout does not match as these transactions were on different dates with slightly different conversion rates.

As such, total fees on a €5 donation reduce from 16.07% down to 9.32% when done via Stripe instead of Ko-Fi.

The downsides we have noted with Stripe donation links so far, is that donators are getting a confirmation screen but no confirmation email to confirm their donation has gone through.

Additionally, one area Ko-fi is justifying their 5% fee is the engagement aspect of being able to send us a message when making a donation and for us as a project to provide updates through Ko-fi.

At this stage, we believe our donors would prefer as much of their donation as possible to come through directly to the project, but will continue offering both options so as to enable supporters to make their own choice in this regard.

ISO refresh

We are releasing our newest Alpha ISO, AerynOS 2026.02, which includes the updates we’ve worked on since the start of February, and which features the 6.18.15 kernel.

The link for our 2026.02 ISO can be found on our download page.

Next Steps

Package addition soft-freeze

As mentioned in the intro section, we are putting new package additions in soft freeze, until our tooling and infra capabilities mature to where they need to be for efficiently managing long term growth.

During this period, additions with little or no reverse dependency rebuild chains will be strongly preferred. Luckily, most Rust (and Go) packages follow this pattern.

Invasive toolchain and full-repo rebuilds

During March, our goal is to complete a set of relatively invasive stack upgrades, including upgrading our Python stack from 3.11, and updating to LLVM 22, once the latter has seen a point update or two.

Once both have been completed, we plan to do a full recipes repository rebuild to ensure all of our packages are up to spec.

General recipes repository housekeeping

As part of our documentation push, we are cleaning up our recipes repository to simplify management and increase reliability.

The stringification of version fields represents one of these cleanup tasks.

In March, we expect to also simplify our recipe license headers and look at how we might make our recipes repository properly REUSE compliant.

Versioned Repos, phase2 work

Last month, we reported that this was now a short term target. We are happy to report that during February, we successfully completed the overall design of our moss-format upgrade process.

This includes a revised on-disk repository format that reuses the design work we did for Versioned Repos, phase1, but which rejiggers (yes, that’s the scientifically accurate term) things to account for our new moss-format repository “root index” feature.

The “root index” feature will show older moss clients how to seamlessly update themselves and user systems, in a way that automagically enables support for new repository and .stone format features.

It will also be responsible for handling repository “tag” versions as well as our “stream” concept.

This is the final step towards realising the core tenet of tooling-based “Install once, update forever” support across packaging format upgrades, which has been our guiding star since the inception of the project.

When completed, this will in turn enable us to begin to tackle some long-standing issues in our tooling and our recipes repository, that we have until now been unable to rectify without significant disruption to installed systems.

Supporting the project

If any hardware vendors are interested in sponsoring the project either financially or through hardware sponsorship, this would be warmly received.

If you wish to discuss other sponsorship details, please reach out to us at contact@aerynos.com.

Sponsor AerynOS

Thank You!

In that vein, we would also like to give Framework Computers a shout out for their generous support in the form of hardware sponsorship for project members now and in the past.

January 2026 project update

Jan 30, 2026

NomadicCore

Comms

Ermo

Co-founder

AerynOS: January 2026 project update

Fresh Start

Kicking off 2026, activity across the AerynOS project has been moving at pace. As covered in our 2025 retrospective, last year the team laid the groundwork for the project’s future growth and development. Starting off 2026, the team is building upon this foundation to keep delivering progress across the project.

From a core tooling perspective, we are preparing for the next phase of our Versioned Repository feature set. During January, we have been refining our existing codebase to make future improvements easier to implement. Once delivered, moss will be able to update itself seamlessly through what would otherwise have been breaking changes to its codebase. This functionality will be very important for our long term goals as a rolling release distro where you never have to reinstall to receive the latest updates.

Progress has been made around our branding with efforts towards a new website design and a healthy discussion and proposals for a new AerynOS logo.

You may recently have seen that AerynOS is taking a stronger stance against the use of LLMs and generative AI. Our LLM policy can be found on GitHub and will be replicated on our website in due course.

Additional progress has been made project-wide on our documentation and we have also made improvements to our CDN setup for improved package delivery.

Our efforts seem to be received well, as we are seeing an influx of new members joining our Zulip server and getting involved with AerynOS, whether through general discussion or through packaging efforts. We have also seen a jump in our followers across our various social media accounts and generally a more positive reception of what AerynOS is trying to achieve in the Linux space.

What’s new in the distro

Package / stack updates for this iteration include:

COSMIC 1.0.3
GNOME 49.3
KDE Frameworks 6.22.0
KDE Gear 25.12.1
KDE Plasma 6.5.5
brush 0.3.0-dev (bash compatible shell written in Rust)
dankmaterialshell 1.2.3 (build your own wayland desktop experience in material design style)
firefox 147.0.2
fish 4.3.3 (user friendly shell written in Rust)
foot terminal 1.25.0 (fast, wayland-native terminal emulator)
ghostty terminal 1.3.0-dev (virtual terminal written in zig)
glibc 2.43+git.144ba302
lact 0.8.3 (gui for tweaking gpu voltages, fan curves and frequencies)
linux 6.18.7
mangowc 0.10.10 (wayland compositor with eye candy effects)
mesa 25.3.4
nodejs 24.13.0
pipewire 1.4.10
prism-launcher 10.0.2 (minecraft launcher)
qemu 10.2.0
quickshell 0.2.1 (building blocks for wayland compositor-based desktop experiences)
ruby 4.0.1
thunderbird 147.0.1
vscode 1.108.2
vscodium 1.108.10359
wine 11
zed 0.221.4 (text editor written in Rust)

… along with sundry additions and updates.

Desktop Updates

Cosmic

Cosmic Install

Given System76’s move to a more regular release cycle for Cosmic DE, we are able to land updates to our repository more frequently. This month, System76 landed Cosmic 1.0.3 with some key updates including support for rounded corners and window shadows across all applications and additional appearance settings being made available.

Cosmic context
Section titled “Cosmic context”

AerynOS is following System76’s “Epoch” git branch for Cosmic DE. This is where System76 stages new updates to the Cosmic ecosystem before publishing new point releases. As such, our Cosmic package is actually somewhere between 1.0.3 and 1.0.4.

Given that Cosmic 1.0.4 has been released, it will be included in our repository shortly. It may already be there by the time you read this blog post.

Gnome

Gnome Install

This month sees the inclusion of Gnome 49.3 which is a stable bug-fix release with updates across the Gnome stack.

Some key updates include:

Nautilus file manager no longer wastes resources on images with larger dimensions
GNOME Control Center fixes for time zone searching and monitoring app filter changes in the Applications panel
Loupe improvements in performance

Plus many more fixes. See the upstream changelog for more details.

KDE Plasma

KDE Install

KDE Plasma has been updated to 6.5.5, KDE Frameworks to 6.22.0 and KDE Gear to 25.12.1.

The latest KDE Plasma update brings:

Improved XWayland app support
Krunner having better matching algorithms for what users are searching for
A bug fix to kwin to properly handle drag-and-dropped text

New Wayland Compositor Environments

With the increased interest surrounding AerynOS, we have seen some new contributors join the community and wanting to package up their favourite wayland compositors and supporting packages. Hence, after some mentoring, MangoWC and a gaggle of ancillary packages were landed this month, and they join the existing Niri and Sway wayland compositors in the package repository.

As a result, if you’re the sort who prefers to assemble your own Desktop Experience, we now have foot, quickshell and dankmaterialshell as a few examples of packages you can use as a base to make your Desktop Experience “just so”.

As a reminder, we do not include MangoWC, Niri or Sway as options to install directly from our lichen installer. Users interested in building their own Desktop Experiences on AerynOS can use our terminal-only option and then install their preferred Wayland Compositor and associated packages of choice from the command line.

Why not use our system-model?
Section titled “Why not use our system-model?”

Our new system-model approach can be used as a way of significantly simplifying the process of setting up a new AerynOS install by declaratively stating which packages you want installed on your system from a given repository (currently either Unstable or Volatile) and even being able to lock your environment to any given fixed-in-time stream update tag.

For more information about our system-model approach, please refer to the our previous 2025 in retrospect blog post

Infrastructure and Tooling Updates

debuginfod

This month, courtesy Joey Riches, we brought up a debuginfod instance at https://debuginfod.aerynos.dev! For those unaware, debuginfod is a service that finds and collects debug information in packages and hosts them to be downloadable over a simple Web API.

In practice, this means that debuggers such as gdb can automatically download the correct debuginfo files matching an executable’s build-id. No more manually searching for and installing the correct -dbginfo packages to successfully populate a backtrace with symbols.

To use AerynOS’s debuginfod service ensure you have https://debuginfod.aerynos.dev in the $DEBUGINFOD_URLS environment variable. Assuming you have libdebuginfod installed, this should be set up to work out of the the box.

-dbginfo packages are still available in the repository in case you don’t want to use debuginfod, or your tool of choice doesn’t yet support it.

This was only possible due to the work Cory Forsstrom put in that made our custom .stone binary interface available over FFI to C via libstone. This in turn allowed him to write a downstream patch to libarchive, the library used by debuginfod for extracting debuginfo from packages, for supporting our .stone package format.

moss error handling

The team have started work on refactoring error handling in moss with an early benefit being that download/unpacking errors will now mention the specific package that caused the error.

This is useful when troubleshooting, as it allows us to diagnose whether there is an issue with the user’s internet connection, or with the package itself on our server.

This error handling workstream will continue in the background to further improve the quality of human readable error output in our tooling.

Wider Project Updates

Website redesign

It has been clear to us for a while that the AerynOS website needs an update and this is something we have highlighted in previous blog posts. Following a review, the team has settled on using the Hugo static site generator with the Hextra theme.

Without going into a lot of detail, the Hextra theme is well maintained and has a lot of functionality built-in, meaning it is well suited to our use case.

NomadicCore has created a new branch on the dotcom repository to work on this in the background. Given the move between Static Site Generators and more generally the different content we want to present on the site, there is still a fair way to go.

If you are interested and want to help out, join our Zulip server and get to know the team.

CDN hit rate

Last year, the team set up Cloudflare as a CDN for our package repository and saw an improvement in download speeds. This was our first real experience administering Cloudflare and we left it pretty much in its default configuration.

This month, we spent a little time tweaking which parts of our repository it was caching for how long. The result has been a respectable jump in the cache hit rate.

The end result should serve to increase the consistency and speed of downloads within moss when using AerynOS from the unstable stream.

Documentation

CookieSource and NomadicCore have continued working on the documentation site over the last month.

The team have also updated documentation on the GitHub org including topics such as contributing, licensing, readmes and more. Some of these had fallen behind our latest position or had never been adequately fleshed out.

This effort is all intended to help interested parties get a better understanding of AerynOS and help them be able to contribute more quickly.

Notable changes include:

A new “Creating a new package recipe” page
Addition of a “How to submit a PR” page
The ability to view our documentation site as a single PDF
Updates to our contributing.md file across our GitHub organisation
The introduction of a policy around prohibiting the use of LLMs when engaging with AerynOS

GitHub vs Codeberg

One area of potential interest is around our use of GitHub. Over time, the team has become increasingly dissatisfied with GitHub for various reasons, including the increased push towards AI and the concern of not having control / sovereignty over our code and/or being locked out with no warning.

Looking to other solutions, Codeberg has come up as a potential alternative and one which the team has started using for internal operational documentation and personal projects, with the initial experience being generally positive.

Codeberg doesn’t do everything that GitHub can do, so if the team does look at moving its core repositories over, this would need to be planned carefully to ensure a smooth transition.

As and when we do make the transition to Codeberg, we would also like to sponsor them at the same time. An important tenet of AerynOS is to support and promote open source projects where we can. By way of example, its why we use Zulip (and previously Matrix) over something like Discord.

Sponsorship, where we are able, is another way we feel we can support the open source community and as our own sponsorship / income grows, we will be in a better position to contribute to the upstream projects on which we rely.

ISO refresh

We are releasing our newest Alpha ISO, AerynOS 2026.01, which includes the updates we’ve worked on since the start of December, and which features the 6.18.7 kernel.

The link for our 2026.01 ISO can be found on our download page.

Next Steps

We have spent the month of January taking stock of our future development direction, including short and medium term goals.

This has resulted in several nice refactors to how moss works internally, which will in turn make it significantly easier for us to implement some of the medium term features we have planned. It has also already helped us prepare a few planned features, such as better error messages and the in-progress moss fetch feature, which can be used to download packages in advance for testing purposes.

Next to that, we are steadily working towards delivering the Versioned Repos, phase2 infrastructure feature set, which is about teaching moss how to seamlessly auto-update itself across breaking on-disk moss-format changes with no user-interaction.

Currently, we are focusing on defining the necessary repository disk-layout and the format of the file at the root of the moss-format repo. This will show moss which updates need to be made in which order before the new package repository versions — containing new packages that rely on new package-management features — can be used.

Once complete, we will likely move our current repository layout to a legacy folder and test that older installs can update seamlessly in the process.

Due to the work done during 2025 on the infrastructure code-base, enabling the seamless format-upgrade process is now a short-term goal, which — when achieved — will in turn open up the ability to focus on medium term goals that were previously gated on this specific functionality.

We’re very, very excited about what this will enable us to do in the future!

Supporting the project

As mentioned in our previous blog post, our sponsorship goals in 2026 will be to continue growing our recurring sponsorships to help recover the backlog of historic project costs, and also to build up towards future hardware investments such as an x86_64-v4 capable builder.

If any hardware vendors are interested in sponsoring the project either financially or through hardware sponsorship, this would be warmly received.

Similarly, we are interested in and open to EU-based CDN sponsorship as a way of strengthening our package delivery and ISO download capabilities, and to lessen our dependency on US-based CloudFlare as a defensive measure. If you are in a position to help make this happen, we would love to hear from you.

If you are following along with our project and are in a position to support us, please consider donating via our Ko-fi page. If you wish to discuss other sponsorship details, please reach out to us at contact@aerynos.com.

Sponsor AerynOS

Thank You!

In that vein, we would also like to give Framework Computers a shout out for their generous support in the form of hardware sponsorship for project members now and in the past.

2025 in retrospect

Jan 2, 2026

NomadicCore

Comms

Ermo

Co-founder

2025 has been a year of significant change for the AerynOS project, not just in terms of development itself but also in name and in the staff working on the project.

We know many people will be following along, waiting for our beta and/or stable releases but for now we want to take a look back at 2025 and summarize what we have delivered and how that is positioning us strongly for 2026.

Camp Fire

The TL;DR summary:

We changed our name from Serpent OS to AerynOS. Aeryn is a different form of Erin, which means Ireland in Irish Gaelic.
The project founder, Ikey, stepped away from the project in mid April. A few weeks later, he sent an invitation to project co-founder and co-architect, Rune Morling (ermo), to be his Github account successor.
After these events, ermo asked the team if they would be interested in carrying on working on the project under his stewardship while waiting for Ikey to potentially make contact, and the team voted unanimously in favour of doing so.
Since then, the team has continued to deliver per the project roadmap:
- We completed the transition of all core tooling to Rust.
  - The new Rust-based infra has now delivered thousands of package builds while proving to be much more stable and robust than the previous implementation.
  - Leveraging Rust has enabled us to confidently deliver new tooling solutions like versioned repositories and a system-model approach to declaratively describe system installs.
- We improved upon the Cosmic DE offering whilst delivering new KDE Plasma and Console-based install options
- We onboarded several new staff members onto the project over the course of the year with differing strengths and interests
- We reworked our cloud hosting setup to deliver more build capabilities at lower cost base
- We renewed our media strategy through increased blog posts and engagement across social media platforms

What’s in a name

The first and biggest change we made this year was changing our name from Serpent OS to AerynOS. We are aware that, to some, this move was somewhat controversial. Our stance has been simple: A name takes on the meaning you give it.

Over time, we have continued to work to deliver on our goals for AerynOS. In return, people have become less focused on the name and more focused on what we are setting out to do, and what we are continuously delivering.

As an aside, we originally utilised AI tooling to create our logo, but this is not something we are overly comfortable with for the long run. We are looking at our options around either iterating on or creating a new “human made” logo where we can feel a lot more confident around licensing and ownership of the artwork itself.

The project founder steps away

In mid April of this year, Ikey went offline as part of a move, and subsequently never returned.

At the time, the team tried to reach out in multiple ways, however there has sadly been no reciprocation, other than ermo receiving a Github request sent from Ikey’s Github account to become Ikey’s account successor towards the end of the first week of May.

Taking into account Ikey’s experiences up until his move, we have made the assumption that his stepping back from the project was related to ongoing personal health problems for him and his family, which were compounded by significant financial strain as we understand it.

Note that, out of respect to Ikey and his family, we will not be elaborating any further on this. Needless to say, we wish him and his family the best and hope that they will have found a way through their travails.

As Ikey has left us with no way to contact him, we cannot share with you how to directly support him financially. All we can say is that you may be able to do so via his personal GH sponsors account, @ikeycode, though we stress that we have no way of knowing whether he will actually be receiving any money you decide to send that way.

Team regroup and reset

To ensure the project would be able to continue in the short and medium term, and on the basis that he felt that the promise of the tech and the underpinnings of AerynOS were too good to be left to languish and wither away, ermo asked the team whether they would like to continue working under his stewardship while we awaited news from Ikey. The team voted unanimously in favour of doing so.

Hence, for the initial couple of months, our focus was to continue delivering as a project, while allowing Ikey the space and the time to return once he’d had a chance to regroup and recover. This included letting all project sponsorship (including Ko-fi sponsorship) flow directly to Ikey.

As time progressed and the likelihood of Ikey getting in contact and returning grew dim, we needed to be in a position to actually have control of project assets and went about gaining said control in order to be able to keep the proverbial lights on.

With the exception of the original AerynOS X account, we have regained access to all relevant accounts, and have continued delivering on the project goals in the meantime.

We have done our best to not let any of this negatively impact our early alpha testers’ experience of the project, and feel that we have largely succeeded in doing so. The only real hiccup was a few months of reduced package delivery from mid April to late May, where we worked flat out to restore our ability to deliver package updates via the then under-development Rust infra port, after the old Proof-of-Concept DLang infra broke down completely.

We are now 9 months on from Ikey stepping back, and taking into account that we have not heard from him and that he explicitly designated ermo as his GH account successor, we are in the final stages of offboarding him into an inactive project alumni role.

We would like to stress that, as a project, we hold no ill will towards Ikey, and that we simply wish him the best. There is no hiding that his work was foundational to AerynOS with respect to both the tooling, the distribution itself, and the vision & ethos he promulgated for the project. We simply would not be where we are now as a project without his years of forward-looking engineering efforts.

At the same time, we think the present blog post will serve to outline why, based on our ability to deliver up until now, we are confident in our ability to continue developing AerynOS and its tooling into the future.

Infrastructure & Tooling development

Late March and April also saw the start of the porting effort of our infrastructure from DLang to Rust. The approach for this porting effort was to develop a so-called “Minimum Viable Product” (MVP) code base that could iteratively be improved.

After several months of development, in late May we started delivering packages to users again, which had all been built from the new Rust based infrastructure code base. This effort included a full rebuild of the whole recipe repo at the time to ensure ABI sanity of the newly built packages.

The development of our infrastructure didn’t stop there. Throughout the year, the code has been continually developed and expanded upon to land new features. In the last three months, the team has successfully delivered an MVP of the Versioned Repositories feature and an accompanying system-model feature as two very important examples of this.

Versioned Repositories

When fully fleshed out, Versioned Repositories will allow the team to seamlessly deliver improvements to our os-tooling (moss and boulder) in a controlled fashion. This is important as AerynOS is a rolling release distribution, meaning we need to be able to deliver improvements to users’ systems without breaking them or requiring complex manual upgrade procedures.

From a user perspective, it will open up the ability to decide which version or “update stream” of our repositories they want to use, for example opening up the capability to “lock” their system to a specific release or point in time, or eventually opening up the capability to use release-candidate or in-testing package builds that wouldn’t otherwise make their way into the “standard” AerynOS Repository.

In short, users will be able to use Versioned Repositories to easily choose a more stable and cautious repository or conversely to choose a more bleeding edge repository based on their needs and desires for their systems.

More broadly speaking, Versioned Repositories will also eventually allow for layered repositories for x86_64-v3 and -v4 packages to sit above our baseline x86_64-v2 repository where we see worthwhile improvements in building -v3 and -v4 packages. AerynOS’s approach here will not be to rebuild full repositories of -v3 and -v4 packages, but instead to overlay these packages on top of the x86_64-v2 repository only where there are verifiable gains to be found by doing so.

In a similar vein, this development workstream will eventually enable us to build repositories targeting ARM and RISC-V instruction sets. This will be a ways down the road and the team would need to acquire ARM and RISC-V builders at the appropriate time, but we want to foreshadow this to highlight that our current up front planning of our infrastructure development roadmap will enable very flexible use cases down the road.

system-model

The system-model approach we are currently developing, is at its heart a declarative definition of the packages comprising a system and where they come from. The team has just landed the first iteration of this to users, and this — along with Versioned Repositories — will continue being a development focus into 2026. The system-model approach declares the repositories & packages a user wants to sync their system against. This is currently an opt-in solution with moss now automagically generating a system-model.kdl file each time a moss transaction successfully completes.

The team has multiple future ideas of how this system-model can be used with a few examples including:

Sharing your system-model.kdl file for reproducing your system state for issue tracking and resolution
Using your system-model.kdl file as part of your install to get your system exactly how you want it at first boot
Changing your system by updating packages and then reverting back to your desired system-model on next sync
Being able to pin to older Versioned Repositories (for stability’s sake) or if issues occur on the latest rolling Versioned Repository until it’s fixed

All of these features, particularly where users could potentially face issues with their installs, are complemented by our offline rollback feature, which gives the ability to manually go back to either of 5 earlier states from the bootloader. This will ensure that users have both offline and online protections to help them recover from a misbehaving system.

Team organisation

We kicked off the year with looking for support in 4 key areas:

Community management
Documentation
Translation
Web development

As the year has gone on, NomadicCore, Jplatte, bhh32 and CookieSource have joined the team bringing with them a vast amount of experience and enthusiasm.

It’s fair to say that community management and documentation development have taken significant steps forward. Web development is a work in progress area with plans being formulated for an eventual re-write of our main website, which frankly needs a lot of work. Translation efforts have been paused (or more accurately never started). The team is prioritising building out the core infrastructure and tooling first, with distro polish taking a back step until the tooling is ready for it.

Website development

The current state of our website leaves a lot to be desired. We are aware that it needs a full refresh, not only for presentation, but also to include important information as a bare minimum.

The team are looking at options such as Zola and Hugo as alternative Static Site Generators however this is a low priority workstream for now. We are open if anyone has experience in website design and would like to work with the team to develop our website into something to be proud of.

Distro updates

DE and WM updates

Over the course of 2025, the team have continued to keep Gnome packaging updated whilst improving upon Cosmic through to its latest stable release. In addition, Reilly Brogan was able to get KDE Plasma packaged up, which expanded our recipe repo size by almost 50%.

All three desktop environments have proved to be stable, though it needs to be said that Cosmic overall has a way to go for true maturity.

We also decided to include a terminal-only option in our installer, which can be used to install and tweak e.g. Sway, Niri (and soon MangoWC) based custom Wayland environments almost from scratch.

For now, we don’t have any concrete plans to package up additional DEs or WMs for AerynOS. There are “enough” options for early alpha testers and our core team to work from, and any additional options would simply add an extra maintenance burden on the team that would in turn detract from necessary tooling, infra and overall feature and integration work.

In terms of future potential, it’s worth mentioning that AerynOS is first and foremost a forward-looking, Wayland-aligned project, which means that Wayland session support is a minimum requirement for any future DE or WM additions to the repository.

Installer

At the start of this year, Ikey shared initial screenshots of a GUI based installer that would eventually replace our text based installer as the primary installation method. With Ikey stepping back from the project, the development of this GUI installer took a back seat to allow the team to focus on higher-priority items such as delivering working infra.

Recently, one of our new staff members, Bryan Hyland, began working in the background on a new GUI based installer to eventually fill this role.

To put it simply, the idea is to eventually use the newly developed system-model approach to define the packages that would be installed per DE / WM and also allow users to use their own custom system-model.kdl files as the basis of an installation.

Sponsorship

With Ikey stepping away earlier this year, for a period of time, it was unclear whether he would be returning or not. As the months passed by, the prospect of his return grew less and less likely. Therefore, as covered earlier in this blog post, we are now in the latter stages of formally retiring Ikey from the project.

As mentioned previously, at the start of this year, the sponsorship was set up in such a way that all funds were sent directly to Ikey in order to enable him to afford to spend time working on the project and its underpinnings.

However, in light of Ikey’s continued absence, the team has taken control of the AerynOS Ko-fi account and redirected the funds from it so they can be used to cover ongoing project running costs.

All other references to sponsorship routes have been removed, leaving Ko-fi as our only official sponsorship medium.

Due to the way Ko-fi sponsorships work, at the point of transition, we had to cancel all existing sponsorships. We are not yet at the point where we are at parity in sponsorship income compared to before they were cancelled, however, we are fully covering direct project costs.

Part of this ability to cover direct project costs can be attributed to how we are spending our money and how we are utilizing as much of our own internal resources (such as ermo’s machines) for build capacity. We have also been developing our infrastructure code base to make it as flexible as possible in terms of adding new builders and scheduling builds efficiently based on individual builder resources.

Our sponsorship goals in 2026 will be to continue growing our recurring sponsorships to help cover the backlog of historic project costs, and also to build up towards future hardware investments such as an x86_64-v4 capable builder.

If any hardware vendors are interested in sponsoring the project either financially or through hardware sponsorship, this would be warmly received.

Similarly, we are interested in and open to CDN sponsorship as a way of strengthening our package delivery and ISO download capabilities. If you are in a position to help make this happen, we would love to hear from you.

If you wish to discuss sponsorship details, please reach out to us at contact@aerynos.com.

Sponsor AerynOS

Going into 2026

For now, there is a deliberate, continued focus on our os-tooling and infrastructure code bases, somewhat at the expense of delivering a fully featured Linux distribution for end users.

As time goes on, the goal is for this balance to naturally shift as our code bases become more capable and featureful.

Over time, and as logical consequence of this shift, we hope to be able to onboard more contributors to help us scale out the recipe repo, once the tooling is in a state where it conveniently allows for this.

If any of this has piqued your interest and made you want to contribute, we invite you to join us over at our Zulip chat platform.

Hope to see you there!

November + early December 2025 project update

Dec 9, 2025

NomadicCore

Comms

Ermo

Co-founder

AerynOS: November + early December 2025 project update

As we near the end of the year, the team has been reflecting on how best to position itself going into 2026. Over the course of November and early December, the team has made a number of changes, some of which have been public facing which you may have already seen if you’ve been following our socials.

On the public side, the two most visible changes has been our transition away from Matrix to Zulip for our instant messaging chat platform, and our transition to netcup for our server requirements. In the background, the team has also changed email hosting providers, however no one would really have noticed any difference there.

Packaging updates are progressing nicely with a nice rhythm for Cosmic and KDE stack updates, which along with fixes to reported issues is all helping to improve the overall experience for the people testing and checking out AerynOS.

On the infrastructure side of things, we have landed simple auto-pruning logic into our Vessel repository manager. This sort of automation ensures that we keep our repository lean and mean.

Things are progressing nicely in terms of getting phase 1 of our Versioned Repositories infra feature ready for production use. It was designed to from the outset to mesh nicely with our Moss system-model feature. Internal testing on a private infra instance has been running smoothly with these new features so far.

What’s new in the distro

Package / stack updates for this iteration include:

COSMIC Beta9
GNOME 49.2
KDE Frameworks 6.20.0
KDE Gear 25.08.2
KDE Plasma 6.5.4
bash 5.3.8
buildah: Add at v1.42.2
docker 29.0.4
ffmpeg 8.0.1
firefox 146.0
gamemode: Add at 1.8.2
linux 6.17.10
llvm 21.1.7
mesa 25.3.1 (with Vulkan anti-lag support enabled)
openvpn 2.6.17
prism-launcher 9.4
scx-scheds 1.0.17
sudo-rs 0.2.10
uutils-coreutils 0.4.0
vim 9.1.1896
vscodium v1.106.37943
wine 10.19
xwayland-satellite 0.8
zed 0.210.4
zlib-ng: 2.3.2

… along with sundry additions and updates.

Desktop Updates

Cosmic

Cosmic Install

Our Cosmic packaging team have automated much of the update process based around the more frequent git tags that the System76 team are now publishing for the Cosmic Beta phase. As such, we are more quickly able to push updates out to our volatile repository for eventual availability in our unstable repository.

As an aside, we believe we have fixed the issue for Cosmic DE sessions where USB devices were not automatically showing up, by adding gvfs as part of the Cosmic pkgset. This is a fairly important usability feature so great to have it fixed.

We have also fixed the issue we reported in our October Project Update blog post about sudo-rs not functioning correctly on Cosmic Terminal and Ptyxis in our Cosmic session.

Overall, Cosmic is progressing nicely in general and we are polishing the experience on AerynOS in lockstep with Cosmic Beta9 having landed in our repositories.

Additional details about Cosmic can be found on System76’s website.

Gnome

Gnome Install

As usual, there isn’t really much to say wrt. GNOME. The team has updated it to 49.2 and it is running nicely as expected.

KDE Plasma

KDE Install

KDE Plasma has been updated to 6.5.4, KDE Frameworks to 6.20.0.

As part of these updates, Reilly added a few custom patches to better support monitors with high refresh rates.

Infrastructure Updates

Auto-pruning in vessel

The new auto-pruning feature in our Vessel repository manager service means that our infra will periodically (currently daily) review what packages are present on our server storage versus which packages are reachable via a repository index.

If packages are no longer reachable via a repository index, they are considered surplus to requirements and pruned.

Automating this workload ensures that our storage doesn’t fill up and that our repository looks after itself over time, freeing the team from encountering unpleasant surprises in the form of having to take corrective action as storage fills up.

Wider Project Updates

Transition to netcup for our server requirements

As part of a wider review of our requirements, the team looked at how we were using our server and whether it was “right-sized” for where we are as a project. The project has shifted over the last 7 months to more heavily using ermo’s four private servers, meaning the “main” AerynOS build server hasn’t been utilised as effectively. Therefore, the team took the decision to sunset this server and look for a server better specified for repo hosting and infra coordination. To this end, we settled on a netcup “root server” based in Germany (where the old server was based in Finland). The netcup server has a 2.5Gbit NIC and has — for European users at least — yielded significantly faster download speeds.

It is still early days, however we are very pleased with our transition to netcup. Their support offering during our initial set up phase was both responsive and very helpful.

Transition from Matrix to Zulip

We want to preface this by saying we love what Matrix is doing by providing a federated open source instant messaging chat solution for users.

However, for AerynOS specifically, we have been having a few issues with our matrix space, partially in administration and partially around federation and delays in messaging (or in some cases, some users not able to see messages from other users in one room but they could in others).

As a team, we looked at what other options were available, and considered what our requirements were before eventually deciding to try Zulip on for size.

After trying it internally for some time, it quickly became clear that it offers great instant messaging capabilities like Matrix whilst also offering many other features such as asynchronous conversations (threads/topics) that allow users, contributors and staff to better focus on the various project conversations that fall in their sphere of interest or responsibility.

Like Matrix, Zulip is an Open Source Project and it also has hundreds of integrations that help elevate the overall User and Moderator experience.

Initial feedback from our wider user base has been overwhelmingly positive and we are excited to continue on this journey!

For transparency, Zulip has a policy of supporting Open Source Projects, and has generously sponsored the AerynOS project with a standard cloud server instance at no cost.

Please feel free to join our Zulip server and get to know the community that is building up around AerynOS.

Transition to Migadu for emails

The team had a look at the options available in the market for emails and decided to make a transition to Migadu’s offering, after Migadu offered us an OSS project discount.

While it does require a little more set up than other providers out there, it has the substantial benefit that you can set up as many email addresses on your account as you want or need, without this impacting the cost base.

This will undoubtedly prove helpful for the project going forward.

Logo / Branding / Website progress

The team has been looking at our branding and reviewing our requirements. It’s safe to say that if we didn’t have to update these, we would keep them as is as there are other higher priority activities.

However, if we take the logo itself, it was created using AI tools, so doesn’t really line up with our project ethos and does leave questions over licensing and ownership.

As such, we have been looking at what we might want from a logo and how it can best represent AerynOS. In keeping with this, we have also been looking at our wider branding including the colour schemes that we use.

Given that the naming conventions for AerynOS’s infrastructure and tooling projects all link to nature, we are leaning towards a nature oriented colour scheme going forward.

Lastly, we have been looking at our website which is in dire need of work. We currently use Astro for the website, however we have found that to be a complex solution. This is especially true as none of the current team / wider contributor base has much experience with Astro. We have been looking at the field of Static Site Generators (SSGs) and have shortlisted it down to Zola and Hugo, both of which are meant to be much easier to work with than Astro.

Zola is the preferred option with it being Rust based, however it’s fair to say that it is a much smaller project so doesn’t have anywhere near as extensive a theme library to choose from. Hugo is a larger / more mature project with greater theming, but is tied to Go which would create an additional maintenance burden outside of the team’s core focus.

Outside of existing themes, one option is to create (and subsequently maintain) a brand new theme, but this requires expertise and would place a burden on the team to ensure compatibility and that everything is updated within the template as time progresses.

If you have experience in website design and would like to help create / shape a redesign for our main website, then please join us on our Zulip server so that we can discuss how to move things forward.

Documentation website progress

With new contributors coming on board, our documentation website has seen significant updates in both content and in presentation. Whilst still a work-in-progress project, it is becoming more usable with less gaps. The team and wider contributor base continues to push updates, the big outstanding one being how to create new package recipes from scratch.

Once these gaps are filled in, it will become an iterative exercise to make sure our documentation stays updated and potentially go down into further detail where feedback suggests this is necessary.

Download choices and BitTorrent

Another area we have been reviewing is how we serve ISO downloads to our users. We currently only offer a direct download via our package server behind a CDN. As a small operation, this has suited us well enough, however users in certain locations don’t necessarily benefit from this solution.

We have had the BitTorrent option on the board for a long while now, and over the last couple of weeks, we silently added a torrent link for the AerynOS 2025.10 ISO. With the 2025.12 ISO we are taking this approach forward. This will have a secondary benefit of reducing the load on our server / CDN at the point of ISO releases, as bandwidth is naturally distributed due to the torrenting approach. We will continue to offer direct downloads for anyone who prefers this option.

ISO refresh

We are releasing our newest Alpha ISO, AerynOS 2025.12, which includes the updates we’ve worked on during the month of November and the first week of December, and which features the 6.17.10 kernel at the time of upload.

As usual, this is a Live GNOME ISO that contains our Alpha/PoC lichen installer. Hence, installing AerynOS requires a network connection over which the latest pkgsets can be downloaded and subsequently installed onto a dedicated AerynOS target hard drive.

The link for our 2025.12 ISO can be found at our download page.

Next Steps

The rest of december is dedicated to bringing phase 1 of the Versioned Repositories feature PR here and its associated Moss system-model companion feature PR here up to snuff and landed for the general public.

As mentioned in the introduction, these PRs are already in testing, and merely need some final bits and bobs wrapped up before they can be landed and put in production.

The fully realised Versioned Repositories feature will enable us to deploy new Boulder and Moss features in a seamless fashion. It will allow us to introduce breaking code and on-disk format changes, that would otherwise cause installed systems to require manual intervention for them to continue to receive updates.

Once Versioned Repositories are in place, the goal is that users will be able to simply update and sync their system as normal via the sudo moss sync -u command.

With this:

Users will be upgraded to the new versions of Moss that uses a new repository format, without having to pay special attention.
It will enable AerynOS to iteratively expand the capability of Moss and Boulder on existing systems without breaking user systems in the process.

Going further, we consider Versioned Repositories a pre-requisite for what we call “try-builds” and eventually for multi-arch support.

Automated try-builds denotes the process whereby the infrastructure discovers an update to the upstream source repository of a package, attempts to auto-update the recipe and then attempts to build the updated package recipe in question.
We think this will be a useful tool for contributors as it will automate some of the packaging tedium related to simple package version updates. It will also help enable automated regression testing and build flag optimisation in a future workstream.
Included under the multi-arch umbrella is our ability to target ARM, RISC-V, and different x86 architecture levels such as x86-64-v3 and v4.

Supporting the project

Following the October blog post, we had a flurry of new donors whom we want to thank for supporting our project. Their support is greatly appreciated, especially given the global cost of living crisis leaving less money in peoples pockets each month. Your support means a lot to everyone on the project as it shows the faith you have in the work we are doing!

2025 has been a significant year for the project, with the team having landed our new Rust based infrastructure, repository-wide rebuilds along with landing KDE and significantly improving our Cosmic offering whilst also landed new features into our tooling. We see our Versioned Repository work continuing into 2026, and this will eventually help lift AerynOS into becoming a serious offering in the Linux distribution space. Onwards and upwards!

If you are following along with our project and are in a position to support us, please consider donating via our Ko-fi page.

October 2025 project update

Oct 31, 2025

NomadicCore

Comms

Ermo

Co-founder

We are firmly in the final quarter of 2025 and what a year it’s been so far. In our last blog post at the end of September, we mentioned that we were delaying the release of our next ISO into October to give our Gnome 49 stack (and the wider extensions ecosystem) more time to mature.

Coming into the final week of October, the team made the decision to transition from clang’s libc++ to GNU libstdc++ and work through the associated rebuild requirements across our repository.

We also mentioned in our September blog post that we had reached the point of having stable build infrastructure. However, over the course of October, we had an old problem re-appear, which proved somewhat vexing to solve initially.

After the issue was debugged, and a fixed version of boulder was deployed to the build infrastructure, we used the transition back to libstdc++, which included hundreds and hundreds of rebuilds, and the landing of the KDE Plasma 6.5 stack to verify that that we have put this particular issue behind us.

In addition to the build infrastructure related boulder fixes, the team has also found the time to hash out a design for, and prepare an early prototype PR of, the moss declarative system-model feature, as well as landing a few small user experience improvements and correctness fixes to both moss and boulder.

With that all said and done, we are ready with our 2025.10 GNOME Live ISO refresh for you to enjoy along with this blog post.

Whats new in the distro

Package / stack updates for this month include:

KDE Plasma 6.5.1
Cosmic Beta3
Gnome 49.1
Linux 6.16.12
Mesa 25.2.5
KDE Frameworks 6.19
KDE Gear 25.08.2
llvm 21.1.4
uutils-coreutils 0.3.0
sudo-rs 0.2.9
ffmpeg 8.0
pipewire 1.4.9
Wine 10.17
nodejs 22.21.0
zed 0.206.6
virt-manager 5.1.0
bash 5.3.3
scx-scheds 1.0.16
vim 9.1.1829
systemd 257.10
uv 0.9.5
perl-module-build: Add at v0.4234
libdovi: Add at v3.3.2
openconnect: Add a v9.12

… along with sundry additions and updates.

Desktop Updates

Cosmic

Our Cosmic environment has seen additional testers and contributors coming on-board to support Bryan who we highlighted in the previous blog post. Our Cosmic packaging team’s approach is to use System76’s repo-release repository rather than waiting for git tags. The team are running roughly on a bi-weekly update cycle keeping us up to date with Cosmic’s development where we had previously fallen behind. At the time of this blog post, our versions are tagged at Cosmic Beta3, however the code is somewhere between Cosmic Beta3 and Beta4, and Beta4 is being worked on as we write this.

Known issues

We are currently experiencing an issue with sudo-rs in Cosmic Terminal and the GNOME Ptyxis terminal emulator in our Cosmic session, however the issue doesn’t present with e.g. the Kitty terminal emulator, which can therefore be used as a workaround for the issue. We are tracking this issue in our bug tracker.

Additional details about Cosmic can be found on System76’s website.

Gnome

The Gnome packaging team has updated our Gnome environment to 49.1. The team has also expanded the number of available Gnome packages in our repository, and we now include Showtime and Gnome-contacts in our gnome pkgsets accordingly. For clarity, these were added in September but were not mentioned in our previous blog post.

Gnome continues to work very well on AerynOS and remains as our default live installation environment.

KDE Plasma

Reilly Brogan has done a fantastic job landing KDE Plasma 6.5, KDE Gear 25.08.2 and KDE Frameworks 6.19.0 into our repository over the last month. The overall KDE Plasma experience continues to improve with users providing largely very positive feedback on its performance and fluidity on AerynOS.

Over the last couple of months, KDE Plasma has now also been promoted as a recommended installation option alongside Gnome.

Distribution Updates

Switching back to GNU libstdc++ from LLVM libcxx

The team decided to switch back to the GNU libstdc++ library from the LLVM libcxx C++ library as a defensive measure. This has resulted in us having to carry fewer patches across the stack, and has resolved a few bugs as a bonus. In particular, this has squashed an annoying bug related to the Firefox Widevine DRM plugin that in turn previously made certain video conferencing tools crash.

Packaging License Matching

New this month, when packagers use the boulder recipe new invocation to create a new recipe from an upstream source archive, boulder will attempt to identify the applicable license of the package, and add it to the autogenerated skeleton stone.yaml recipe file.

This is a nice quality of life and user experience feature for our packagers. The AerynOS ethos is to try to help lessen the burden on packagers by automating things that are simple in nature, yet time-consuming. This in turn, we hope, will enable packagers to spend more of their time on the parts of the packaging process that genuinely benefit from a human touch in terms of building a distro that they and users enjoy.

Parallelize the `moss state verify` command

Until recently, when employing the moss state verify command to ensure the integrity of all moss states, boulder was limited to running on a single thread. The team updated the moss state verify command to run certain tasks in parallel (using rayon) which has significantly reduced the time to complete the verification process.

The moss state verify command currently does not have any indication of progress such as a progress bar. As such, a user may be unsure if their system is frozen. Whilst parallelizing performance-sensitive aspects of this task significantly reduces the time to completion, we have an issue raised to add a progress bar to the command for an improved user experience.

Infrastructure Stabilization

As it turned out, the issue we kept seeing (and have kept seeing since May) where build tasks would hang for no apparent reason, was actually related to the boulder threading code leaving threads running at the point where we called the clone2 syscall to enter a user-namespace container for build isolation purposes — fearless concurrency indeed…

However, as these issues only manifested for us when boulder was invoked by the build infrastructure, and even then only rarely, it was not a problem we could trigger on demand.

After some head-scratching and debugging sessions where we attached debuggers to live builds that had happened to hang, we finally found the root cause, and in turn were able to simplify our threading code significantly by always using a dedicated, explicit runtime that gets automatically shut down once the parallel (rayon) or threaded (tokio) work unit goes out of scope.

This in turn now ensures that all threads have been shut down as required, before entering the cloned user-namespace container.

Massive thanks to Cory Forsstrom for the work he put into solving this problem very elegantly over a couple of Pull Requests.

moss `system-model` prototype feature

The system-model feature is inspired by the feature of the same name from the Conary system and package manager. However, in practice, it will likely end up more like a hybrid of the Conary feature and e.g. the simpler and more limited Gentoo (and FreeBSD) ‘world’ set.

We are still hoping to be able to, when the time comes, offer versioned system-models that match our planned versioned repository work.

The goal of the moss system-model feature is that it will eventually enable us to define, and switch between, system installs declaratively and seamlessly. This feature is slated to be suitable for use on both live systems, recovery initramfs environments, and for installing new systems.

It is important to note that the system-model feature will be an opt-in feature, and that it will continue to be possible to use moss in an imperative fashion, where the system state is defined by the sum total of historical moss operations executed on the command line.

The design we have settled on makes it possible in the future to import (“resolve”) a declarative system-model to an imperatively defined system, just as it will be possible in the future to export (“derive”) a declarative system-model of any given imperatively created moss system state.

We hope that, as we evolve and flesh out the system-model feature work, this will give both users and system integrators the flexibility they need to define their systems in ways that matches their requirements and preferences.

Wider Project updates

Donations

During October, we have made changes to our donation accounts and updated our site accordingly. Most importantly, the primary way to now provide donations to AerynOS is via our Ko-fi page.

Due to the way Ko-Fi works, we have had to cancel all existing recurring donations as the money would not automatically flow to our new accounts. We have sent messages out to our donors and are hoping that they will want to sign-up again to provide continued donations towards the project.

All donations received by AerynOS are going towards paying our running costs, reimbursing ermo for his project hardware investments, and building a buffer for acquiring new hardware once the current hardware reaches its End of Life.

Taken together, these three budget items result in a target income of around €500 per month, with a third of that projected to go to each of the outlined items.

Due to the cancellations mentioned above, we are currently down to €25 of new/committed monthly recurring donations.

Rest assured however, that this will not impact the project’s long term viability as we have the ability as a team to cover these costs. However, if you are able to provide any sort of donation, it would be greatly appreciated.

As a side note, we have also moved away from taking donations in USD by default and switched to EUR as this is the currency we are primarily spending in for our project costs. This does not preclude us from accepting USD based donations however.

Presenting the 2025.10 ISO refresh

Whilst we provide a bootable ISO image with a live GNOME environment, its main purpose is to serve as a vehicle for our ‘lichen’ installer which is able to install all of our ‘GNOME’, ‘KDE Plasma’, ‘Cosmic’ and ‘Terminal Only’ versions of AerynOS. Given that ‘lichen’ is a net installer, it will always pull the latest packages from our repository at the time of installation and as such, it requires an active internet connection. This means there is less of a need for regular ISO updates, as we have carefully designed our installation routine and our pkgsets to not be tightly coupled to the packages that happened to be available at the time of the live ISO creation.

Installer preview

That said, with the addition of some significant updates in the Gnome stack, we felt it was a good opportunity to land a new live ISO for you all to try out. The link for our 2025.10 ISO can be found at our download page.

Next steps

Going into November, the team will continue to execute on our development plans related to improving our infrastructure in terms of both capability and user experience, just as we will be continuing to work on improving our moss and boulder tools via their planned workstreams, particularly the moss system-model related workstream.

On a final note, we are progressing on the documentation front, and work is steadily trickling into our documentation site. Thank you to everyone who has contributed!

If you are interested in our work or generally want to hang out, the best place to join us is in our Matrix space. Alternatively, our Forums can be found on GitHub

September 2025 project update

Oct 1, 2025

NomadicCore

Comms

Ermo

Co-founder

As we reach the end of September, it’s time for another monthly update. For this month, we are deferring our technical blog post into October but wanted to provide wider project updates.

Kernels

AerynOS is currently running on 6.16.8. While we are aware that Linux 6.17 was released on 28th September 2025, the AerynOS team takes the view that for larger software stacks, the wise thing to do is to hold back for a few point releases after a major version update to ensure there are no unexpected issues.

Even though AerynOS is a rolling release distribution, we don’t aim to be on the bleeding edge just for the sake of being on the bleeding edge.

Gnome 49

The team updated to Gnome 49 within a couple of weeks following its release meaning it’s now available in our repository. Given the way Gnome updates and how extensions then follow suit, we are aware that many extensions have not yet been updated for compatibility with Gnome 49.

As such, we are holding back our next live installer ISO release until a larger number of extensions have had time to go through their own update cycles.

Infra stabilization

2025 has seen a massive lift in our infra code with the rewrite to Rust taking place in Q2, and its stabilization being pursued from the end of Q2 and up until now.

In this period, the team has been incrementally refining the code, and resolved bugs as they have arisen. This is work that average users will not have seen or been aware of as it only affects workflows behind the scenes. However, this has been extremely important work that has increasingly allowed us to become more confident in our new infra foundations.

We are now at the point where we are confident in calling our new infrastructure “stable”, with no bugs having presented themselves in several weeks and hundreds of packages built.

It cannot be overstated how much better the state of our infra is compared to the start of the year. In turn, this now allows us to focus our efforts in other areas of our technology stack to similarly work through bugs, implement features and generally refine our code base.

Cosmic maintainership

A couple of months ago we had a new contributor, Bryan Hyland, join us and begin working on the Cosmic stack. Bryan’s contributions have been invaluable to the team and we thank him for his engagement.

Unfortunately, due to personal reasons, he is having to take a temporary step back from the project. He has shared his methodology for providing updates to the Cosmic packages and we already have willing contributors from our community picking up the in-flight tasks.

We know he will be back once his circumstances improve, but in the meantime, if you are curious about supporting Cosmic on AerynOS, please do get in touch in our Matrix packaging channel as it’s always good to have several people capable of maintaining each of our DE stacks.

Web development

One area in which we often receive feedback is our website. Given we are in an alpha state, it has not been our highest priority to improve it. We do have a couple of interested contributors wanting to work on our website to help improve the overall look and feel of the project, though.

Hence, if you are interested in website design and want to lend a hand, please feel free to join our Matrix webdev channel as we will likely begin work on this area in parallel to our core development work.

Documentation

Similarly to our web development work, we are seeing increasing numbers of contributors willing and able to help with our documentation. Outside of our core tooling development, this is arguably our second most important workstream and another area in which we often receive feedback.

If you are interested in supporting our documentation drive, feel free to join us in our Matrix documentation channel.

Holidays

NomadicCore has been leading our social media presence for the last few months. Someone (unwisely) gave him permission to go on holiday for the next three weeks, which means that our social media presence is likely to be less active than usual in the interim.

Please do not take this as any indication of the project slowing down. If anything, we have an increasing number of developers working on different areas of our tooling stack and are seeing our development and packaging activity picking up.

October plans

We hope to release out next ISO in October, once:

We have landed the Cosmic Beta packages and have had a few weeks to shake out any bugs
Gnome extensions have had more time to update for compatibility with Gnome 49
We are a few point releases into the Linux 6.17.x kernel cycle and there are no confirmed bugs we need to address

We are also aware of the planned release of KDE Plasma 6.5 on the 21st of October 2025. Similarly to Linux 6.17.x, we are not planning / trying to be first out of the gate with a large stack update. This will likely land at a later point after a few weeks / a point release or two.

That’s it for this monthly roundup. Thanks for reading, and we hope to see you around in our community.

August 2025 project update

Aug 31, 2025

NomadicCore

Comms

Ermo

Co-founder

Closing out August 2025, we are proud to announce our third release of the year. This release follows an intense development period where the team has reevaluated its priorities / timelines and refocused efforts on “delivering core Linux distribution tooling that will simplify our ability to scale out over time”.

We have documented some of our progress in our last two blog posts and spent the last two months further progressing towards these goals. We have implemented a basic version of virtual packages (Package Sets), continued our hardware (and VM) enablement efforts and have selectively been growing our repository where we feel it’s beneficial to our users.

What’s new Distro-wise

Whilst not an exhaustive list, some of the top line repository updates include:

GNOME 48.4
Plasma 6.4.4
Sway 1.11
Cosmic Alpha 7
Linux 6.15.11
Mesa 25.2.1
LLVM 20.1.8
uutils-coreutils 0.1.0
sudo-rs 0.2.8
ffmpeg 7.1.1
fastfetch 2.51.1 (adds AerynOS logo)
Waydroid: Add at 1.5.4
openvpn: Add at 2.6.14
protontricks: Add at 1.13.0
winetricks: Add at 20250102

In addition, we fixed a subtle issue with our PATH configuration that mostly affected our console logins. With this fix, we have made our login experience fully stateless. We have also enabled sulogin for a single-user root shell to diagnose and repair boot failures.

Package sets

AerynOS is transitioning to a package set model for core packages installed on a user’s system.

Package sets are a collection of packages that are related or used together for a specific purpose. In AerynOS, they are used for consolidating our base system packages and for each of our offered Desktop Environments / Window Managers.

Each Desktop Environment offered by AerynOS has an associated package set (usually “recommended”). Depending on the environment, we may optionally offer a “minimal” and/or “full” solution with less or more packages to better suit our users requirements.

The package set model we have implemented is a stepping stone technology, not the final solution we are looking to implement. It introduces the basic premise of virtual sets of packages and is a precursor to our “system-model” work that will allow for exact reproduction of a user’s installed system.

Our installation experience with lichen

To fully integrate the new package set model within the AerynOS project, we have adapted lichen to install Desktop Environments based on the associated package set. The TUI (Text User Interface) prompts guide a user to select which DE they want, and depending on the DE, the version we have curated for the install (recommended for the DE’s, minimal for sway) with moss determining what dependencies are required for a successful install based on that package set.

Lichen is a network installer meaning that it downloads the latest set of packages from the AerynOS repository for installation. Tweaks to our package sets don’t necessarily require a new ISO. Whilst the “live environment” may not be up to date, the user will get a fully up-to-date installation without requiring a post-install update step.

lichen usability updates

In its current state, lichen requires that a user pre-format their disk prior to attempting to install AerynOS. Given its nature of being a network installer, lichen also needs an active internet connection to complete an install. Before, if either of these prerequisites were not met, the installer would output a very unhelpful Rust code error. We have added the appropriate prompts to guide users to ensure they have an active internet connection and to remind them that they need to pre-format their disk.

Whilst we could fix the pre-format requirement, a conscious decision has been made to keep this “anti-usability” feature as a barrier to entry for “beginner Linux users”. This may seem very counter intuitive, however whilst we are in an alpha state, we need to be careful not to position ourselves as a “beginner Linux distro” that could attract many support requests. We need to focus our time on developing our tooling and infrastructure.

Do note that in time, we will fix this issue and become more beginner friendly!

Virtual Machine usage and hardware enablement

Significant work has taken place to enable virtual machine support, both with AerynOS as the host and as the guest.

For host support, we have packaged virt-manager into our repository. The team utilised VMs for testing package set configurations and other potential breaking changes over the last month. It has sped up our development pace as VMs are more disposable by their nature.

For guest support, we have enabled a significant amount of hardware in our kernel and specifically enabled HyperV (based on user requests). We are also actively seeking user feedback for other VM environments. If you try AerynOS in other VM solutions and have any problems, you can report those issues here.

By better supporting both VM host and guest scenarios, we hope to unblock potential contributors from exploring our distribution and tooling.

Please note that we still classify AerynOS as alpha status project. We do not recommend anyone install it on hardware required for “production environments”! Our key goal is to enable the hardware / software that developers and contributors may need to make the transition to and/or explore AerynOS.

Other updates

Reilly Brogan has added scx-scheds to our repository and set scx_flash as our default scheduler. Scx_flash is a scheduler that is focused on ensuring fairness among tasks and performance predictability. More details about it can be found on the sched-ext website.

For our use case, this is helpful as it allows an AerynOS system to still be responsive whilst heavy tasks such as building packages are happening in the background.

Whilst this has been implemented in our ISO (ie. for new installs), it will not retroactively apply for existing installs. If you want to transition to this, you will need to install scx-scheds.

What’s new DE-wise

There has been considerable effort around our DE provision this year, some of which is yet to materialise. We are happy to report that we are now offering KDE Plasma in our repository and it will be installable from our ISOs going forwards.

In addition, we have created a console only installation option for more advanced users and for our own testing purposes.

KDE Plasma

It’s fair to say that KDE Plasma has been one of the biggest requests we have received and we are happy to report that it is now available in our repository. Reilly Brogan has done a fantastic job packaging up the latest 6.4.4 version into our repository with both sddm and plasma-login-manager offered as login managers.

A running bug tracker can be found here to report any issues. Please do test it out and help us find and resolve any undocumented bugs that remain.

We are far enough in our bring up and testing process for KDE Plasma that we are comfortable offering it as an installation option in our ISO’s going forwards.

Cosmic

Within the AerynOS repository, Cosmic DE sits at the release tag of Alpha 7. Given the significant pace of development of Cosmic, we are looking to move to a more frequent update cycle to incorporate bug fixes and new feature releases tracking System76’s repo-release repository. Whilst still in flux, we are looking at a bi-weekly update frequency to balance maintainer burden with keeping the DE up to date. The first of these updates bring Cosmic DE packages to their most recent versions will land in the AerynOS repository in the coming days.

Following recent engagement efforts, we have been seeing new users and contributors checking out AerynOS and specifically our Cosmic spin. Through this additional testing, we are seeing more active engagement on keeping Cosmic updated and bugs squashed. Given its alpha status, we don’t expect a fully bug free user experience so please bear this in mind if choosing Cosmic. We are classifying our Cosmic spin as a “Technical Preview” given that both Cosmic and AerynOS are currently in alpha status.

Moving forward, we are looking to package up more of the available Cosmic applets and generally polish the Cosmic experience on AerynOS. If you have an interest in packaging and specifically in the Cosmic Desktop, feel free to get in touch as we could always use more support in testing and improving upon our DE experience.

Sway

We have had Sway within our repository since early last year but did not really highlight its inclusion. Sway has been updated to v1.11 and we have also included Waybar and a few other packages to make ricing Sway a nicer experience on AerynOS.

We debated including Sway as an installable option from our ISO, however we have made the decision to defer this for a future release. We have created an initial “minimal” package set for Sway which includes the bare minimum to get started on ricing it. However, it has not yet been validated to a level that we are comfortable shipping it to users, even in our alpha state.

It remains in our repository and will continue being worked on as we progress into the second half of the year. In time, we would also like to develop a couple of pre-configured Sway configs as additional package sets so users not already familiar with Sway can jump right in without having as much background experience.

Console-only

In addition to the other environments, we have created a very minimal package set that will boot into the Linux console without any Desktop Environment. Users can use this console-only option as a starting point to configure a system install exactly to their requirements with only the packages they wish to have included or as the basis for a new DE/WM to be included within AerynOS.

Given the way we have layered DE/WM package sets over our base package sets, a user is able to install any of the other DE/WM options on top of this console only solution. This has been very helpful for the AerynOS team in testing our different offerings.

Gnome

Other than updating to the latest 48.4 version, there is not else much to say on GNOME. To its credit, it has been working smoothly on AerynOS so there has not been any major work required.

It remains our default option for our ISO live environment and should only require updating to new versions as they release. If you do happen to discover an “undocumented feature”, please feel free to report it here.

What’s new tooling-wise

Moss state diff

Joey Riches has delivered a new command “moss state diff” which allows users to check the differences between two states. This is very useful when you want to revert back to an older state.

Each state is identified as a number and there was no way to understand what a state had. With this new command, you will be able to inspect the state and see if it’s the right state that you are looking for.

The command requires you provide two state numbers and it will return back the differences in package versions and new / removed packages between the two states you have specified.

Moss search-file

We have also landed another new moss command “moss search-file”. This works similar to “moss search” however it works at the file level and enables users to ask moss to which package any given installed file under /usr belongs.

What we are working on

Packagekit and appstream generator work

Joey Riches has picked up and continued his previous packagekit integration for moss to integrate into our various DE software centres (GNOME Software, KDE Discover and Cosmic Store).

Until now, users could only install AerynOS .stone packages through the terminal. This integration is a significant usability upgrade though we still recommend our users have familiarity of how to interact with moss via command line.

Alongside packagekit, we now also have appstream meta data hosted on our dotdev site. Work is on-going with both packagekit and appstream but the groundwork is completed. We can build from this point towards fully developed software centre integration.

Once this lands in our repository, we will take another step towards making AerynOS a more user friendly distribution.

Documentation improvements

Outside of the code development, there is a renewed focus on our documentation site. This is a continuing and incremental exercise with improvements coming across the board.

Over the last few months, we have improved the FAQ page, added more information on how to update packages on an AerynOS system and added additional information around the Desktop Environments we offer. We have also added specific background detail about how AerynOS is different to other distributions on our Philosophy page.

We will continue fleshing out our documentation in the coming months, with a specific focus on how to contribute, both to the project itself and how to create packages and submit them for inclusion in our repository.

Some of the feedback we have received is that documentation is fragmented and/or not yet created. This is a frustration we can remove through our documentation efforts and we have new contributors helping out in this aspect too.

We are always looking for more support so if you have any interest in getting involved with our documentation efforts, please feel free to reach out on Matrix and specifically engage with NomadicCore.

Next steps

Our focus for the second half of the year remains similar to what we have detailed in our previous two blog posts.

We are working towards versioned repositories which will allow the team to deliver new features to our os-tooling (moss and boulder) in a seamless fashion. Versioned repositories are a prerequisite and gateway to future features that we will deliver in AerynOS hence its prioritisation.

For the os-tooling, we are adding structured logging for better insight and reporting, improving error handling and ensuring we deliver more helpful message output and looking towards adding JSON output for all of this for nicer parsing of “structured output” across process barriers. We continue to add low hanging fruit features

Download AerynOS

The link for our latest iso can be found at our download page.

Development update: os-tools

Jul 11, 2025

NomadicCore

Comms

Ermo

Co-founder

In our recent mid-year blog post, we mentioned that it would be the first in a short series of posts providing updates on the various work streams we have been actively progressing during the last few months. Whilst that post focused primarily on our infrastructure, in this one, we will shift our focus towards the work we have been doing around our os-tools.

Short overview

To recap, our os-tools consist of Moss and Boulder. Whilst also originally written in DLang, initial ports of these were built in Rust during the latter half of 2023. Though we made the odd improvement here and there during 2024, through Q2 2025 we set out to review the code, develop an improvement plan and then put that into action.

The TL;DR is:

Moss: Existing PRs reviewed, refined and merged (including a PR enabling faster package installation via parallel blitting). The code received various bug fixes and refactors for correctness and maintainability.
Boulder: Similarly, existing PRs reviewed, refined and merged including a few smaller features and packaging macros being added. Since Boulder uses the moss crate, it now builds packages slightly quicker due to faster buildroot creation. The odd bugfix was also made.
User Experience: Improved error reporting features were added to both Moss and Boulder to improve the troubleshooting experience for users and packagers. More to be done on this front.

We will be continuing our os-tools work over the coming months with a specific focus on tidying up the code, improving documentation, and ensuring better error handling and status reporting throughout the codebase. This will come in particularly handy when we do the feature work to make us able to produce JSON output as the final part of the alpha2 os-tools milestone.

Overall, this work will help users when they come across unexpected errors and also be beneficial when onboarding new developers.

The JSON output feature, in contrast, is largely targeted at convenient machine parsing of structured output for automation and integration purposes, which we are banking on will come in handy for future development work currently in the planning stages.

Introducing NomadicCore

Before going any further, you may have noticed my name as one of the authors of our mid-year update blog post. If you hang around our Matrix rooms, you will likely already know me but I thought it prudent to provide a formal introduction.

I first became aware of SerpentOS about three years ago but only joined the Matrix chat rooms in September 2023. I’m not actually a developer or have any coding experience, however, I am interested in open source projects and Linux distributions that can help me get the most out of my hardware. I liked what I saw with SerpentOS and over the course of 2024, started getting involved, trying to help out where I could.

Earlier this year, I ended up formally joining the team, around the time of the AerynOS rebrand, taking more of a support/communications role and providing feedback from an “average user” perspective of what I think might be important.

My focus will mainly be around working on our documentation, writing blog posts and engaging on our various social media platforms and Matrix rooms. I’m looking forward to getting stuck in and helping support a Linux distribution I want to use on my various devices.

A deeper dive into our os-tools work

Since their port to Rust, our os-tools have been working well enough. However, we are self-aware enough to know that our initial porting efforts left room for improvement, both in code quality and performance.

The following subsections outline some of the os-tools work we have been doing throughout Q2.

Refactor work

Both tarkah and new contributor, Jonas Platte, have been working on refactoring our existing codebase. To increase the available insight and diagnostic information, we have decided to align around the use of the tracing crate given that important parts of our code base are asynchronous, for which tracing is particularly suited.

For error handling, Jonas suggested that we move away from thiserror towards snafu. Whilst thiserror suited our requirements during our initial porting work, snafu offers some nice quality of life features and forces us to be more explicit about handling different types of errors, which we hope will yield better longer term maintainability. Moving over to snafu requires a little more upfront work to get high quality error output, but we believe that the reward will be worth it once the transition is completed across all of our code base.

Along with this refactor work, tarkah, Jonas and ermo are also improving the documentation within the codebase itself. With the infrastructure code having been ported to Rust, there is now greater scope to reuse and consolidate code between the various tooling crates.

Keeping our dependencies up to date

One aspect of managing our tooling is ensuring that our codebase remains up to date. Part of this effort is also to ensure that we are updating our code’s dependencies to their own respective latest versions to benefit from bug fixes and performance improvements. Whilst this is an on-going task, some of our dependencies had been allowed to get a little stale. Through multiple commits, Jonas has systematically been updating the dependencies in our os-tools repo.

Part of this upgrade work also involved being able to lock dependencies for Rust packages as a way to ensure robustness of the Moss and Boulder builds we use in production.

Moss: Parallel blitting

Long-time contributor Joey Riches identified a parallelization improvement in Moss’s blitting process which was merged after several months of local testing.

In our testing, the code showed significant speedups across all three of our supported file systems (XFS, ext4 and F2FS). The previous single-threaded blitting made using ext4 and F2FS particularly slow, to the point that we did not recommend users use either filesystem as the basis of an AerynOS install.

However, blitting speeds with the new parallel approach — particularly with a “cold” kernel VFS cache — have significantly improved. Whilst ext4 and F2FS are still not as performant as XFS for our use case, they are at least more serviceable as the basis of an AerynOS install than they used to be. By way of an example, I saw a ~2x blitting speed improvement on my Gen4 NVMe SSD using XFS with the new parallel blitting code.

It’s worth restating that, to our knowledge, the moss approach to atomic updates, is the only one of its kind (at least in the Linux space) where users do not have to rely on containerization or A/B system swaps to deliver package updates. Eliminating download speeds as a variable, Moss is capable of atomically installing/updating hundreds of packages on your system in a matter of seconds to tens of seconds on SSD drives, and the installed/upgraded applications are ready to use next time the application is opened. No reboots and no messing with container permissions necessary.

Given that boulder also needs to blit files when it creates buildroots, the code has also had a positive impact on reducing package build times. This will be more evident on larger package builds and will have a cumulative impact, the more package work you end up doing.

Moss: Sync available before installed packages

As we were testing the upgrade path from our old packages.aerynos.com/volatile repository to our new, CDN-backed cdn.aerynos.dev/unstable repository, we ran into some unexpected small niggles related to how packages are resolved.

While tarkah’s fix to Moss was relatively small in terms of code, it served to ensure that updates would install properly on the first go when syncing to the new repo.

Consequently, we synced this bug-fix to the Moss version in the old repository to ensure that users will be able to seamlessly upgrade to the new rolling unstable repository.

Moss: Add index output directory option

As we were preparing the process for syncing the packages in our volatile build-server repository to our new downstream rolling unstable repository on our public-facing server, we ran into an issue with the existing moss index code path.

Before this issue was fixed, the stone.index file would be unconditionally written next to the actual package .stone files. This was useful when indexing local repos, but not as useful when indexing actual stone pool/ directories and sub-directories.

In the end, this was another small feature with somewhat large consequences, in that this enabled us to do the actual manual indexing in a way that is identical to how our infrastructure organizes things when indexing.

This in turns made it possible to make sure that the new rolling unstable repo presents with the same URI “pattern” as our volatile build repo:

❯ moss lr
 - oldrepo = https://packages.aerynos.com/volatile/x86_64/stone.index [0]
 - unstable = https://cdn.aerynos.dev/unstable/x86_64/stone.index [5]
 - volatile = https://build.aerynos.dev/volatile/x86_64/stone.index [10]
 - local = file:///home/ermo/.cache/local_repo/x86_64/stone.index [100]

Boulder: Fix up phase timing in end-of-build report

When Boulder successfully completes a package build, it emits a report detailing how long each phase of the build process took.

ermo noticed that the output was wrong when the time exceeded an hour. For example 136m would be formatted as a rather silly-looking 1h76m instead of 2h16m. He fixed this in the following commit.

Boulder: Improve cache hit rates when updating packages

Boulder is designed to cache files and hash them as part of its build process. By hashing files, it uniquely identifies each file and stores this for later utilization. When a package is updated from one version to the next, where the package has files that have not changed, we have the opportunity to reuse the cache from a prior build (as long as the cache has not been purged for space saving).

Given the way boulder previously cached files, in named directories based on the package source names, there was a high likelihood that new caches would have to be built because the source file names contain version numbers, which by design will always change.

Reilly implemented a change to boulder so that our ccache entries will persist over version number updates improving the hit rate and therefore performance on boulder package builds.

Boulder: Tweak how we use sh-compatible shells

For user-facing recipes, our recipe snippets are now always interpreted by bash.

However, testing with hyperfine has previously shown that dash is ~20% faster to start up during ./configure runs when compared to bash.

To reap the benefits of faster dash process startup times, ermo and Reilly implemented a change to our GNU autotools macros to use dash as the default shell.

Having said that, there are certain packages that just expect and/or are hardcoded to require bash. So to cater to that use case, we have also added autotools macros that packagers can use to make Boulder execute GNU autotools with bash on a per-package basis.

This gives our tooling the “dash as /bin/sh” ./configure speed improvements by default, yet allows packagers to still successfully invoke ./configure et al. with bash, where doing so is necessary for the build to complete.

Boulder: Update build macros

On reviewing Boulder’s build macros, we found some low hanging fruit improvements to make to our cmake, ninja, and meson macros, which we landed back in April. Implementing and improving the various build macros available in AerynOS makes it easier and more convenient for packagers to package up applications with Boulder; either for their own personal use or for submission into the official repositories.

Boulder: Use bsdtar-static for unpacking

At Reilly’s initiative, we moved our decompression solution away from GNU tar to bsdtar-static. This change reduces the likelihood of compatibility issues and ensures Boulder package creation doesn’t rely on a dynamically linked system library, thus making it more of a reliable solution.

With this move, we have also added the ability to decompress tgz based source packages as part of the boulder build process.

Other notable work

Some of the work we have done has been aimed more at how we use our os-tools rather than the os-tools themselves.

Update build triple and fix up ARM AArch targets

With our recent transition from SerpentOS to AerynOS, we needed to update our build triple accordingly. This step has been completed in the background whilst allowing for seamless updates from older SerpentOS systems onto AerynOS based systems. This is part of a wider rebranding effort that is still on-going through our documentation site, repo READMEs and anywhere else we have an official presence.

In this same area, whilst AerynOS currently only supports x86_64 based devices, there is a desire to be able to target other system types longer term. One of our contributors has been experimenting with RISC-V so we have added preliminary support for this to aid their testing. Don’t expect to see AerynOS on RISC-V any time soon, but it’s great to see our distro becoming a sandbox for fun and experimenting on alternative systems.

Emul32 ELF Machine type migration

During the port of boulder from DLang over to Rust, there was a change in how we expressed the ISA for packages built for the x86 emul32 architecture target. The old DLang version of Boulder expressed it as x86 where the Rust elf crate expresses it as 386 (EM_386 for those of you familiar with ELF parsing internals).

Reilly took point in implementing a series of changes that retained full compatibility between packages originally built on the old DLang infrastructure, and newer packages built on the Rust infrastructure.

The end goal was to flush out the packages containing references to the x86 Emul32 ISA through the recent rebuild of our whole recipes repository. This was accomplished by first ensuring that all packages exposed both x86 and 386 provider patterns, and then subsequently dropping the code that wrote the x86 provider patterns during the second full repo rebuild, ensuring that packages only contained the 386 provider patterns.

In the end, this worked out nicely for us.

Current and near-future os-tools focus

We reviewed the open issues in our various GitHub repositories and the plethora of ideas we have for our tooling, and developed a high level set of milestones for our os-tools. For this milestone (os-tools alpha2), we want to focus on:

Adding structured logging for better insight and reporting
Improving error handling
Ensuring we deliver more helpful message output
Adding JSON output for the above, for nicer parsing of structured output across process barriers
Adding low hanging fruit features and fixing misfeatures as we review the code

As our readers may have surmised, we have slowly accumulated bits of technical debt here and there over the course of development. The os-tools alpha2 milestone is our chance to address that and make our code crisp, clean and ready for already-planned future development work, while we stabilize our new infra code in parallel to this effort.

A special thank you goes to Jonas for the work he has already done in terms of this sort of refactor work.

How to join in the fun

As already covered in this blog, we have been reviewing all open issues and PRs in our os-tools repository on GitHub for prioritization and to set internal milestones.

We are open to and actively looking for contributors who might be interested in looking through our code and providing feedback.

If you would like to try your hand at contributing, look out for issues marked “good first issue” and get in touch on Matrix.

We hope to see you there!

Mid Year Update

Jun 30, 2025

NomadicCore

Comms

Ermo

Co-founder

As we hit the middle of the year, it’s time for another update for those of you following along with AerynOS’s development.

Over the last few months, things may have seemed unusually quiet, however rest assured that there has been A LOT going on in the background. As such, we are preparing a short series of blog posts to go over the relevant topics in the coming weeks.

For this blog post, we are going to cover our infrastructure port, along with the process of rebuilding our entire package repository.

The TL;DR is that:

All core AerynOS tooling is now written in Rust
Every recipe in the repository has been rebuilt (twice!) with many packages then having been updated to newer versions after the rebuilds were completed
A CDN has been implemented for faster package installation and ISO downloads

New infrastructure tooling

When delivering a Linux distribution, its infrastructure and associated processes effectively act as the “spine” of the project. But spine surgery can be a delicate affair, particularly when it comes to rehabilitation after successful surgery.

For us, this cycle has been particularly demanding, as we have completed an MVP (Minimum Viable Product) port of our infrastructure tooling code to Rust, meaning that all core AerynOS tooling has now fully transitioned away from DLang.

We have covered the reasons for this transition previously, and it’s fair to say that we are already feeling the benefits of easy and native reuse of code in our tooling repositories and welcoming more Rust contributors into our community.

Why now?

Earlier this year, our existing DLang build infrastructure started showing signs of instability and required more and more manual intervention to successfully land packages.

Given our prior decision to transition our tooling over to Rust, we had already stopped further development of the DLang based infrastructure. Hence, we decided to accelerate our transition timeline for the infrastructure re-write to Rust with tarkah and ermo leading the development activity, which began at the end of March.

Towards the end of May, we put the first infrastructure prototype to the test, and then iteratively fixed bugs and built out missing functionality to the point of being able to put our MVP into production on our build infrastructure.

This MVP will serve as the development base of the code that will be used for all future package builds.

What do we mean by infra?

Our infra is comprised of the Summit, Avalanche and Vessel service components:

Summit: Package build-controller, build-orchestrator and build-dashboard, monitors recipes tree and automatically builds new, incoming recipes once they show up
Avalanche: Build agent middleware, takes build orders from Summit and builds them with Boulder on a remote system, sends build logs back in real time to Summit, and reports the build result to summit at the end of the build.
Vessel: Package repository manager. Summit tells Vessel which packages and other build artefacts to expect from a build task that Avalanche has completed, and then Avalanche pushes those packages and build artefacts to Vessel, which then saves them in the appropriate place and re-indexes the repository with the new packages, so users can install/update them.

We have some cool features planned in AerynOS that we envision will make package maintenance a lot easier to manage through smart use of automation.

Until these features are implemented, however, maintaining the AerynOS repository will remain somewhat human resource intensive. This is the main reason why the repository is consciously being kept “small” for now, with us deliberately focusing on having packages that will help developers and contributors improve AerynOS, while still delivering a nice Daily Driver experience.

Until the new features are implemented, this will necessarily be a balancing act between maintaining the package repository so it doesn’t go stale vs. having the development time to implement the new features.

How did you test it?

Aside from porting the infrastructure code to Rust, proper testing was required to yield confidence that packages were both successfully built on the new infrastructure and that they worked as expected.

The end goal was to prove that we were able to rebuild the full AerynOS recipes repository (currently at ~950 recipes) from start to finish without infra-related build errors on the new infra.

To enable the rebuild, ermo set up a distributed build cluster of four builders of varying hardware specifications. A separate branch of the ‘recipes’ repository was created, and was used to test both the Rust infrastructure and to land packages for internal testing without them being seeded to user installs.

In addition, compared to the old infra, we made it simpler to add new avalanche build agents to the build cluster, thus making it very simple to scale out our build cluster as required.

To summarise the infrastructure Rust re-write and testing effort, we have:

Completed more than 3k recipe builds
Deployed the new Rust infrastructure on the AerynOS builders and continue to use it on ermo’s build cluster
Validated that the new infrastructure code is more stable and performant at runtime than the previous DLang version

How did testing add value?

The full rebuild of the recipes repository has also served to ensure ABI sanity for dependencies. Additionally, we can now say that at this point in time, the whole AerynOS repository is known to be buildable and works with all the latest toolchains.

A special thanks goes to Reilly Brogan, who worked diligently with ermo to not only drive the rebuild process, but also to ensure that some longstanding repository issues were corrected as part of the rebuild process.

During this process, we have delivered updates to our os-tools (Boulder and Moss), toolchains and build systems. A selection of the updates and additions include (but is certainly not limited to):

Linux 6.14.11 (6.15.x on the way)
LLVM 20.1.7
GCC 15.1.1
Rust 1.88.0
Golang 1.24.4
Mesa 25.1.4
GNOME 48.2
COSMIC 1.0.0_alpha7
Sway 1.11
Firefox 140.0.1
Thunderbird v139.0.2
Uutils-coreutils 0.1.0
Nodejs 22.14.0
Wine 10.8
Distrobox added at v1.8.1.2
Exfatprogs added at v1.2.9
Fzf added at v0.62.0
Kitty added at v0.41.1
Waybar added at v0.12.0

A New Package Repository

As mentioned earlier, the testing work was conducted on a separate branch of the recipes repository. Consequently, those of you on the old packages.aerynos.com/volatile/ repository, have not received any updates over the last 10-12 weeks.

This was a conscious decision to ensure that the mostly untested packages built during the infrastructure testing process did not reach end users immediately. Even though AerynOS is in Alpha and under continuous development, we still do our best not to break user systems if we can avoid it!

Now that we have a level of testing in place, with this blog post, we are announcing a new rolling unstable package repository for users. The old volatile package repository has received one final update to Moss that fixes an important bug when transitioning to the new unstable repository.

To ease the transition to the new repository for existing users, we are working on a script that can automatically modify the active repository on the system.

Once this script has been sufficiently productized, the next time existing users update their systems, they will notice that every single package will show an update available.

The exact number will vary from system to system depending on how many other packages are installed from the repository but for context, on a base AerynOS GNOME install, this is around 500 packages.

In the meantime, we have created a manual guide on how to transition existing installs to the new repository in our GitHub Discussions forum here. The process is fairly simple, but if you do have any issues transitioning manually, do get in touch via a comment under the GitHub Discussions post or via Matrix.

Content Delivery Network for Packages and ISOs

A common bit of feedback we have been receiving relates to the download speed of our repository, namely that it is not fast or even acceptable, especially if you live outside of Europe. This became more evident for those using the rebuild repository on ermo’s rebuild testing server, which felt noticeably faster for people in Europe in particular.

To remedy this, we have implemented CDN caching for our new cdn.aerynos.dev hosted assets. This means there will be synced copies of our ISOs and package repository on CDN servers around the world, which should help improve download speeds.

In particular, the new rolling unstable package repository mentioned above will be served via this CDN for the benefit of our users.

Please let us know how you get on with AerynOS ISO and package downloads in the coming weeks, as we would love to validate the improvement outside of our own internal testing.

Future infrastructure development targets

So far, we have only outlined what we have already accomplished since late March.

The next part of this blog post is going to be a brief outline of where we are going from here in terms of infrastructure and repository development.

Versioned Repositories

With the transition to the new infrastructure and the new unstable repository, we have been freed up to begin planning out the necessary steps to be able to deliver versioned repositories and versioned Moss format upgrades.

These topics have been mentioned in a previous blog post.

How do versioned repositories add value?

Versioned Repositories will enable us to deploy new Boulder and Moss features in a seamless fashion. This will enable us to introduce breaking code and on-disk format changes, that would otherwise cause installed systems to require manual intervention for them to continue to receive updates.

Once versioned repositories are in place, the goal is that users will be able to simply update and sync their system as normal via the sudo moss sync -u command.

With this:

Users will be upgraded to the new versions of Moss that uses a new repository format, without having to pay special attention.
It will enable AerynOS to iteratively expand the capability of Moss and Boulder on existing systems without breaking user systems in the process.

Additional Opportunities

We consider versioned repositories a pre-requisite for what we call “try-builds” and eventually multi-arch support.

Automated try-builds denotes the process whereby the infrastructure discovers an update to the upstream source repository of a package, attempts to auto-update the recipe and then attempts to build the updated package recipe in question.
We think this will be a useful tool for contributors as it will automate some of the packaging tedium related to simple package version updates. It will also help enable automated regression testing and build flag optimisation in a future workstream.
Included under the multi-arch umbrella is our ability to target ARM, RISC-V, and different x86 architecture levels such as x86-64-v3 or v4.

Project Status

Infra

Within the previous 3 month period, we have rebuilt a brand new Rust version of the infrastructure tooling that is robust enough to run in production on AerynOS servers, delivering packages to our contributors and users. This new version has proven to be more stable and performant than the old DLang version we were previously using.

Packaging

From a day to day perspective, unlocking the infrastructure means that we can get back to reviewing and landing recipe PRs for our package maintainers or accepting new contributors into our AerynOS ecosystem. For those wishing to contribute to AerynOS, please make sure that you have manually switched over to our new repositories before making submissions to ensure you are using all the latest tooling.

Alternatively, you can wait until the automatic transition script is functional and have it make the change for you.

Where to get in touch with us

If you want to engage with the team, feel free to drop by our GitHub Discussions, raise issues across our various repositories or if you’re interested in contributing, feel free to raise PRs where you think our code can be improved or where you want to submit recipes for our repo.

We also have our matrix space that you can access via this link:

The Development room in particular is a great place for discussions around our code.
The General room is a great place to drop by and get to know the team.
The Packaging room is where you want to be if you’re interested in building packages for yourself and/or submitting them to the repository.

What’s next?

Concurrently to our work around the infrastructure re-write and repository rebuild, there has been several additional workstreams running in the background.

The team has been refactoring our existing Rust code, mainly focused on our os-tools (Moss and Boulder) and we are working on several additional improvements that we want to get over the finish line before our next ISO release.

We will be sharing details of this work in upcoming blog posts over the next few weeks.

🏗️ AerynOS: The OS As Infrastructure

Mar 30, 2025

Ikey

Founder

Taking a break from our usual release-oriented updates, I thought it was high time to dive into what AerynOS actually is and what sets it apart from other distros. Fair warning, this is a meaty, in depth post, and still only scratches the surface. If you’re interested in the future of Linux distributions, read on. It may also help to visit our work-in-progress documentation site for more information.

🧱 The OS As Infrastructure

Firstly, AerynOS isn’t Yet Another Linux Distribution. It’s a platform, a foundation, and a set of tools engineered in accordance with a vision and design that just so happens to produce a Linux distribution. Were we to go back in time and build a design brief for AerynOS, the initial question might be:

What if the operating system itself behaved like modern infrastructure?

AerynOS is the answer to that question. What if, instead of doing things the way they’ve always been done, we started from the ground up and produced a well designed system, rather than the traditional model of in-place mutation internal to a distribution?

🏗️ The AerynOS Foundation

Leveraging years of experience across the industry, including some notable pioneering projects such as Solus, Clear Linux, and others, we set out to build things the hard way, in order to make them easy.. eventually.

🔧 Toolchain

At the very core, is the decision to not be another GNU/Linux distribution. We default to the LLVM toolchain, using libc++ and compiler-rt by default. This isn’t just a case of “we like LLVM”, but rather a strategic decision to leverage the superior diagnostics, and ensure correctness / portability of packages. Whilst in our very early stages a few years ago we did trial musl, we do default to glibc for compatibility reasons and its superior performance characteristics. The performance advantage of glibc over musl is well-documented, particularly for compute-intensive workloads and applications requiring optimal threading performance. We’re here to build a working, usable system, for a multitude of use cases and verticals, and glibc provides the best balance of compatibility and performance.

Note, we do still package gcc and it’s trivial to enforce its use in a package recipe by setting the toolchain field in stone.yaml to gnu.

Lastly, it’s worth noting we build all packages for x86_64-v2 as our minimum baseline, and perform targeted optimisation in our package recipes using the extensive tuning options configurable in boulder.

📄 Statelessness

Our packages are forbidden from containing any files outside of /usr. In order to enable this, packages and/or configurations are altered in AerynOS to ensure they can operate in the absence of a user provided configuration. This forces us to ensure sane-defaults are baked in at all levels, and eliminates dreaded 3-way merge conflicts on package updates. There are no conflicts, because everything in /etc and /var is yours. Likewise, /usr belongs exclusively to the system.

This approach was coined and developed during Clear Linux and Solus days, and we’re refining it further in AerynOS. Other projects have adopted similar approaches, termed “hermetic /usr”.

You might be wondering how files end up in /etc automatically. To this end, we support two forms of package triggers:

🔄 Transaction Triggers

These triggers are run at the end of a transaction in an ephemeral container (linux namespace) and may affect the contents of the transaction-specific /usr tree. This is useful for interdependent packages that need to dynamically produce plugin registries, for example.

🖥️ System Triggers

These triggers do not run in an isolated container, but instead are run in the context of the host system after the transaction has been successfully built and applied. It is these (minimally used) triggers that invoke systemd-tmpfiles, systemd-sysusers, etc. Even for these cases we take special care to ensure that our default configs are sane and that a rebuild is always possible.

👤 System Accounts

Another important aspect is the handling of local system accounts. Traditionally these are snippets/shell scripts that invoke useradd, groupadd, etc. In AerynOS we default to systemd userdb for any users/groups that do not explicitly need groups that users would join. Thus, using drop-ins in /usr/lib/userdb, most accounts are defined and made available via NSS. We use this already in gdm, polkit, colord, and many others.

For the other cases, we utilise systemd-sysusers using snippets in /usr/lib/sysusers.d to create the necessary system accounts. In time, when GNOME and other desktops (as well as shadow et al.) are more adapted to userdb and systemd-homed, we won’t need to rely so much on sysusers. The goal of course is elimination of /etc/passwd and /etc/group entirely, facilitating quicker recovery, provisioning, etc. For now some packages will ship sysusers-based groups, ie docker group, etc.

⚛️ Atomic Updates

Every moss transaction is atomic. Very high level: we produce a new usr tree (rootfs essentially due to mandated usr-merge) very quickly using hardlinks from a deduplicated cache. Once successfully built and primed, we atomically swap the new tree into place. Effectively, the staged transaction is swapped with the real /usr using renameat2 with RENAME_EXCHANGE. It either works or it doesn’t. Nothing in between.

🥾 Boot Management

Leaning heavily on our blsforme and disks-rs projects, we’ve taken a very different approach to boot management. As part of applying a transaction, we produce a Boot Loader Specification (BLS) entry for the new transaction. The tooling will discover the EFI System Partition (ESP), and mount it if necessary. It then synchronises the bootloader itself (systemd-boot), the relevant BLS entries, and the kernel/initramfs. Additionally it will automatically garbage collect older entries, currently retaining 5 of the last transactions.

It’s not just a case of copy/paste and hope for the best. We dynamically produce the root parameters for the kernel command line by directly reading the superblocks (natively, in Rust) of the devices all the way up the chain for the root filesystem. It means there’s no configuration file anywhere in AerynOS containing your root= parameter, and we’re even able to read LUKS2 encrypted devices to add the rd.luks.uuid parameter.

If that wasn’t cool enough, we also encode the moss transaction ID into the kernel command line. This is picked up during early boot in our initramfs, before /sysroot is pivoted to. Long story short it means that every kernel is correctly synchronised with the right rootfs, and that rollback is cheap, easy, and accessible directly from the boot menu.

In addition, we also automatically support XBOOTLDR partitions. In the absence of LoaderDevicePartUUID EFI variable, we’ll scan the GPT table itself relative to the rootfs to find the ESP, and we’ll always scan GPT relative to ESP to find the XBOOTLDR partition.

Long story short? No /etc/default/grub. In fact, if you wipe your ESP, moss can rebuild it from scratch.

📦 The Stone Format

At the heart of moss lies our .stone format. At a basic level, it’s our binary package format. Using a version-agnostic header, we’ve ensured that we design for the future. We also version every payload within the stone, allowing us to refine and evolve the format over time. Currently we support 4 payloads in a single stone:

📄 Content payload: A sequential blob of deduplicated data
🔍 Index payload: Contains offsets for the content payload, keyed by the XXH128 hash of the content
🗂️ Layout payload: Describes the intended filesystem layout when the stone is applied
📋 Metadata payload: Sequence of strongly typed, tagged metadata entries such as name, providers, etc.

Right now we default to XXH128 for hashing, but Blake3 is on the horizon (and used in blsforme). We compress all payloads using Zstd, offering great decompression performance whilst still providing a decent compression ratio.

The process for “installing” a .stone is quite different to other systems.

📥 The stone is fetched according to the repository data
🧠 Index payload is unpacked in memory
🗜️ Content payload is decompressed in a single run to a temporary location
🔗 Using the index payload, the content payload is spliced into the content addressable storage (CAS)
📂 The layout payload is loaded and merged into the LayoutDB, keyed by the unique package ID (SHA256 for the entire .stone, per the repo)
📊 Using the same key, the metadata payload is loaded into the “InstallDB”.

Notice that at no point are we actually “installing” anything. We cache into the CAS, and store metadata and layout details. This is used when producing a transaction. Also note there are various safe guards, integrity checks and such in place. For example every payload contains a “CRC” in the payload header, verified on read. We actually use XXH64 for this.

🔁 What is a transaction?

In AerynOS, a transaction is an entirely self-contained rootfs produced by moss. Right now we have to emulate imperative package management, by producing a new dependency graph seeded from the previous system state as recorded in the StateDB. Alterations are made and validated, and then the new DAG selects the packages to be included in the transaction.

At a high level, the transaction is produced by:

📊 Producing a valid dependency graph, seeded from the previous system state
🧮 Determine cache availability for each package
📥 Fetch and cache every missing .stone to produce the transaction
📂 Load all layouts for the stones by ID
🌐 Using our multi-layered vfs approach, we build an arena graph of the target filesystem, detecting conflicts ahead of time, whilst also precomputing reparented nodes (i.e symlink redirection) and produce an optimal iteration order for transaction application
🚶 Walking the vfs iterator to produce the new rootfs in a staging location, using linkat, mkdirat, etc, to optimally produce the new filesystem, linked from the deduplicated cache
📦 Binding an ephemeral container to the new rootfs, and running transaction triggers
💾 Recording the transaction in the StateDB.

As mentioned above, once a successful transaction is produced, it is atomically swapped into place. If the transaction fails, no ill effects are observed on the system, and your work day continues as normal. In the event of a successful transaction, the system is updated and ready to go, along with system trigger execution and boot management updates.

🔮 The Future

All of this is just the start. It’s taken a few years, and perhaps now it’s clear why. We’ve not even covered our package build tool, boulder, or indeed our build infrastructure! With that said, lets skip forward a short while and see what’s coming down the pipe.

🌍 Beyond the distro model

To reiterate, we emulate imperative package management. And to be honest, it’s totally pointless. It actually introduces more bugs than it solves. Given that we produce a new rootfs for every transaction, we could just as easily produce an entirely new graph each time too.

That’s exactly what we’re planning to do. In a similar vein to Gentoo or Nix, the intent is a global file to explicitly state the desired state of the system, whilst the tooling will simply fulfil it based on the moss plugin cascade. Whilst we have no intention of conflating state and configuration, we will in time extend this system model to support variants of packages in order to provide a kind of “slots” system, and to implement a richer, saner alternative to the infamous update-alternatives.

Obvious candidates include mutually exclusive packages like coreutils and uutils-coreutils (a gnu variant weight, perhaps?) or coinstallables such as clang, ensuring a default vendor version whilst allowing overrides at the local install or indeed package recipe level.

🔒 Immutability

Often AerynOS is described as an immutable OS, and that’s not strictly true. Granted, every transaction results in a new /usr tree, so local changes won’t persist and recovery is immediately available. However, we’re not immutable in the sense of read-only.

Having produced a composition-first, developer&user friendly atomic update implementation, we want to take this further in order to also support immutability without compromise: no unnecessary reboots. To do so we’ll implement something similar to composefs, without the drawbacks. Using the same transaction driven approach we have now, we’ll simply produce an erofs metadata image dynamically instead of the exploded filesystem tree. Utilising trusted.overlay.redirect xattr, and an overlayfs mount, we’ll expose our own CAS through the layouts in erofs, and also support fsverity hashes. Icing on the cake? We’ll leverage mount stacks to retain our composition-first, no unnecessary reboot ethos.

📚 Versioned repositories

We’ve touched on this a few times in our blog posts. Essentially the binary repository will be produced as a result of available build artefacts correlating to the state of our git repo manifest files (boulder proofs of build). The main repository index will link releases to moss/stone format versions, such that moss would only update to the latest release that it supports. This allows us to stagger breaking changes in a way that nothing breaks at all.

📊 Current Status

🚀 Actively shipping GNOME ISOs
🎮 Quite usable for gaming with NVIDIA drivers, Steam, Flatpak, etc
👥 Real users are already praising stability and innovation
🛠️ Focused heavily on disks-rs and lichen-installer to leverage disk strategy files (automatic provisioning described in KDL)

🎉 Conclusion

This isn’t just another distro. It’s us redefining distributing Linux. We’ve achieved a tremendous amount, having successfully integrated all of the above into a cohesive, singular whole. The amusing part of course being that the resulting system is “boring” - it just works.

Now, we are alpha. We’re not done, we’re not without issues. That said, we’re building the future and with your support, we’ll get there even faster. This post has only scratched the surface of AerynOS, but if you like what we’re doing, please do get involved or support us.

Or see other ways in which you can support the project financially.

🚀 Hello AerynOS

Mar 25, 2025

Ikey

Founder

Hello from AerynOS! ✨ As you recall from February, we set about rebranding Serpent OS into what you now see, AerynOS. It’s not been without challenges, and where possible we’ve ensured continuity. Additionally we silently dropped a few ISOs, and can now take our first formal steps under our new identity. TLDR: the transition has been entirely fluid and you only need to keep updating, no manual intervention is necessary! 🎉

Progress has been steady but quiet lately, as due to unfortunate circumstances I’ve been working in bursts from my wifes hospital bedside.

💿 But first, an ISO drop!

OK, lets get right to it. We’ve released AerynOS 2025.03, with the following shinies:

🖥️ GNOME 48.0
🐧 Linux 6.13.8
🦊 Firefox 136.0.2
🎮 Mesa 25.0.2
🚀 Vulkan SDK 1.4.309.0
🛠️ LLVM 19.1.7

We’ve also decoupled the internal tooling version from the ISO versions to make them a little bit more.. well.. readable, for humans.

Grab it now from the download page.

Installer preview

💰 Funding

I want to thank everyone for their support of the project since we got more transparent around goals. It’s been a huge help and has facilitated massive progress on the project! 🙏 AerynOS no longer feels like a hacked together PoC, but a very solid daily runner. Yes, it’s certainly alpha, and the installer leaves a lot to be desired. For daily use and updates? It’s really quite something.

Please note that I never realised the Ko-fi goals don’t automatically reset, so the currently listed goal has been running for way more than a month.. xD However, over a period of time, we did manage to achieve it! 🎉 I’ll be resetting it shortly on a fixed date for each month ahead.

🔄 Rebranding, the sane way.

Rebranding can be more challenging than one expects. The easy stuff is out of the way, HTTP redirects and DNS trickery to ensure old URLs continue to work without manual intervention. In the case of the repos, we’ve updated the scheme:

🧪 Unstable presync repos

packages.serpentos.com -> packages.aerynos.dev

🌟 Post-sync user-facing repos

dev.serpentos.com -> packages.aerynos.com

📚 Documentation site

docs.serpentos.com -> aerynos.dev

Note the former scheme made no sense at all whereas now we take advantage of the dotdev for unstable work. A future release of moss will automatically handle the transition on installs for the sake of consistency.

📊 Introducing `os-info`

Previously we had moss generate the /usr/lib/os-release file on demand using compiled in defaults, which was somewhat inflexible. Now, we ship a JSON file (/usr/lib/os-info.json) containing a description of the OS, the composition of technologies, and the capabilities. While os-release and lsb_release exist, they provide very primitive identification and metadata. os-info is designed to provide compatibility with those formats while being far more expressive. Importantly, it also contains a mechanism for identifying the former identities of an operating system:

...
"former_identities": [
  {
    "id": "serpentos",
    "name": "Serpent OS",
    "start_date": "2020-06-15T00:00:00Z",
    "end_date": "2025-03-17T00:00:00Z",
    "end_version": "0.24.6",
    "announcement": "https://aerynos.com/blog/2025/02/14/evolve-this-os/"
  }
]
...

moss now utilises os-info to generate the os-release file, as well as to provide identification and history for our blsforme crate to manage each entry on the boot partition. This has allowed us to sync the branding on a per transaction basis, while still “owning” the legacy branded transactions on the ESP too (i.e /EFI/serpentos) and ensure they are correctly garbage collected. Interestingly this means that for a short period of time the boot menu will still show some of the older Serpent OS entries, allowing you to roll back to it.

Please visit the os-info repository for more details, the schema, etc. We’re very keen to open collaboration and adoption of this project, envisaging use cases in installers, welcome apps, and indeed even container introspection tooling.

🔧 Installer progress

We’ve officially begun the revamp of lichen-installer using a proper split between the privileged backend and frontend. In order to permit a number of frontends and use cases in future, we’re using tonic gRPC messages to communicate, currently just on a UNIX domain socket. In future, we’ll have use cases for TCP using WASM frontend.

More importantly than that, we’ve now implemented the foundational aspects for disks-rs. Long story short - we’re using provisioning strategies defined in custom .kdl files that are dynamically tested/validated against an input storage pool to produce a working set of changes. Such as, create a new partition table and add some partitions to it, using some constraints logic.

It is early days, but its quite awesome that we’re able to automatically partition disks based on these strategy files! ✨ These will form the backbone of lichen, allowing us to offer rich automatic storage strategies as well as manual partitioning options. Notable is the ability to set the volid or uuid of a specific filesystem, and partuuid, etc. This in essence allows shallow reproduction of an installation/disk topology using a configuration file.

🙏 Acknowledgments

This transition wouldn’t have been possible without our amazing community. Both long-time contributors and newcomers have stepped up to the plate to make AerynOS a reality, pushing us across the finish line during this critical transition.

Special thanks to Cameron for his instrumental work in porting our websites to a more maintainable, Astro-based infrastructure. This migration has not only improved our development workflow but also enhanced the user experience across all our web properties.

Whether you’ve contributed code, reported bugs, tested releases, spread the word, or supported us financially - thank you. AerynOS is truly a community effort, and we’re excited to continue this journey together.

As always, you can join our community on Matrix or contribute on GitHub.

🔄 Migration Guide

For existing Serpent OS users, migrating to AerynOS is straightforward:

Simply run the following command in your terminal:
Terminal window
```
sudo moss sync -u
```
Some branding changes may require a second moss operation to fully “take” effect (as they’ll be using the new moss binary). After the initial sync, run one of these commands to complete the transition:
Terminal window
```
sudo moss sync
# or
sudo moss install some-package
```

That’s it! Your system will seamlessly transition to AerynOS while maintaining all your existing configurations and installed packages.

🔮 Next up

We’ll continue on the path we set at the start of the year. By and large we’ll improve the core tooling to continue delivering a better experience for users, developers, gamers, etc.

By the way, we do have nvidia-open-gpu-kernel-modules / nvidia-graphics-driver for the linux-desktop users wanting to game on their shiny AerynOS installs. Our Steam package works great, and any feedback is appreciated on improving it (ie udev/controller stuff, 6.14 kernel is planned).

Shifting more towards the installer images, using Slint for the installer frontend, and dropping unnecessary packages from the installer image. Live ISOs will of course be available, but our primary target is installer images.
Integration of upstreams-rs and ABI tracking into the tooling in order to vastly lighten the load for maintainers.
Accelerated delivery of milestone ISOs.

Evolve This OS

Feb 14, 2025

Ikey

Founder

A long overdue, and highly requested update today. We’re finally rebranding the project!

🤔 Why the change?

The “Serpent OS” name was a quickly chosen name that stuck. Unfortunately “serpents” are often associated with negative connotations, and we’ve had a lot of feedback over the years that the name was off-putting. Let’s be completely honest, it’s not the most inviting name for a project. Who wants to trust a serpent? Generally speaking they’re considered dangerous at best.

It’s fair to say we’ve spent a long time in prototype and alpha phases. In order to move forward, our identity needs to be more befitting of the project we’re building. A move into the real world. This isn’t a hobby project, it’s a full blown Linux distribution with serious technical underpinnings, achievements and goals. Getting the tone right from day dot is critical.

Do note, there is no change to the internal core team, so we’re aiming for full continuity and a seamless transition. Had enough of these snakes on this plane? We have too.

✨ What’s the new name?

Hah, it’s not “Evolve OS”, despite the clickbait title. Our new name is… drum roll please…

🌟 AerynOS

Pronounced like “Erin”, it’s a name that we feel is more befitting of the project. Pulling from multiple etymologies, it’s a name that better describes the project now versus the project that started as Serpent OS.

“Aer” is rather obvious, Latin in origin. The phonetic “Erin” is a nod to the Irish roots of the project, and of course a home. There are a number of reasons for the name, which will form part of the initial documentation on the new website.

Our intent is to have a name that is more inviting, and more descriptive of the project’s goals and aspirations. We’re not anti-establishment or anti-corporation - if anything, we’re a statement that without the fiscal handcuffs, we can produce a technically sound and user-friendly operating system.

⏰ Timeline

Having already secured AerynOS.com, AerynOS.dev, plus the associated social media accounts, we’re now in the process of rebranding the project. For sheer irony, we’ll make the final transition day the 17th of March, 2025. Yes, St. Patrick’s Day. The Matrix space has already been updated, and any areas where a new name isn’t feasible will see almost immediate renames.

A new GitHub organisation has been created, with our first priority there to create a proper organisation structure for once. Over the coming weeks everything will be migrated over, and we’ll mark the old organisation read-only/deprecated. Can’t have folks thinking Serpent OS was abandoned, after all.

Stick with us, we’ve got plenty of exciting news to come over the next days and weeks, as we embrace new transparency and shed our old skin. Plus, we’ll be starting to onboard for web properties, translations, artwork, and more. Cherry on the cake? Admitting that we’re actually building a full distribution will allow us to put relevant priorities in place to expose Serpent OS.. I mean AerynOS .. functionality to the wider world. Get ready for a reboot of the grassroots, independent Linux scene. We’re here to stay.

Hello 2025

Feb 6, 2025

Ikey

Founder

TL;DR: Serpent OS is facing funding challenges but development continues. Alpha2 is coming soon with an improved installer. We’re seeking community support through donations and volunteers for key roles. Our technical roadmap includes versioned repositories, immutable OS features, and improved package management workflows.

We had a flurry of activity around the Christmas period, including our first alpha release as well as enabling offline rollbacks early in January. We’re actively working on alpha2, but we also need to talk about the elephant in the room.

🐘 The elephant in the room

Recently I posted a tweet that has been covered by a few news outlets. The long and short is quite simple: distinct lack of funds. A very long story short: a change in my contract led to a loss of income protection, combined with a herniated disc (C6-C7), culiminated in having to resign. Naturally I’ve been working exclusively on Serpent OS since then, but I’ve also got to keep the lights on at home.

This doesn’t mean the project is closing or anything like that, but it does mean that I actively need to keep costs down whilst using daytime hours for financing. This is what’s led to the significant slowdown in the last few weeks, and I’m just trying to be as transparent as possible and manage expectations.

💼 But couldn’t you just get a job?

Sure. The reality is that with a job, and kids, there’s very little time left for Serpent OS. I personally believe very strongly in Serpent OS, and I want to see it succeed. As a truly independent project, we’re in a unique position to do things that other projects can’t. We haven’t got to answer to shareholders, or concern ourselves with market share. We can focus on making the best possible OS for our users.

But it’s not just that. We’re building the tools so that others can build their vision of the best OS. Or provide applicances. Or <insert your idea here>. We’re building a set of tools and principles to facilitate innovation and disruption, for projects of any size. We’re not just building an OS, we’re building a platform. Bring your own distro, if you will.

My sincere hope is that others also believe in this vision, and can help us get there. It’s make or break time, and you have the power to help us make it.

💰 Funding

OK, so how can folks help? The most obvious one is funding, so we’ll get that out of the way first. Right now it’s possible to support us via Ko-fi or GitHub Sponsors. For the sake of transparency, as noted on the Sponsor page, we haven’t got an established company or business bank account. In time, we’re looking towards something more cooperative. For now, the reality is the daily running and development largely falls on me.

🏗️ Infrastructure

One way that we could certainly cut some costs, and improve scaling, is through additional infrastructure. We’re currently hosted on a single Hetzner server, and we’re looking to expand that to a more distributed setup. Rune Morling currently hosts 2 of the builders in his home at his own expense, and the main server is also the primary builder, which is less than ideal.

We need to split the repository hosting as its grown enormously, and could do with a few more builders to facilitate “try builds” of pull requests.

Lastly we’re looking at CDN solutions to improve the speed of package downloads (efforts currently ongoing).

👥 Roles and responsibilities

We’re growing quickly and that comes with some pain points. We’re looking for immediate help in the following key areas:

Community management: Someone to manage social media, forums, and other community platforms.
Documentation: Once we pivot to Astro + Starship, we’ll need to focus on adding high quality documentation.
Translation: We’re enabling gettext + fluent where appropriate, and will need extensive translation work.
Web development: We need to improve our web presence, management of web properties, transition to Astro, etc.

While of course we’d also love more developers and packagers, most of the time is spent in these areas already, and the other roles desperately need attention.

🚀 Latest developments

Before we get into the future, let’s talk about what we’ve been working on lately.

💾 Disk Management (disks-rs)

A couple of weeks back we broke the news that we were working on a new disk management library, disks-rs. The scope wasn’t made quite clear at the time, but it’s another building block to facilitate smarter applications. Eventually we want to support the manipulation of various filesystems and disk structures, for use in CI pipelines, etc.

Another requirement for disk-rs is to not only expose sane partitioning logic, but also to provide a way to declaratively define disk layouts. This will be used in the installer, but also for reproducing entire installations. It forms the basis for our future provisioning system.

Right now we’re working on a very cool .kdl based format for defining disk layouts. It takes advantage of kdl-rs streaming parser to implement a procedural syntax complete with miette error handling for a very nice developer experience:

// Create a partition table. Defaults to GPT
create-partition-table type="gpt" disk="root_disk"

// Create the ESP
create-partition disk="root_disk" id="esp" {
    constraints {
        min (GB)1
        max (GB)2
    }
    type ...
}

🛠️ Installer

Our installer has quite a nice backend, but honestly the frontend experience leaves a lot to be desired. Given that our current focus is being able to not only onboard developers but build tooling against an installed system, we need to make the core of this effectively scriptable. To do so will build upon ongoing IPC work for moss and disks-rs, and the introduction of a new GUI frontend based on Slint. While early days we’re trying to think 10 miles down the road, where we can utilise the LinuxKMS backend for minimal installer ISOs.

Early preview:

Installer preview

🔮 The road to stability

So we have some exploratory works in progress, and it’s time to discuss how they all relate to the future of Serpent OS. You’ll immediately notice that these are highly circular in nature, meaning we need to switch between different areas routinely. In order to design an effective architecture, whilst minimising disruption, we need to have a clear vision of the future that we’re referring to as “10 miles down the road”.

📚 Versioned repositories

As previously explained, one of the built-in mechanisms to support a rolling release with breaking changes to the format is to introduce explicitly versioned repositories. Long story short, a local moss client will only “see” the latest version of the repository that it can actually use. Breaking changes will mean format bumps, and newer repository versions will be unavailable until the client has updated to a client/Serpent OS version that actually supports it.

This work requires changes to the repository format, infrastructure, and a migration path from the current repository to a new archive. However, it is also dependent on having a new format version being available for the versioned-indices.

🔒 Immutable OS

Frequently Serpent OS is described as an immutable OS, but in actuality it is an atomic OS. That means we’ve focused entirely on providing a reliable update mechanism, rollbacks, etc. We have not yet enforced immutability, aka a read-only root filesystem.

Our plan is heavily inspired by composefs. Currently moss produces new filesystem trees using a hardlink farm approach, and relies on renameat2() to atomically switch the root filesystem. This is a very fast operation, but does not provide immutability.

Leveraging the ability of moss to dynamically construct filesystems from packages, it will be abstracted into a driver mechanism. This will allow the classic hardlink approach for buildroots, but employ a new erofs strategy for immutable installs. The newly generated erofs image will essentially rely on extended attributes to “point” to the underlying content addressable storage, and utilise overlayfs to present a read-only view of the filesystem. Further building on this we’ll use so-called “mount tucking” to provide the mechanism for atomic updates without the need for a reboot.

🤔 Why not use composefs?

We’re actually going to make moss support exports to composefs format, making it an ideal match for podman use cases, deduplicating the image content. Whereas composefs produces content from an existing tree, we’ve already got content addressable storage, independent transactions and databases to facilitate this. This means we’ll be able to retain our composition features and very quickly produce the new erofs images. Using composefs directly would significantly slow down the time to produce each transactions, whereas we can instead build a similar internal functionality and integrate at higher levels.

We also plan to integrate the file digests in a new revision of the moss stone format, which will open the door to supporting SELinux xattrs, as well as fsverity for the immutable images.

Note that this goal will inform the requirements for the next stone format, which will in turn require versioned repositories. Of course, this also impacts the build pipeline infrastructure.

👷 Maintainer burden

Truly the most time consuming task in most distributions is the cost of maintaining packages. This is being approached, as with all of the planned goals, in an incremental fashion. However, it often pays to have a view 10 miles down the road to determine exactly how to get there.

🧬 ABI Tracking

In order to provide sanity for developers, and make guarantees about the stability of the system, we need to track the binary ABI for all packages. Furthering on this, we can of course immediately detect any breakage and schedule rebuilds as appropriate. This will also add a higher degree of confidence for pull requests as we’ll know if its consistent or introducing problems.

🔄 Pull Request workflows

The workflow for packaging right now is for someone to open a pull request, which contains their recipes, the automatic “build records” (manifest) and our assumption that they built it using a local repo based on top of the latest repository. What is actually needed here is for the infrastructure to perform automatic (low priority) builds of each pull request in a transient repository, making those builds available for verification purposes.

⚛️ Atomic repo merges

Another limitation with the current infrastructure is the reliance on a global build queue. Simply put, once a PR is merged, all of those builds are scheduled and individually published to the target repository upon completion. This is very naive, and leads to a lot of “nannying” in the event of a failed merge.

In future, those builds will be scheduled in a transient repository, and upon successful completion the resulting artefacts would be published to the target repository in a single atomic operation. This will also allow for the introduction of “try builds” for pull requests.

🩸 The bleed-through problem

Another issue which partially relates to bootstraps, is the bleed through problem. This is an artefact of using layered repositories, such as a local repository, to build updates that would change the providers available in the underlying repository upon merge. Take for example, LLVM.

main repo:
    ├── rust [runtime-depends: libLLVM-18.so]
    └── build-dep: binary(rust) [runtime-depends: libLLVM-18.so]

local repo:
    ├── rust [runtime-depends: libLLVM-19.so]
    └── build-dep: binary(rust) [runtime-depends: libLLVM-18.so]

After completion of libLLVM-19.so, the local repository would contain the new version, but the main repository would still contain the old version. Locally this would allow the circular build dependencies to work as rust builddep for rust could still resolve libLLVM-18.so. However, upon, merging to the repository, we have a new problem. The main repository would now contain libLLVM-19.so, but the rust package would still be built against libLLVM-18.so. This then breaks the dependency chain for rust, requiring manual intervention.

To solve this, we’ll ensure that filtered repository views are introduced both for local builds and our build infrastructure. This will mean that the index of the local repository will occlude the main repository, and eliminate bleed-through.

⭕ The bootstrap problem

Whilst ABI tracking is very important, it doesn’t actually solve the bootstrap problem of circular dependencies. This is a consequence of using an incremental development model for a binary-first distribution. In reality the only real way to solve this is by eventually altering the tooling to support a “world view”, with intermediate recipes that are not published to the final repository. In effect, combined with ABI tracking for dependency chain invalidation, we can determine which recipes need to run again as part of an invaldation-solver loop that relies on “cached artifacts”, i.e. the final packages.

This isn’t an immediate issue to solve, but it will inform our next steps to ensure we get to that point without requiring extensive overhauls or “stop the world” rewrites. Our eventual vision is that building any new package will feel much like it does now, but changing a reverse dependency will evaluate the dependent chain for rebuilds. To make this streamlined, the infrastructure will by this time have grown to a point where we can share the compute power by way of so-called “try builds”, and a shared cache of artifacts.

📦 Keeping the packages up to date

A large portion of the burden is in reality chasing updates. Currently we have a very simplistic utility that cross-references release-monitoring.org to spit out a list of recipes that have available updates. Truthfully we could easily generate diffs for these updates, and even automate the process of updating the recipes. This is dependent however on having confidence in the pull request, which in turn is dependent on ABI tracking being in place.

🗂️ Organising it all

We have a complex series of interdependent goals, which require research and frequently shifting between focus areas. Not all of them will be immediately realised, and some are simply informing the direction for our future. Additionally, we need to keep moving without any “rewrite the world” pauses.

It’s important to highlight that none of this is a reinvention of the project, but a series of steps to “1.0” and beyond to ensure that the project is sustainable and reliable. This means that delivery continues, and the delivery itself becomes simpler and more trustworthy.

Despite the interdependencies, once we’ve finished scoping the end requirements we can work towards the immediate concerns:

Pull request workflow
Repository consistency (initially ABI story)
Update automation
Immutability

Note: These timelines are estimates and may adjust based on available resources and community support.

Scale-up considerations

We’re at the point where we need a functional OS for contributors to be able to work on the project. It also presents a paradox because we wish to keep the project lean whilst we work on extending the scale-out capabilities. In reality this means that we’re going to permit a limited amount of repository growth:

Addition of the Plasma desktop for daily use
Container-management tooling (podman, docker, etc)
Development tools (IDEs, etc)
Onboarding requirements (such as Matrix clients, documentation, etc)

This in turn also requires a more reliable installation experience, which is why we’ve been spending cycles on the installer project. Internally we want the installer to support a variety of scriptable configurations in order to provide better test coverage for our updates and boot management, whilst we also intend to incrementally improve the user experience.

📅 Conclusion

Alpha2 is coming, and will feature a preliminary version of the updated installer. The plan initially is for a simple “use whole disk” strategy and will be followed up with supported for encrypted installations (post alpha2). Quite simply, some employers require their developers have encrypted installs, and not supporting this configuration will restrict our ability to onboard developers.

We’re doing the Alpha2 as a baseline, from which all update testing, infrastructure adjustments, etc, will be measured against. This is the point at which the project will formally “open the doors” for more developers and testers, with the explicit view that growth is sustainable and manageable. Per the requirements set above, we’re not going to “explode”, but enable specific workflows that in turn will permit us to scale out the project and inform direction.

It’s important to me that we’re explicit in how we take our next steps. As stated earlier, pausing the world is not an option. Continued delivery whilst enabling an incremental path to the future is the only way to ensure that we can deliver on our promises. Contrary to a reboot, this is a solidification of our vision and a commitment to the future. Sure, new features will also appear in time, such as easier packaging recipe formats, but no decision is being made that will require a rewrite of the project. However, these goals require stating and explaining up front so that the high level vision is clear and provides the context for any short-term deliverables. The last thing we need is to be in a position where we’re “stuck” and unable to deliver, due to a lack of foresight.

🙏 Thank you

On behalf of the project, I’d like to thank everyone who has supported us so far. Whether that’s through donations, code contributions, or even having the time of day to try out the ISO, it’s deeply appreciated. Being the underdog comes with its own set of challenges, and of course the “indie cost”, but it’s also an opportunity to challenge the status quo and find new, more optimal ways, to do things.

We’re not just building “Serpent OS”, we’re building a set of tools to solve the problem of distributing an OS. Way down the line this will be powerful enough to support a number of use cases, from appliances to desktops, and even servers. In short, we want to bring high quality tooling to the masses, and we’re doing it in a way that’s sustainable and reliable.

Offline Rollbacks Enabled

Jan 4, 2025

Ikey

Founder

If a picture tells a thousand words… well then this video .. Yknow what, sod it. Here’s a video of the offline rollback support in action, as tested in a newly installed VM using xfs for the root filesystem. We jokingly refer to this as the “LTT” test due to LinusTechTips’ “interesting encounters” with package management systems in the past.

This has already landed in our volatile repository, so simply sudo moss sync -u and grab the latest updates. Any subsequent transactions will automatically generate the boot entries for you, no intervention required. Or mounting. Or anything. Just update and go. Srsly.

We actually saw this as a golden standard for the feature, as it’s a real-world scenario where you’d want to roll back to a known good state. The video shows the system booting and recovering in multiple ways, from complete system nuke (via glibc) and simpler scenarios like removal of the GNOME desktop environment.

The intent is to ensure a user can quickly revert to the last transaction that worked in the instance an update goes awry, without needing to boot into a rescue environment or live CD.

Like what we’re doing?

Developing a project with the scope of Serpent OS is no small feat. In order to maintain our current cadence, and to land huge features like the offline rollback, or impending system model work, we need your support. I’m as happy as the next person to return to the job market once my medical conditions ease up, but the simple matter is this will greatly slow down our recent progress.

Our aim is to entirely redefine the distribution of Linux, and the years of prior effort are finally coming together rapidly to make this a reality. If you like what we’re doing, consider supporting us!

Visit our sponsor page for more information on how you can help us grow and deliver the future of Linux distributions.

Technical Details

This is all achieved using moss’s internal content addressable storage for deduplication, where each old transaction is swapped with the live /usr at the time into the archive.

The behaviour is implemented in the initramfs (dracut) by invoking moss state activate $transaction_id -D /sysroot, ensuring that the activation is performed before the new root is mounted and the system is booted. We actually shrunk our moss and uutils-coreutils builds to half their usual size so that we can fit them comfortably into the initramfs.

For now, we’ll automatically generate entries on the ESP (or XBOOTLDR) for the last 5 transactions, and you can select them during early boot to perform the rollback. It’s not slow, as it’s just doing yet another renameat2() to atomically exchange the live /usr with the archived transaction.

TLDR: Boot time rollback, no network required, no live CD required, no rescue environment required. Just a working system.

What’s next?

The system model is coming next, which will expose the non-imperative core of the architecture to users. The usual commands will continue to emulate an imperative OS, by mutating the system model and applying the internal sync operation. This ensures dependencies are resolved only according to the layered repository configuration, without special exceptions for “installed” (which.. we spent a lot of effort faking this behaviour).

It’s good to remember that moss is effectively a huge cache, and that each state has no relation to the other. We simulate upgrades by building a new state with the same selections as the old one, then begin to upgrade those candidates.

The new model (somewhat Gentoo inspired ❤️) will allow moss to build the dependency graph for the finished system state. As a sneak preview of what we’re enabling, imagine the newly landed transactional reboot code linked with multiple system models… such that you might decide to moss sync from your existing model to an entirely different one, rebooting from your GNOME desktop to a KDE desktop, for example. Cleanly, atomically, and without any manual intervention.

Investing in the Future

Dec 31, 2024

Ikey

Founder

Here we are, at the end of 2024, and what a year it’s been! We’ve made significant progress on Serpent OS, and we’re excited to share some of the highlights with you.

Attained Alpha

Virtually 5 years in the making, we recently attained alpha status. Our tooling and concepts have aligned, allowing us to now rapidly iterate on the core deliverable itself: Serpent OS. We’ve seen an explosion in cadence, with our tooling enabling us to quickly and easily deliver updates, new packages and enabling new features.

Since the Alpha 1 ISO (0.24.5), we’ve fixed a number of issues in relation to hardware and booting. We’ve also added new features to moss (refined boot management) and boulder (our build system). Even now we have multiple PRs open to tidy up moss, add automatic generation of monitoring.yaml files, and more.

Keeping the cadence

We’ve been working hard to keep the cadence up, and we’re now at a point where we can deliver new ISOs on a regular basis. Our main focus now is the tooling, in enabling a far more automatic delivery pipeline. The vision is having automated pull requests generated for package updates, automatically handling changes in dependencies and ABI, and ordered by build tier. This will significantly reduce the maintainer burden, in conjunction with planned security monitoring.

Short term deliverables

In the short term we’re going to land the following features:

Offline Rollbacks - We’re going to land offline rollbacks in the boot menu, allowing you to easily revert to a previous system state.
Versioned Repositories - We’re going to introduce versioned repositories, allowing us to gracefully handle format changes in moss and boulder, permitting an “update forever” epoch-staggering approach. Additionally, this will allow users to stay on an older repo version if they’re awaiting a fix to a regression in a newer version.
Improved Documentation - We’re going to improve our documentation, making it easier for new users to get started with Serpent OS.
Tools as a service - We’re adding IPC layers to our tooling, allowing them to be launched as helper services for integration in other applications. This will allow us to integrate moss with GNOME Software and COSMIC Store, for example. Additionally it will ensure the latest version of the moss binary is always used by the system, gracefully handling version repo changes. Chiefly, this will allow moss to cheaply integrate anywhere we need it, including for lichen.
Lichen Improvements - Lichen will also be converted to an IPC based backend, allowing it to be executed via pkexec for integration into a GUI (or even CLI) frontend. This is a necessary step to allow the frontend to operate without privileges, facilitating live updates to the NetworkManager configuration, or indeed keyboard layouts. Long term we’re planning a cage based kiosk with a full screen installer mode, on a far smaller ISO.

Long term vision

Our long term vision is to deliver a system that is easy to use, reliable, and secure. We’re building Serpent OS to be a system that updates forever, confidently. We’re also shaking things up by being first-adopter of new technologies as default solutions, with a view to enhancing the robustness, origin-diversity and security of the system through Rust based replacements and alternatives to key components. You can already see this with our adoption of sudo-rs, coreutils (uutils), ntpd-rs, and not to forget rustls by default for our curl build.

Outside of that, moss, boulder and the supporting crates are also written in Rust, giving us a strong foundation for the future.

Serpent OS is a project that is built to last, and challenge the status quo. Deliver a Linux distribution for use on multiple verticals, with the feeling of a conventional Linux distribution, but with key management facilities and architecture designed to make a rolling release distribution that can be trusted for long term use, extensive updates, and the ability to entirely re-engineer the entire OS, delivering in a safe atomic update without breaking client installations.

How you can help

In order to give Serpent OS the development time it needs, we’re looking for sponsors to help fund development. If you’re interested in becoming a sponsor or investor, please do not hesitate to get in touch. We want to secure funding for development and infrastructure for the next 2 years.

While of course we need to iterate on our own tooling, we also need to be able to work more upstream and with kindred projects to drive the ecosystem forward. In time, we wish to become the incubator for new technologies and solutions, and we need your help to do that.

With your support, we can expand our builder network and enable more hosting, bandwidth, and compute resources. This will in turn enable our automated rebootstraps, which will of course allow downstreams to extend upon Serpent OS as a kit-distro. Indeed, it’ll allow us to expand beyond x86_64 when the time is right.

While Serpent OS currently delivers a desktop ISO, this is not the limit of our potential. As indicated, our priorities in the short term are designed to massively aimplify project maintainer bandwidth and free up more time for feature iteration and development. Once we’re quite happy with the base (and we’re not far from that point) - we’ll quite happily extend to other desktops, and even server/cloud offerings.

Nothing is bullet proof, but with the architecture and tooling of Serpent OS we can get pretty close. Gotta admit, having a transactional, atomic OS with rollbacks, and the full granularity of a conventional package manager, is pretty cool. 😎 Especially when things “Just work” out of the box thanks to the stateless (“hermetic usr”) requirements of the system.

Know what’s even cooler? When we land our system-model implementation, you’ll be able to completely duplicate the setup between local development and production. When we also land our export/compose functionality in moss, you’ll be able to reap these benefits in containers too. 🐍

Here’s to a great 2025, and we hope you’ll join us on this journey. 🚀

Alpha Refresh Available

Dec 29, 2024

Ikey

Founder

In response to feedback from the community, we’re issuing a refresh of the Alpha 1 release, 0.24.6. This refresh includes a number of key updates and improvements, including:

Now works with etcher
Fractional scaling enabled by default
Prebuild icon theme caches to speed up trigger execution (except for hicolor)
Add new moss boot status command for introspection into blsforme functionality
Fix amdgpu initialisation for older devices, previously leading to black screen on boot

Fractional scaling, much loved

Clarification on moss boot management

Frequently in Team Serpent we joke that moss is the “systemd of package managers”. It actually does a whole lot more than is easily visible.. for example:

Content addressable storage for the entire OS
Atomic updates of /usr by renameat2() w/ RENAME_EXCHANGE
Executes transactional triggers in a namespace (container) with RW /usr and RO binds of /etc and /var …
Executes system triggers postblit, with dependency ordering..
Via blsforme, manages the ESP, XBOOTLDR, boot entries, sync and promotion of bootloader, kernel and assets, initrds..
Entirely zero configuration. No /etc/default/grub, no remembering your root={}, it’s determined automatically.
Basically does all the work of OS provisioning, barring disk mounting and partitioning. Our installer, lichen, is basically a thin wrapper of moss.

We’ve had no debugging capabilities around the boot management, so we’ve landed moss boot status command. It’s super primitive but does offer an insight into the introspection capabilities of blsforme. In this simple screenshot, moss/blsforme are supporting the Boot Loader Protocol, querying the systemd-boot entries in efivars. When found, LoaderDevicePartUUID is used to determine the boot ESP, but moss/blsforme will quite happily read the GPT directly (relative to / mount) and find the ESP and XBOOTLDR…

Remember - we happily use systemd-boot. We do not use bootctl or kernel-install. Entries are automatically managed and we’re extending this to support offline rollbacks via boot menu (feel free to extend with /etc/kernel/cmdline.d/*.cmdline drop-in snippets!) Also note we do not allow systemd to automount the ESP or XBOOTLDR via the GPT autogenerator, instead moss/blsforme will automatically discover and if necessary, mount these partitions, when dealing with bootloader or kernel updates.

With this explicit control over the boot management, we’ll happily be landing secure boot and fwupd support in the near future, without the downsides of trying to provide EFI assets to /boot in a package itself. 😬

moss boot status

A note on virtual machine usage

It must be clearly stated that Serpent OS does not support anything other than UEFI booting on x86_64-v2+ capable hardware. Please note that GNOME Boxes will default to BIOS, not UEFI, and will need hand-mangling of the XML file. Also note in other solutions such as VirtualBox you will need to specifically select EFI in the VM settings.

In order to maximise compatibility with other tooling and firmware, we’ve modified the generation of our ISO to include the isolinux MBR/GPT hybrid components, however we deliberately did not include a functioning bootloader for BIOS boot. Instead, you will see a boot failure indicating that ldlinux.c32 could not be found, which is a sure-fire way to know your VM or hardware is incorrectly configured.

Invalid VM configuration

Installer notes

Lichen is due to receive some love in the next few cycles, so please be aware of the following limitations:

A pre-configured GPT disk is required before lichen is launched
A working internet connection is also required, this is a network installer
A FAT32 formatted EFI System Partition is required (marked as esp in gparted)
For optimal usage, or when your ESP is tiny, please create an XBOOTLDR partition:
- Create an FAT32 partition in gparted of 2GB in size
- Mark it as bls_boot in gparted
Further filesystem support for XBOOTLDR will be added to blsforme when we’ve finished packaging efifs.
XFS or F2FS are strongly recommended for the rootfs, due to the limitations of ext4 with hardlink counts.

COSMIC Desktop

We’ve made the decision to pause our offering of COSMIC Desktop ISOs. We’re huge fans of the project, and we strive to keep it up to date and functional. However, it is very early in alpha, and so are we. It doesn’t make a lot of sense for us to extend our workload to two alpha codebases!

We still offer COSMIC as an option in the installer for those who wish to try it out, and do encourage testing and engaging upstream with the developers. For now, we will offer just the GNOME ISO.

Lichen + COSMIC

Next up

In the new year we’ll be focusing heavily on our tooling to facilitate a scale up of our packaging efforts. We’re quite proud of our tooling, but we’re not oblivious to the warts and issues. The next couple of weeks will be spent improving our documentation (or making it exist!) and making sure that we have the ideal workflow in place to permit large stack updates along with the ABI consistency verification we need. This will also pool work for ent, moss, boulder, by streamlining packaging, and making it trivial to update an entire stack of packages in one go. Eventually this will be an automatic thing, with successful PRs being verified and then merged into the volatile repository.

Don’t forget, versioned repositories are also coming your way, and very soon we’re landing offline rollbacks in the boot menu! blsforme has been significantly restructured for cmdline handling, and understands entries in the context of a distinct system root (ie moss filesystem transaction) - which is the bulk of the work needed to facilitate offline rollbacks. The last piece will be running moss at an early stage in the initrd to swap the /usr pointer to the older transaction, which is instanteous and atomic.

Final thoughts

5 years and no release, and now two alpha ISOs before the end of the year. We’re pretty happy with that, and we’re going to keep surprising you next year with more releases, more features, and more stability. Happy new year from all of us at Team Serpent! 🐍

If you like what we’re doing, please consider sponsoring us - this will allow us to grow our infrastructure and deliver builds quicker. Due to health issues in the last few months, I was forced into a position whereby I had to resign from my job. Any and all contributions to the project are greatly appreciated, and will help us to continue delivering what folks are frequently now calling “the cleanest Linux implementation they’ve ever seen”.

Try it. You might like it. 🚀

Serpent OS Enters Alpha

Dec 23, 2024

Ikey

Founder

Following a successful prealpha phase, we’re excited to announce that Serpent OS has now entered the alpha stage of development. This milestone marks a significant step forward in the project’s journey. While the usual disclaimers apply, as a certain level of project fluidity is to be expected, we invite you to explore the latest alpha release and experience the cutting-edge features and improvements we’ve introduced.

Serpent OS

Supporting Serpent OS

Serpent OS is a community-driven project, and we welcome contributions from developers, testers, and users alike. If you’re interested in helping shape the future of Linux distributions, we invite you to join our growing community and get involved in the project. Whether you’re interested in packaging, development, documentation, testing, or community support, there are many ways to contribute to Serpent OS.

Lastly, you can support the project by becoming a sponsor. Your support helps cover our hosting costs and allows us to dedicate more time to making Serpent OS exceptional. We’re grateful for the support we’ve received so far and look forward to continuing to build something meaningful in the Linux ecosystem.

Sponsor Serpent OS

What is Serpent OS?

Serpent OS is a heavily engineering-led Linux distribution seeking to redefine how we distribute Linux. It is a stateless OS that leverages atomic updates, cutting-edge tooling, and rock-solid reliability to deliver a safe and efficient system. Built by industry veterans with decades of experience, Serpent OS represents the next evolution in Linux distributions.

We currently offer x86_64-v2 desktop builds, for both GNOME and COSMIC desktop environments. We only support UEFI systems, and plan secure boot support via shim in the near future. The majority of the distribution (including the kernel) is built against the LLVM toolchain, with libc++ as the default C++ standard library.

Through our unique architecture, we’ve combined proven concepts into a cohesive system where updates either fully complete or safely roll back, keeping your system running reliably. Our atomic updates let you see changes immediately, without requiring reboots just to apply updates. While certain updates like kernels will still need a reboot, we’ve designed Serpent OS to be as seamless and user-friendly as possible.

What’s new in the alpha release?

The alpha release introduces several key features and improvements that enhance the overall user experience and system performance. Here are some highlights of what you can expect from Serpent OS alpha:

Hardware support

We’ve rounded out our hardware support, now including patches for ASUS devices and Surface devices. Additionally, we now have the NVIDIA driver available in the repositories. Note we only offer open-gpu-kernel-modules, which we prebuild against our linux-desktop package to streamline installation.

Steam

You can now install Steam from our repos, making use of our newly enabled multilib drivers:

mesa-32bit
nvidia-graphics-driver-32bit

Steam on Serpent OS

More Rust

As part of our efforts to modernise and strengthen the Serpent OS base, we’ve switched some components out for Rust alternatives:

uutils-coreutils replaces coreutils
sudo-rs replaces sudo
ntpd-rs replaces timesyncd
curl is built with rustls support (and hyper but this is being dropped upstream)

Additionally we include starship by default on our GNOME ISO, along with zed, loupe, resources and other Rust applications.

Packages

Our recipes repository has seen over 1600 commits since our prealpha, with some of the highlights including:

Linux 6.12.6
Firefox 133.0.3
LLVM 18.1.8
GNOME 47.2
COSMIC 1.0.0_alpha4

Tooling updates

We’ve made significant improvements to our tooling, including updates to moss, boulder, and blsforme. These tools are essential for managing packages, building software, and handling boot management in Serpent OS.

moss - Package Management

Our atomic package manager swaps the entire /usr directory during system updates, ensuring a stateless, bulletproof upgrade process. Updates either succeed completely or not at all. Recently we made improvements to moss to handle certain special cases:

EMLINK - ext4 is limited to 65k hardlinks per inode, and we were hitting this limit on some systems due to empty files. We now handle this case gracefully by creating new inodes for empty files.
ENOSPC - We now mitigate the risk of ENOSPC on the boot partition by performing automatic cleanup of old kernels and initrds. This paves the way for us to land transaction-specific entries for the offline rollback feature.

blsforme - Boot Management

Automated boot management that just works. Seamlessly handles your EFI System Partition and boot entries without manual intervention.

We’ve improved blsforme to support drop-in command line snippets for kernel parameters, allowing for easy customisation of boot entries. These can be supplied by packages, or by the user directly. This has allowed us to make plymouth work out of the box, in tandem with our prebuilt reproducible initrd images.

We also now support multiple initrd images, which in turn allowed us to land support for early KMS in the initrd for NVIDIA users.

lichen - Installer

We’ve revamped the UI for Lichen, and now allow picking between xfs, ext4, and f2fs for the root filesystem. The team advises against ext4 for systems where more than a few hundred transactions will need to be retained, due to ext4 limitations with hardlink counts.

Sync protections

In preparation for “versioned repositories”, we’ve added a gate to prevent the automatic builds on our infrastructure from directly reaching users. Once we’re happy with our validation, we then “sync” the repositories to the public mirrors. This ensures that users are not exposed to any potential issues with the latest builds.

What’s next?

With us now entering the alpha phase, we’re looking to grow our contributor base and community. Our immediate focus is improving/creating documentation to enable more people to use Serpent OS as a daily driver in the near future. We’re also going to keep iterating on our tooling, ensuring a reliable and efficient system for all users.

We’ll begin deeper integration of the tooling, such as moss with packagekitd for GNOME Software and COSMIC Store use.

Issues and feedback

If you encounter any issues or have feedback to share, please don’t hesitate to reach out to us. You can report bugs on our GitHub issue tracker or join our Matrix space to chat with other users and developers.

We are aware of some known issues in the alpha release, and we’re actively working to address them in future updates. Your feedback is invaluable to us, and we appreciate your help in making Serpent OS the best it can be.

Performance of moss reduced significantly in VMs on first run
NVIDIA driver not working on some systems
Lichen still deliberately limited to “seeing” pre-formatted partitions. Requires GPT disk, with ESP and / partitions pre-created.
Lichen locked to net install mode, a moss export functionality will be added in the future to address this.
XBOOTLDR partition (fat32, bls_boot flag in gparted) of 2GB recommended for ideal boot management
Small repos - deliberate, to ensure maintainer story is solid before we scale up
Secure boot support pending secure key distribution support within the build infrastructure
Due to how our atomic upgrades work using renameat2, GNOME Shell doesn’t currently detect new applications (.desktop) files when installed, requiring a reboot. We’re working on a fix for this.

Finally

We’re excited to share the alpha release with you and look forward to your feedback and contributions. Thank you for your continued support and interest in Serpent OS. We’re committed to building a safe, efficient, and reliable system that empowers users to get the most out of their computing experience.

Serpent OS Prealpha0 Released

Aug 1, 2024

Ikey

Founder

Well, it didn’t take us that long, really … Our technical preview, prealpha0, is now available for testing!

wip boot code

Head to our download page to grab serpent-os-prealpha-0.iso now!

This is a super rough version of Serpent OS that is capable of being installed on baremetal hardware and VMs that support UEFI and OpenGL acceleration. It is however not recommended for use daily due to the early nature and a bunch of packages being wildly out of date. It features a minimal GNOME desktop with the Firefox web browser and a terminal.

Right now it contains a CLI installer that can be accessed from the terminal by typing:

sudo lichen

To detract from casual use it is necessary to manually partition the disk before running the (net) installer. You can use fdisk to create a GPT disk with a (mandatory) EFI System Partition and an optional XBOOTLDR partition. This currently also needs to be FAT32 until we integrate systemd-boot driver support. Lastly of course you will need a large enough root partition.

You will need a working network connection, so make sure you’re connected before starting the installer!

Technologies involved

This will of course appear to be a very rough (crap) prealpha ISO. Underneath the surface it is using the moss package manager, our very own package management solution written in Rust. Quite simply, every single transaction in moss generates a new filesystem tree (/usr) in a staging area as a full, stateless, OS transaction.

moss

When the package installs succeed, any transaction triggers are run in a private namespace (container) before finally activating the new /usr tree. Through our enforced stateless design, usr-merge, etc, we can atomically update the running OS with a single renameat2 call.

As a neat aside, all OS content is deduplicated, meaning your last system transaction is still available on disk allowing offline rollbacks.

Translated: Like A/B swaps? Don’t like rebooting? … Ok that explained it.

blsforme

A few crates deep we find blsforme - a library for automatically managing the ESP and XBOOTLDR partitions. Whenever kernels and associated files are present in the OS filesystem, they are synchronised to the boot partitions along with automatically generated boot entries and cmdlines. Specifically this means moss can discover and mount necessary partitions according to the disk topology, GPT entries, and Boot Loader Specification, generating configurations by scanning your local rootfs to build the proper root= parameters for you.

Translated: Magic boot management makes way for offline rollbacks.

Warning

Super pre-alpha. Will 100% break! We just wanted you for the journey <3

Start of public iterations

Ok you have this super rough ISO, what next? We now have an actual startpoint and will continue to iterate on the ISOs, delivering new installer updates/improvements and removing the unnecessary live mode from the ISO completely.

The next release will also feature more installer options so you can fresh-install the Cosmic Desktop (in repos now) For the brave, go forth and sudo moss sync -u ! (ok you want moss help first.)

Closing word

As a distro, it’s kinda crap right now. The tooling has been our focus for years and now we can actually build something with it. With only a handful of packages, flatpak is your best friend. Or you could swing a PR into our recipes repo!

The project is only possible with your support. Something tells me that putting out this ISO is going to somewhat increase our hosting costs and stress the datacenter hamsters.

Feel free to sponsor to support our work and increase our capacity!

Calm Before The Storm

Apr 30, 2024

Ikey

Founder

It’s that time of month again, and we have some details to share with you on boot management, as well as plans for real installs landing in May. Additionally, plans for “deferred updates” via mandatory reboots have been dropped!

wip boot code

Project priorities

To better understand the technical deep-dive on boot management further below, we should clarify a slight change in priorities for the project as a whole. For some time now we’ve served as a technical beacon for Solus to guide them towards better strategy and technical solutions whilst awaiting a situation where a rebase is feasible.

While we still look forwards to that future, the present day is the most important to us. Serpent OS must now transition from a collection of tools and promises into a daily driver with actual users. We still aim to cater to the more developer/technical oriented end of the spectrum rather than a “desktop distro”, and want to start this journey yesterday.

The boot management code is currently landing over the coming days, and will be immediately followed by an incredibly rudimentary system installer to facilitate an initial adoption of Serpent OS for the general public. Our intent is to ship some early test ISOs for supporters, and by the end of May have a feasible option for people to use and contribute to Serpent OS on real hardware.

We acknowledge this is an exchange of the pursuit of perfection for a pragmatic rapid iteration cycle and believe the best way forward is to open the doors wide, fixing issues on the job. The initial system will likely be rough and buggy, but it is an official starting point from which we can all build Serpent OS together.

Moss + Boulder

Our Rust tooling has now been in “production” use in our development environment and build servers for the last month without any major hiccup. We have recently discovered some minor issues in our emul32 (32-bit) package generation which are being resolved, chiefly our move from the improperly named ISA x86, now updated to 386 affecting 32-bit dependency chains.

Boot

Our main topic of conversation this month is the enabling of boot management for Serpent OS installations. To be crystal clear, we are not talking about a boot loader. In fact, Serpent OS uses systemd-boot for UEFI. Instead, we are talking about the automated management of boot loader entries, their corresponding kernel/initrd assets, cleanup from old states, etc.

In a former life I authored the clr-boot-manager tool to manage this in an agnostic fashion and this served well for a number of years. However, existing as an externally invoked binary meant that the tool had to become responsible for all accountancy and asset lifetimes.

Enter … moss-boot

Ok - technically it’s crates/boot in the moss repo. This is a Rust crate that will be integrated into the main moss project to provide the aforementioned boot management facilities. A huge advantage for us is that moss can only allow a single version of a package to exist within a given state, and all states are numerically identified due to the transaction nature of our package manager.

In simple terms, one “state” == “one kernel candidate” (per type..) so all we really need to do for now is identify the ESP in use by way of the Boot Loader Interface and any XBOOTLDR partition. Then, map our “boot environment” into something moss-boot can convert into boot entries, and voila..

At a more technical level, moss-boot will handle the following:

form candidate cmdline from filesystem snippets
auto-generate the root= parameter by probing the / partition (LVM2, etc)
record the initrd needed
compare and atomically write “missing” files in FAT-respecting fashion (copy/unlink/rename)
garbage collect unused boot entries

Live atomic updates and kernels…

An interesting, perhaps not widely considered result of upgrading a kernel, is filesystem path changes. If for any reason the kernel’s module tree becomes inaccessible (say, due to updating to a new version) then it becomes impossible to load the old kernel modules. To remedy this, distributions will tend to leave multiple versions of the kernel on disk, invariably within mounted /boot.

The approach Serpent OS is taking will mean that like Solus, /boot will not need to be premounted nor will any package ship files in that directory. Unlike the existing design in Solus 4, we will not leave kernels behind on upgrade. Instead, they become locked to the system state (transaction) in which they are used.

To solve the “missing version” issue, we will have moss record the qualified snapshot path (in /.moss) within the /run tree so that once the /usr tree has been swapped with a new transaction atomically, a patched kmod package will fall back to probing the suddenly orphaned kernel tree in the cached /run location.

This will mean that the /lib/modules tree may not have the current kernel version, but the OS will still be usable while having had a live atomic update. Of course, to use the new kernel you must reboot. Unlike other atomic OS implementations, it will be up to you when you do so: no more deferred updates!

Next time

We are aiming to get back to monthly blogs on the 19th, but the cycle for the coming 6 weeks may be quite interesting. Expect out of schedule posts looking for testers and volunteers while we use Serpent OS to build Serpent OS, enabling a community of pioneers benefiting from the unique design of our architecture: a full-featured OS featuring hermetic /usr, offline rollbacks, and live atomic upgrades with containerised system triggers.

The March Of Progress

Mar 31, 2024

Ikey

Founder

Despite a brief excursion out of the country for a first-in-a-lifetime holiday, I’m happily back at the desk to bring you up to date with the latest goings in Serpent OS. TLDR: Loads of awesome, baremetal is enabled, ISO cycle in next couple of weeks.

GNOME on Serpent OS

Tooling Improvements

OK, we all know you want to know about the baremetal stuff, but c’mon, check out this changeset!

Boulder

We’re officially at a point where the Rust boulder implementation has become our “blessed” build tool. This is actually a crate within the moss repository, allowing the two tools to share huge chunks of code. Our focus over the past few weeks has been to ensure we can drop the tool into our existing build network and “just work”, while making it significantly faster.

One of the most important high level changes is performance, in some cases virtually 50% faster packaging times. Our package payloads are now compressed using multi-threaded zstd compression, and a bunch of internal components were reworked to use faster code paths leading to a significant speed up in the raw generation of each .stone package.

!!! warning “Legacy boulder tool has been retired”

As of March 31st, 2024, the legacy D implementation of `boulder` is no longer supported. The runtime component, `mason`, has been removed from the
volatile repository. Our new tool is written in Rust and contains the `moss` code, enabling a self-contained all-in-one approach to buildroot  management.

It relies on clone-based user namespaces, so is fully "rootless". Please migrate immediately to the new Rust based tool. Not only is it more advanced,
it is significantly faster at generating the raw `.stone` packages.

Comparison

Other, quite awesome changes:

Fix manifest encoding for 0 nul byte, interop with build systems
Explicit --update flag to refresh the repositories prior to build
Defaults to multithread zstd compression
Slick timing reports
Base-enabling landed for “meta” packages (contain no files or content)
Added BuildRelease control via tooling for automated package rebuilds

Moss

Moss also got some love these past few weeks, including a full migration over to diesel.rs. Combined with a limiting of async to largely network bound operations, and some clever performance optimisations under the hood, moss is now faster than ever. In fact, we’re able to generate desktop ISOs in under 20 seconds, and install the full GNOME desktop to a virtiofs VM in almost no time.

Dropped some unnecessary dependencies
Significant code cleanups
Further refined our system + transaction triggers to fully respect dependency ordering
Augmented state management with new functionality to activate/rollback to a specific transaction by ID using offline store.
Various optimisations:
- Drop expensive Path usage in vfs crate
- Optimise queries in our transaction management to speed up DAG initialisation and filesystem blit
- And more to be covered in a future blog post

Hardware/Desktop Enablings.

Rather than extensively widen our repository at this point we’ve opted for further low-level enabling of components required for the developer experience desktop. This includes Network Manager, the desktop kernel package, Vulkan + mesa integration, etc.

Gnome Software

As well as enabling core applications and features such as GNOME Software and Flatpak, we’ve also enabled preliminary usage on baremetal booting hardware. Note, this isn’t yet installable but we do plan to offer early access installable images to our sponsors!

Also worth noting.. we’ve begun to package the Cosmic Epoch desktop from the cool guys over at System76. We’re highly interested in this project and plan to track progress closely, offering it within Serpent OS as a fully featured development focused desktop experience.

The Agenda

This post was finished moments before the March deadline, so needless to say we’ve been super busy! Thankfully a lot of pivotal pieces have landed now and we can focus on some big ticket items. Our primary concern is to ship one last “pre-alpha” image for the public, and a test installable pre-alpha ISO for our sponsors to help gain early feedback. The hope is this will help fund a widening of our infrastructure to support more concurrent repository users and our own debuginfod compatible extensions to our repository.

The April (and beyond) agenda:

Finish pre-alpha stage with some shiny ISOs!
Enter alpha by adding versioned repository support to moss for an “upgrade forever” guarantee
Enhance our ABI tracking story to vastly simplify huge rebuild queues and automation
Expand our native hardware support
Add some missing basic triggers such as Fedora-compatible SSL certificate management
Begin work on the all-important boot management in moss! (not boot LOADER, this will be systemd-boot)
Add battery testing of our tooling to our CI pipelines to protect users
Ensure all of our code is fully documented
Greatly improve our onboarding and documentation for packaging and usage of Serpent OS, focusing on stone.yaml in depth
Get core team running Serpent OS daily. WiFi and GPU support are in place already.
Build an amazing developer experience

On behalf of the Serpent OS team, I look forward to sharing some amazing updates with you towards the end of April! Keep your eyes peeled for the ISO drops in the coming weeks.

End of February Update

Feb 29, 2024

Ikey

Founder

import { Aside } from ‘@astrojs/starlight/components’;

This update came a little later in the month, as we’ve got a lot of exciting news to share. Everything from boulder in Rust, to the GNOME 45 Desktop complete with moss triggers built atop a rebootstrapped toolchain.

Rusty Boulder

We’re pleased to announce that over the course of this weekend, once testing has completed we’ll deploy the latest version of boulder, our packaging build tool. This has been given the Rust treatment, directly sharing the codebase with moss.

Boulder in action

Unlike our PoC implementation, the new boulder makes use of the clone syscall to execute portions of itself under user namespaces, eliminating a longstanding deployment irritation of having the main build binary inside packaging.

Just like moss it makes use of tokio for async right where we need it, and is rootless to make life easier for our contributors. Far from being just a direct port of the proof of concept, it adds new tools to empower developers, such as breakpoints in packaging recipes.

Breakpoints, in recipes

Triggers, integrated!

After much discussion we finally integrated support for triggers in moss. These are actions that are executed at different stages of package management operations to finalize or “bake” some state based on the unified components of the installation.

Note that right now triggers are re-executed for each transaction, and no caching support is yet in place. With that said, they’re still very fast and we plan to add a store based cache to prevent unnecessary execution. One of the greatest benefits of our triggers is ensuring these actions run within the right context under a namespace, or container.

In pursuit of stateless

We’ve iterated many times over the years our belief in the stateless philosophy, and as a design decision we mandated that packages can only contain /usr files. This has meant we’ve had to get a bit inventive in shipping working packages out of the box.

One such strategy has been to abolish the reliance on package triggers for system users and groups. In the past we’ve relied on systemd-sysusers to construct these accounts, such as gdm. A downside is there is no simple way to handle the cleanup of these accounts.

We’re now happily using systemd userdb user and group drop-in records via nss-systemd to provide the default users + groups (including users) group with centrally managed, fixed UIDs and GIDs in our git recipes.

GNOME Desktop

Thanks to triggers now being fully integrated in Serpent OS via moss, we’ve actually managed to package up most of the basic parts of GNOME 45. This includes a functioning GDM, GNOME Shell and a handful of applications.

GNOME on Serpent OS

Packaging work

Outside of the heavy work on the tooling and GNOME, we’ve also been tidying up our recipes repo and recently performed a fresh rebootstrap using:

glibc 2.39
binutils 2.42
GCC 13.2.0
LLVM 17.4.0
Rust 1.78.0

Note that our primary toolchain is glibc, lld, clang / clang++, libc++, libunwind, etc. The majority of packages are built with this LLVM + glibc toolchain, and a subset of packages currently build with the GNU toolchain due to various wrong assumptions or use of GCC extensions.

Our kernel has always been built with clang.

Images

Our shortterm plan is to deliver a prealpha image for the general public to test, and to get a “feel” for Serpent OS and associated tooling. As soon as this is out of the door we will immediately pivot to getting Serpent OS fully functioning on baremetal, with explicit targets in mind: the hardware that core developers are currently using.

These images will be available but not promoted for general use - given their intended use as dogfooding systems. This is part of our shift to the oxide-alpha-1 target milestone, which will be discussed further in our next post announcing the prealpha image.

Next time

Our next monthly blog post should return to its regular slot, with updates on the following topics:

Limiting async in moss to targeted areas (PR ready and pending review)
Life on Serpent: How baremetal testing + enabling is going.

Until then, please, stay tuned!

January Updates

Jan 19, 2024

Ikey

Founder

Precisely one month since our end of year summary, so, what have we been up to? With a new year, a new start was long overdue. We’re pleased to announce that we’ve finally shifted our butts over to our own server (again).

While this isn’t quite as chunky a news update as the last post, we rather hope you appreciate the regularity of the news.

moss in action

Infrastructure, revisited

‘tarkah’ has been tirelessly porting moss-service, the core foundation of our automated build system, to a shiny new Rust codebase. While we don’t have an immediately pressing need for the port to come online, it will certainly address some “v1 issues” and ensure the best-in-class development experience for our contributors.

Licensing

In our pursuit for a sustainable, community oriented project, we’ve been doing a lot of in house organisation. Very importantly, we’ve now relicensed our packaging recipes to MPL-2.0, bringing them in line with the license chosen for our current engineering efforts (moss, etc).

To reiterate a critical point, we require that our contributions are attributed to the virtual collective “Serpent OS Developers” to ensure we’re unable to relicense any contributions without express consent of the contributors involved.

Additionally we picked MPL-2.0 over Zlib due to some distinct advantages in relation to patent trolls. In combination we believe both the project and our users will have the greatest protection from patent trolls and fiscally-motivated relicensing.

As I Raise My Trigger Finger

As you may recall from the last post, we announced our intention to add system triggers to moss. We also stated that our triggers must run in isolation thanks to the unique architecture of our package management. After much consideration, we’ve landed on a strategy that will work for us.

Post-blit triggers

Every mutable operation in moss results in a fresh internal transaction, which is present as a full /usr tree inside a staging tree. The majority of triggers can be executed here, just before we “activate” the new rootfs. This allows us to detect potential trigger failures, and abort the activation.

These YAML-defined triggers will be run in an isolated environment (clone-based container) with the new /usr and read-only access to /etc.

Post-activation triggers

Once we have a freshly activated /usr (ie new system view) we can run a very limited set of post-activation triggers. An obvious example may be systemctl daemon-reexec. Note that by splitting post-blit and post-activation triggers, we leave the door open to multiple activation strategies for non-simple updates, including soft reboots and kernel patching.

Packaging updates!

With all of the migration work, our builds have also resumed. In the space of a few short days we’ve had 32 package builds. Ok, not a huge number, but we also have 15 outstanding pull requests to our new recipes repo!

In order to make life not only simpler for us, but for Solus in its future role as a source-derivative of Serpent, we’ve unified all of our git recipe repos into a new single repo, recipes.

Improved hosting

Our devops team has been working hard to mitigate bus factor with our transition to the new infrastructure, Bitwarden account, shared DNS, and opentofu configuration.

Self Hosting. Again

As part of the recovery process for Solus, the two projects shared resources, including a large Hetzner node. This has actually worked remarkably well for a long time now, however with Serpent growing we do not wish to impact the availability and stability of the Solus update experience.

Long story short, money from GitHub Sponsors (You guys! < 3) is now paying for an AX52 server from Hetzner. It’s sufficiently powerful that we’re running a backup builder there as well as repo management and build controller.

El Cheapo Web Hosting

Looking at our project’s use of static generators - we decided to review our hosting for web content. Long story short, we deemed it entirely unnecessary. We’re now using GitHub pages to deploy this website and our documentation site via GitHub actions. Don’t worry, we use git, we have backups, and we manage the infrastructure using IaC practices.

Clarifying our baseline target

Lately there have been some questions in both Serpent and Solus as to what our baseline will be. In Serpent OS, this will be at minimum x86_64-v2, with x86_64-v3x packages automatically offered if the system base criteria are met.

This decision has been made to offer the widest baseline compatibility without compromising heavily on system performance (Such that x86_64-generic is unsupported in Serpent OS). We will continue offering both v2 and v3x until it is no longer feasible from a storage/financial perspective. Source derivatives (Including Solus) are free to chart their own course here, if necessary, and still benefit from tight upstream integration.

Towards Next Month

In our next update, we’ll have quite a bit to cover, including:

Integrated triggers
Progress with GNOME packaging/readiness
Boulder / Infrastructure porting progress
Actually useful Serpent images

End Of Year Summary

Dec 19, 2023

Ikey

Founder

Ermo

Co-founder

Burning question - how long before we can use Serpent OS on our development systems? It’s a fair question - and one that requires context to answer. Let’s start our new monthly update cycle with a recap of progress so far.

screenshot of moss

Introducing tarkah

We’d like to extend a huge official welcome, and thank you, to newest team member Cory Forsstrom (aka tarkah)!

Firstly, a quote from Cory:

I’ve been a long time Solus user and was very excited about what serpent os was doing. Really got invested once I started diving into the D code and seeing how powerful the tools and ideas were. The Rust rewrite was just a perfect storm for me with my experience there and my desire to make contributions. Getting to know you and ermo has just been icing on the cake, you’ve both been so welcoming and friendly. So lots of fun times.

On the personal side, I’m on the west coast in the States, have a lovely wife, just had a baby girl and am enjoying my time with fatherhood and coding

Chances are you know tarkah for his contributions to iced-rs, and to Solus. His contributions to the moss-rs project have been absolutely critical, and his patience essential as we got ourselves quickly up to speed with Rust. It is entirely fair to say that our Rust efforts would not have been possible without him!

Rust Progress

I think it is fair to say that people collectively facepalmed when we announced our plans to adopt Rust - assuming this would be a huge set back. We’re really happy to report this is not the case, and we’ve made tremendous progress in this area.

moss

Our Rust based moss system package tool is now leaps and bounds ahead of its predecessor, supporting the original features and more!

State management
Garbage collection of old transactions (via state prune)
Optimised write order of transactions (see more below)
File conflict detection
Parallel fetching and decompression of packages
Update support (as sync)
Local and remote repositories with priority-based layering
Support for building with musl for fully self-contained, static builds of moss with zero system dependencies.

We now have a version-agnostic “stone” crate that is more efficient at reading packages than our original D code. In addition, it benefits from the memory safety promises of Rust.

Virtual filesystem management

OK, lets look at the problem: Every transaction in moss requires us to generate a new staging tree containing all of the directories and files for the /usr tree, using hard links, from content addressable storage. As one might expect, creating tens of thousands of new nodes is really, really slow!

To alleviate this issue we created a vfs crate in moss to optimise how we construct each transaction tree:

Organise all incoming registered paths into a specific order
Inject all missing implicitly created directories for accounting
Re-parent nodes in tree which live under redirected symlinks
Compile into a recursive data-structure using relative names.

There is a multipart “build” for the tree that detects any filesystem conflicts before any files have been written to disk, preventing broken transactions. With the fully built data structure we can recurse a virtual filesystem, making use of the at family of functions like linkat, mkdirat, to “blit” a filesystem tree without requiring any path resolution, in the most efficient order possible.

The net result? New transaction roots are created in milliseconds, not seconds.

Upgrade support - kind of

moss now offers a sync command in place of the conventional upgrade one might expect. Rather than upgrading to the latest version of a package, we rely on moss and our infrastructure tooling work in conjunction to ensure we can simply synchronize the package selection with the the “tip” candidates in the configured repositories, as ordered by priority. From a distro-builder PoV, we can now revert and pull builds. For a user PoV, you can remove a test repository and moss sync to go back to the official versions of packages.

Generally speaking, when you moss sync you will be seeing new packages, however. :smile: This feature is being built to enable a future feature: exported states. Want to try a new edition? Sync to the lockfile. Need to quickly and reproducibly deploy containers from a locked down configuration? You get the idea.

moss sync

Fully `async`

The moss tool is now completely asynchronous, making efficient use of both coroutines and OS threads to perform as much work possible in the shortest space of time. This allows us to download, fetch, check and unpack many packages at the same time without being blocked on network or disk, greatly speeding up the fetch part of a transaction.

This is achieved by building on the famous tokio runtime and :material-github: reqwest crates, among others.

boulder

At the time of writing our port of boulder hasn’t quite yet caught up with the original implementation, given all of the ground we covered with moss. With that said, we decided to step back and evaluate where we could improve upon the first version of boulder.

boulder in action

User namespaces (rootless)

Arguably one of the most important changes: boulder no longer requires running as root. Even better, we make use of user namespaces and execute using clone - meaning we no longer need a separate binary (mason) within the container environment!

Superior error handling

Thanks to everything being a self-contained binary, we can tell you up front that you’re using unknown action or definition macros (i.e. %(datadir)) before attempting any builds, saving you valuable time.

Breakpoints

When this was first proposed, I did a double take. Breakpoints.. in recipes… ? No, seriously. Not only was it proposed, but between ermo and tarkah, they only went and implemented it. You can now use special macros within your recipe to add debugging breakpoints in your recipe to drop to a shell within the container and debug the more difficult builds!

Our documentation will be updated when the new boulder ships a stable build.

Unified codebase

We now build both moss and boulder from the moss-rs repository, allowing them to share the same code and mechanisms for dealing with dependencies, system roots, etc., allowing for far higher levels of integration than were previously possible.

A feature we’re currently working on allows you to use a single directory for the system “root”, where all the unpacking, fetching and databases happen, while almost instantly creating new build roots!

What We Already Have

I get it. “All this rewriting - how will you ever make progress” ? It’s a fair question. We didn’t throw anything away, quite the opposite. Our repos, tooling, are all still in use enabling us to build the distribution in parallel to the ongoing rewrites.

Build infrastructure

For a long time, we’ve had our :material-view-dashboard: dashboard up and running for our build infrastructure. Sure, we’re going to rewrite it beyond the “Proof of concept” stage when required - but for now it still serves us well. Admittedly there is a deep issue within the druntime causing thread leaks over time, but we have it restarting via systemd every 24 hours as a bandaid. Needless to say, we’re big fans of memory safety now.

Our current deployment watches git repos in our :material-github: snekpit collection, and automatically forms build queues from the pending builds, ordering them by dependency. Using two of ermo’s beefy builders, the packages are automatically built and deployed to our repository. Our workflow means that maintainers only have to merge a pull request for builds to appear in the volatile repository.

Legacy tooling

Despite having some warts, the build infrastructure is still able to use legacy moss and boulder to make our new builds available, and we’ve been minimally patching these old tools to support our transition to the Rust based tooling. As of this moment, moss-rs officially supersedes the D version, and our new boulder is quickly catching up.

Long story short, we’re still building on all fronts and aren’t blocked anywhere.

Live repository

Builds are published to our volatile repository, and we do have a fully functioning software repository despite being quite small. Once we’re happy with the new moss we’ll speed up in our packaging efforts. With that said, we do have two kernels (kvm and desktop) and even gtk-4 at this early stage!

The immediate future

OK let’s tldr this:

Rust ports are coming along fantastically
moss has been replaced
still making use of existing tools and infrastructure to build the OS
actually building the fully bootstrapped, self-contained OS, built on libc++, clang, glibc, etc.
actively pushing updates to binary repository with fully automated pipeline
new stuff is :rocket: blazing fast

System triggers

We’re experimenting with system triggers right now, execution units that run to fully bake the OS state after each transaction completes. In keeping with our stateless philosophy (and pending adoption of hermetic /usr), we need triggers that understand the union state of the operating system, not individual per-package triggers.

A YAML-based trigger recipe has been baked up, which deliberately avoids the use of shell helpers and execution security weaknesses by using an extended variant of globs:

    let pattern = "/usr/lib/modules/(version:*)/*".parse::<fnmatch::Pattern>()?;

Our new :material-github: fnmatch crate, heavily inspired by Python fnmatch, compiles specially format glob strings to Regex, extending with capture groups for named variables. In YAML, it looks a bit like this:

handlers:
    depmod:
        run: /sbin/depmod
        args: ["-a", "$(version)"]

## Link paths to handlers and filter on the type
paths:
    "/usr/lib/modules/(version:*)/kernel" :
        handlers:
            - depmod
        type: directory

Once we roll out the trigger support, we unblock the packaging of critical items like gtk-3 for gdm, as well as enable usable installations and ISOs. Note that they differ from classical triggers due to the architecture of moss: they need to run in an isolated namespace (container) so we can ensure the staged transaction won’t break. Additionally we cache and collect the trigger artefacts to speed up each filesystem operation, giving us a first: versioned state artefacts.

Boot management

Our plan is to build on my prior work in the Solus boot management design and :material-github: clr-boot-manager, addressing some long standing shortcomings. Most importantly, our new module will not need to inspect or manage the OS filesystem as moss will be able to provide all of the relevant information (full accounting for all used paths in all transactions).

Initially we will focus on UEFI and supporting the :material-github: Discoverable Partitions Specification by way of XBOOTLDR partitions, the ESP and the Boot Loader Interface. Currently we have no plans to support Unified Kernel Images as the approach taken by CBM (and soon, moss) alleviates the data concerns of dealing with vfat. However, as and when UKIs gain configurable, standardised behaviour for cmdline control we will investigate their use. Until that point please note we prebuild our initrd images and ship them directly in our packages, as Solus has done so for years already.

The net gain for taking control of boot management will be the deduplication and garbage collection of assets installed to either the ESP or XBOOTLDR partitions, along with generation of boot entries relevant to Serpent OS: Transaction specific entries allowing us to directly boot back to older installs in case upgrades go wrong.

Last, but certainly not least, this approach will make dual-booting with another OS less irritating by no longer being bound by a fixed ESP size.

Desktop

Our plan is to produce a GNOME Shell based live ISO containing the bare basics to get everything else validated, and open the path to installations. In short, it will include:

Desktop.
moss tooling
web browser (firefox)
terminal (potentially gnome-console)
flatpak
Eventually: GNOME Software (integrated with flatpak+moss)

It should be obvious we’re not building a general purpose home-use distribution. However, we also won’t stop people using Serpent OS for their needs, and we plan to beef up the repository, fully support flatpak and eventually custom moss source recipe repos. (ala AUR)

Safety is important

Our journey over the last few years through D, DIP1000, Rust and memory safety has been incredibly eye opening for us. To our own surprise, memory safety has become a huge interest and concern for the team. Not only have we embarked on a full transition of our tooling to Rust, we’ve started looking at the OS itself.

As an early indication of our intent, our curl package is now built with the rustls backend and not OpenSSL. Needless to say, we’re very keen to reduce the surface area for attacks by adopting alternative, safer implementations where possible. Other projects we’re keeping a very close eye on include:

Looking to next year

With most of us entering our holiday leave, and the new year practically around the corner, the Serpent OS team wishes you many happy returns and a wonderful new year. With one blocker being worked on (triggers), and the last on the horizon (boot management), your dream installation of Serpent OS is firmly within reach.

In the spirit of the season… if you feel like supporting our project, you can help with the rising cost of living and our expanding hosting requirements via GitHub Sponsors!

Sponsor us!

Oxidised Moss

Sep 6, 2023

Ikey

Founder

Allow me to start this post by stating something very important to me: I absolutely love the D programming language, along with the expressivity and creative freedom it brings. Therefore please do not interpret this post as an assassination piece.

For a number of months now the Serpent OS project has stood rather still. While this could naively be attributed to our shared plans with Solus - a deeper, technical issue is to be acredited.

D isn’t quite there yet. But it will be, some day.

Again, allow me to say I have thoroughly enjoyed my experience with D over the last 3 or so years, it has been truly illuminating for me as an engineer. With that said, we have also become responsible for an awful lot of code. As an engineering-led distribution + tooling project, our focus is that of secure and auditable code paths. To that extent we pursued DIP1000 as far as practical and admit it has a way to go before addressing our immediate needs of memory safety.

While we’re quite happy to be an upstream for Linux distributions by way of release channels and tooling releases, we don’t quite have the resources to also be an upstream for the numerous D packages we’d need to create and maintain to get our works over the finish line.

With that said, I will still continue to use D in my own personal projects, and firmly believe that one day D will realise its true potential.

Rewarding contributors

Our priorities have shifted somewhat since the announcement of our shared venture with Solus, and we must make architectural decisions based on the needs of all stakeholders involved, including the existing contributor pool. Additionally, care should be taken to be somewhat populist in our choice of stacks in order to give contributors industry-relevant experience to add to their résumé (CV).

Playing to strengths

Typically Solus has been a Golang-oriented project, and has a number of experienced developers. With the addition of the Serpent developers, the total cross-over development team has a skill pool featuring Rust and Go, as well as various web stack technologies.

Reconsidering the total project architecture including our automated builds, the following decisions have been made that incorporate the requirements of being widely adopted/supported, robust ecosystems and established tooling:

Rust, for low level tooling and components. Chiefly: moss, boulder, libstone
ReactJS/TypeScript for our frontend work (Summit Web)
Go - for our web / build infrastructure (Summit, Avalanche, Vessel, etc)

Lets start… 2 days ago!

The new infrastructure will be brought up using widely available modules, and designed to be scalable from the outset as part of a Kubernetes deployment, with as minimal user interaction as needed. Our eventual plans include rebuilding the entire distribution from source with heavy caching once some part of the dependency graph changes.

This infrastructure will then be extended to support the Solus 4 series for quality of life improvements to Solus developers, enabling a more streamlined dev workflow: TL;DR less time babysitting builds = more Serpent development focus.

Our priority these past few days has been on the new moss-rs repository where we have begun to reimplement moss in Rust. So far we have a skeleton CLI powered by clap with an in-progress library for reading .stone archives, our custom package format.

The project is organised as a Rust workspace, with the view that stone, moss and boulder will all live in the same tree. Our hope is this vastly improves the onboarding experience and opens the doors (finally) to contributors.

Licensing

It should also be noted that the new tooling is made available under the terms of the Mozilla Public License (MPL-2.0). After internal discussion, we felt the MPL offered the greatest level of defence against patent trolls while still ensuring our code was widely free for all to respectfully use and adapt.

Please also note that we have always, and continue to deliberately credit copyright as:

Copyright © 2020-2023 Serpent OS Developers

This is a virtual collective consisting of all whom have contributed to Serpent OS (per git logs) and is designed to prevent us from being able to change the license down the line, i.e. a community protective measure.

Glad to be back

Despite some sadness in the change of circumstances, we must make decisions that benefit us collectively as a community. Please join us in raising a virtual glass to new pastures, and a brave new blazing fast 🚀 (TM) future for Serpent OS and Solus 5.

Cheers!

Snakes On A Boat

Apr 21, 2023

Ikey

Founder

We had intended to get a blog post out a little bit quicker, but the last month has been extremely action packed. However, it has paid off immensely. As our friends at Solus recently announced it is time to embark on a new voyage.

dashboard preview

General gist of it

There is very little point in rehashing the post, so I’ll cover the essentials

We assisted the Solus team in deploying a complete new infrastructure. Some of this work is ongoing
The package delivery pipeline is in place, meaning updates will soon reach users.
We’re working towards a mutually beneficial future whereby Solus adopts our tooling.

Is Serpent OS over?

Oh no, no. The most practical angle to view this from is that Serpent OS is free to build and innovate to create the basis of Solus 5. Outside of the distribution we’re still keen to continue development on our tooling and strategies.

It is therefore critical that we continue development, and find the best approach for both projects, to make the transition to Solus 5 as seamless as possible. To that end we will still need to produce ISOs and have an active community of users and testers. During this transition period, despite being two separate projects, we’re both heading to a common goal and interest.

Finances

We have an exciting journey ahead for all of us, and there are many moving parts involved. Until we’re at the point of mutual merge, we will keep the entities and billing separate. Thus, funds to the Solus OpenCollective are intended for use within Solus, whereas we currently use GitHub Sponsors for our own project needs.

Currently Solus and Serpent OS share one server, which was essential for quick turnaround on infrastructure enabling. At the end of this month Serpent OS will migrate from that server to a new, separate system, ensuring the projects are billed separately.

Long story short, if you wish to sponsor Serpent OS specifically, please do so via our GitHub sponsors account as our monthly running costs will immediately rise at the end of this month. If you’re supporting Solus development, please do visit them and sponsor them =)

Next steps?

Glad you asked! Now that the dust is settling, we’re focusing on Serpent OS requirements, and helping Solus where we can. Our immediate goals are to build a dogfooding system for a small collection of developers to run as an unsupported prealpha configuration, allowing us to flesh out the tooling and processes.

This will include a live booting GNOME ISO, and sufficient base packages to freely iterate on moss, boulder, etc as well as our own infrastructure. Once we’ve attained a basic quality and have an installer option, those ISOs will be made available to you guys!

Infrastructure Launched!

Mar 18, 2023

Ikey

Founder

After many months and much work, our infrastructure is finally online. We’ve had a few restarts, but it’s now running fully online with 2 builders, ready to serve builds around the clock.

Firstly, I’d like to apologise for the delay since our last blog post. We made the decision to move this website to static content, which took longer than expected. We’re still using our own D codebase, but prebuilding the site (and fake “API”) so we can lower the load on our web server.

Introducing the Infra

Our infrastructure is composed of 3 main components.

Summit

This is the page you can see over at dash.serpentos.com. It contains the build scheduler. It monitors our git repositories, and as soon as it discovers any missing builds it creates build tasks for them. It uses a graph to ensure parallel builds happen as much as possible, and correctly orders (and blocks) builds based on build dependencies.

Avalanche

We have 2 instances of Avalanche, our builder tool, running. This accepts configuration + build requests from Summit, reporting status and available builds.

Vessel

This is an internal repository manager, which accepts completed packages from an Avalanche instance. Published packages land at dev.serpentos.com

Next steps

Super early days with the infra but we now have builds flowing as part of our continuous delivery solution. Keeping this blog post short and sweet… we’re about to package GNOME and start racing towards our first real ISOs!

Don’t forget

If you like what the project is doing, don’t forget to sponsor!

Lift Off

Dec 24, 2022

Ikey

Founder

Enough of this “2 years” nonsense. We’re finally ready for lift off. It is with immense pleasure we can finally announce that Serpent OS has transitioned from a promise to a deliverable. Bye bye, phantomware!

We exist

As mentioned, we spent 2 years working on tooling and process. That’s .. well. Kinda dull, honestly. You’re not here for the tooling, you’re here for the OS. To that end I made a decision to accelerate development of the actual Linux distro - and shift development of tooling into a parallel effort.

Infrastructure .. intelligently deferred

I deferred final enabling of the infrastructure until January to rectify the chicken/egg scenario whilst allowing us to grow a base of contributors and an actual distro to work with. We’re in a good position with minimal blockers so no concern there.

A real software collection

This is our term for the classical “package repository”. We’re using a temporary collection right now to store all of the builds we produce. In keeping with the Avalanche requirements, this is the volatile software collection. Changes a lot, hasn’t got a release policy.

A community.

It goes without saying, really, that our project isn’t remotely possible without a community. I want to take the time to personally thank everyone that stepped up to the plate lately and contributed to Serpent OS. Without the work of the team, in which I include the contributors to our venom recipe repository, an ISO was never possible. Additionally contributions to tooling has helped us make significant strides.

It should be noted we’ve practically folded our old “team” concept and ensured we operate across the board as a singular community, with some members having additional responsibilities. Our belief is all in the community have equal share and say. With that said, to the original “team”, members both past and present, I thank for their (long) support and contributions to the project.

An ISO.

We actually went ahead and created our first ISO. OK that’s a lie, this is probably the 20th revision by now. And let’s be brutally honest here:

It sucks.

We expected no less. However, the time is definitely here for us to begin our public iteration, transitioning from suckness to a project worth using. In order to do that, we need to get ourselves to a point whereby we can dogfood our work and build a daily driver. Our focus right now is building out the core technology and packaging to achieve those aims.

So if you want to try our uninstallable, buggy ISO, chiefly created as a brief introduction to our package manager and toolchain, head to our newly minted Download page. Set your expectations low, ignore your dreams, and you will not be disappointed!

All jokes aside, it took a long time to get to point where we could even construct our first, KVM-focused, UEFI-only snekvalidator.iso. We now have a baseline to improve on, a working contribution process, and a booting, self-hosting system.

The ISO is built using 2 layered collections, the protosnek collection containing our toolchain, and the new volatile collection. Much of the packaging work has been submitted by venom contributors and the core team. Note you can install neofetch which our very own Rune Morling (ermo) patched to support the Serpent OS logo.

Boot it in Qemu (or certain Intel laptops) and play with moss now! Note, this ISO is not installable, and no upgrade path exists. It is simply the beginnings of a public iteration process.

Next steps

In January we’ll launch our infrastructure to scale out contributions as well as to permit the mass-rebuilds that need to happen. We have to enable our -dbginfo packages and stripping, which were disabled due to a parallelism issue. We need to introduce our boot management based around systemd-boot, provide more kernels, do hardware enabling, introduce moss-triggers, and much more. However, this is a pivotal moment for our project as we’ve finally become a real, if not sucky, distro. The future is incredibly bright, and we intend to deliver on every one of our promises.

As always, if you want to support our development, please consider sponsoring the work, or engaging with the community on Matrix or indeed our forums.

You can discuss this blog post, or leave feedback on the ISO, over at our forums.

The Big Update

Sep 14, 2022

Ikey

Founder

Well - we’ve got some big news! The past few weeks have been an incredibly busy time for us, and we’ve hit some major milestones.

Funding

After much deliberation - we’ve decided to pull out of Open Collective. Among other reasons, the fees are simply too high and severely impact the funds available to us. In our early stages, the team consensus is that funds generated are used to compensate my time working on Serpent OS.

As such I’m now moving funding to my own GitHub Sponsors page - please do migrate! It ensures your entire donation makes it and keeps the lights on for longer =) Please remember I’m working full time on Serpent OS exclusively - I need your help to keep working.

Moved to GitHub

We’ve pretty much completed our transition to GitHub. We’ve now got the following organisations:

snekpit - Packaging work
serpent-os - Main code development (moss/boulder/etc)

Forums

Don’t forget - our forums are live over at forums.serpentos.com - please feel free to drop in and join in with the community =)

Rehash on the tooling

OK so what exactly are moss and boulder? In short - they’re the absolute core pieces of our distribution model.

”What is moss?”

On the surface, moss looks and feels roughly the same as just about any other traditional package manager out there. Internally, however, its far more modern and has a few tricks up its sleeve. For instance, every time you initiate an operation in moss, be it installation, removal, upgrade, etc, a new filesystem transaction is generated. In short, if something is wrong with the new transaction - you can just boot to an older transaction when things worked fine.

Now, it’s not implemented using bundles or filesystem specific functionality, internally its just intelligent use of hardlinks and deduplication policies, and we have our own container format with zstd based payload compression. Our strongly typed, deduplicating binary format is what powers moss.

Behind the scenes we also use some other cool technology, such as LMDB for super reliable and speedy database access. The net result is a next generation package management solution that offers all the benefits of traditional package managers (i.e. granularity and composition) with new world features, like atomic updates, deduplication, and repository snapshots.

boulder

It’s one thing to manage and install packages, it’s another entirely to build them. boulder builds conceptually on prior art such as pisi and the package.yml format used in ypkg. It is designed with automation and ease of integration in mind, i.e. less time spent focusing on packaging and more time on actually getting the thing building and installing correctly.

Boulder supports “macros” as seen in the RPM and ypkg world, to support consistent builds and integration. Additionally it automatically splits up packages into the appropriate subpackages, and automatically scans for binary, pkgconfig, perl and other dependencies during source analysis and build time. The end result is some .stone binary packages and a build manifest, which we use to flesh out our source package index.

Major moss improvements

We’ve spent considerable time reworking moss, our package manager. It now features a fresher (terminal) user interface, progress bars, and is rewritten to use the moss-db module encapsulating LMDB.

nspawn

It’s also possible to manipulate the binary collections (software repositories) used by moss now. Note we’re going to rename “remote” to “collection” for consistency.

At the time of writing:

$ mkdir destdir
$ sudo moss remote add -D destdir protosnek https://dev.serpentos.com/protosnek/x86_64/stone.index
$ sudo moss install -D destdir bash dash dbus dbus-broker systemd coreutils util-linux which moss nano
$ sudo systemd-nspawn -b -D destdir

This will be simplified once we introduce virtual packages (Coming Soon ™)

Local Collections

Boulder can now be instructed to utilise a local collection of stone packages, simplifying the development of large stack items.

sudo boulder bi stone.yml -p local-x86_64

Packages should be moved to /var/cache/boulder/collections/local-x86_64 and the index can be updated by running:

sudo moss idx /var/cache/boulder/collections/local-x86_64

Self Hosting

Serpent OS is now officially self hosting. Using our own packages, we’re able to construct a root filesystem, then within that rootfs container we can use our own build tooling (boulder) to construct new builds of our packages in a nested container.

The protosnek collection has been updated to include the newest versions of moss and boulder.

self hosting

Booting

As a fun experiment, we wanted to see how far along things are. Using a throwaway kernel + initrd build, we were able to get Serpent OS booting using virtualisation (qemu)

booting

ISO When?

Right now everyone is working in the snekpit organisation to get our base packaging in order. I’m looking to freeze protosnek, our bootstrap collection, at the latest of tomorrow evening.

We now support layered, priority based collections (repositories) and dependency solving across collection boundaries, allowing us to build our new main collection with protosnek acting as a bootstrap seed.

Throughout this week, I’ll focus on getting Avalanche, Summit and Vessel into shape for PoC so that we can enjoy automated builds of packages landing in the yet-to-be-launched volatile collection.

From there, we’re going to iterate + improve packaging, fixing bugs and landing features as we discover the issues. Initially we’ll look to integrate triggers in a stateless-friendly fashion (our packages can only ship /usr by design) - after that will come boot management.

An early target will be Qemu support via a stripped linux-kvm package to accelerate the bring up, and we encourage everyone to join in the testing. We’re self hosting, we know how to boot, and now we’re able to bring the awesome.

I cannot stress how important the support to the project is. Without it - I’m unable to work full time on the project. Please consider supporting my development work via GitHub Sponsors.

I’m so broke they’ve started naming black holes after my IBAN.

Thank you!

You can discuss this blog post over on our forums

Infrastructure Update

Jul 10, 2022

Ikey

Founder

Since the last post, I’ve pivoted to full time work on Serpent OS, which is made all the more possible thanks to everyone supporting us via OpenCollective <3.

We’ve been working towards establishing an online infrastructure to support the automation of package builds, while revisiting some core components.

moss-db

During the development of the Serpent OS tooling we’ve been exploring the possibilities of D Lang, picking up new practices and refining our approach as we go. Naturally, some of our older modules are somewhat … smelly. Most noticeable is our moss-db module, which was initially intended as a lightweight wrapper around RocksDB.

In practice that required an encapsulation API written in D around the C API, and our own wrapping on top of that. Naturally, it resulted in a very allocation-heavy implementation that just didn’t sit right with us, and due to the absurd complexity of RocksDB was still missing quite a few features.

Enter LMDB

We’re now using the Lightning Memory-Mapped Database as the driver implementation for moss-db. In short, we get rapid reads, ACID transactions, bulk inserts, you name it. Our implementation takes advantage of multiple database indexes (MDB_dbi) in LMDB to partition the database into internal components, so that we can provide “buckets”, or collections. These internal DBs are used for bucket mapping to permit a key-compaction strategy - iteration of top level buckets and key-value pairs within a bucket.

Hat tip, boltdb

The majority of the API was designed with the boltdb API in mind. Additionally it was built with -preview=dip1000 and -preview=in enabled, ensuring safely scoped memory use and no room for memory lifetime issues. While we prefer the use of generics, the API is built with immutable(ubyte[]) as the internal key and value type.

Custom types can simply implement mossEncode or mossDecode to be instantly serialisable into the database as keys, values or bucket identifiers.

Example API usage:

Database db;
/* setup the DB with lmdb:// URI */

/* Write transaction */
auto err = db.update((scope tx) @safe
{
    auto bucket = tx.bucket("letters");
    return tx.set(bucket, "a", 1);
});

/* do something with the error */

err = db.view((in tx) @safe
{
    foreach (name, bucket ; tx.buckets!int)
    {
        foreach (key, value ; tx.iterator!(string,string)(bucket))
        {
            /* do something with the key value pairs, decoded as strings */
        }
    }

    /* WILL NOT COMPILE. tx is const scope ref :) */
    tx.removeBucket("numbers");

    return NoDatabaseError;
}

Next for moss

Moss will be ported to the new DB API and we’ll gather some performance metrics, while implementing features like expired state garbage collection (disk cleanup), searching for names/descriptions, etc.

Avalanche

Early version of avalanche, in development

Avalanche is a core component of our upcoming infrastructure, providing the service for running builds on a local node, and a controller to coordinate a group of builders.

Summit will be the publicly accessible project dashboard, and will be responsible for coordinating incoming builds to Avalanche controllers and repositories. Developers will submit builds to Summit and have them dispatched correctly.

So far we have the core service process in place for the Controller + Node, and now we’re working on persistence and handshake. TLDR; fancy use of moss-db and JSON Web tokens over mandated SSL. This means our build infra will be scalable from day 1 allowing multiple builders to be online very early on.

Timescale

We’re planning to get an early version of our infrastructure up and running within the next 2 weeks, and get builds flowing =)

Packaging Automation, Next Steps

Jun 22, 2022

Ikey

Founder

Hot damn we’ve been busy lately. No, really. The latest development cycle saw us focus exclusively on boulder, our build tooling. As of today it features a proof of concept boulder new subcommand for the automatic generation of packaging templates from an upstream source (i.e. tarball).

Before we really start this blog post off, I’d like to thank everyone who is supporting the project! All of the OpenCollective contributions will make it easier for me to work full time on Serpent OS =) Much love <3

Look at all the buildiness

But, but build flavours …

Alright you got me there, certain projects prefer to abstract the configuration, build and installation of packages and be provided with some kind of hint to the build system, i.e. manually setting autotools, etc.

Serpent OS packaging is declarative and well structured, and relies on the use of RPM-style “macros” for distribution integration and common tasks to ensure consistent packaging.

We prefer a self documenting approach that can be machine validated rather than depending on introspection at the time of build. Our stone.yml format is very flexible and powerful, with automatic runtime dependencies and package splitting as standard.

..Doesn’t mean we can’t make packaging even easier though.

Build discovery

Pointing boulder at an upstream source will perform a deep analysis of the sources to determine the build system type, build dependencies, metadata, licensing etc. Right now it’s just getting ready to leave POC stage so it has a few warts, however it does have support for generating package skeletons for the following build systems:

cmake
meson
autotools

We’re adding automation for Perl and Python packaging (and Golang, Rust, etc) so we can enforce consistency, integration and ease without being a burden on developers. This will greatly reduce the friction of contribution - allowing anyone to package for Serpent OS.

We’re also able to automatically discover build time dependencies during analysis and add those to the skeleton stone.yml file. We’ll enhance support for other build systems as we go, ensuring that each new package is as close to done on creation as possible, with review and iteration left to the developer.

License compliance

A common pain in the arse when packaging for any Linux distribution is ensuring the package information is compliant in terms of licensing. As such we must know all of the licensing information, as well as FSF and OSI compliance for our continuous integration testing.

…Finding all of that information is truly a painful process when conducted manually. Thankfully boulder can perform analysis of all licensing files within the project to greatly improve compliance and packaging.

Every license listed in a stone.yml file must use a valid SPDX identifier, and be accurate. boulder now scans all license files, looking for matches with both SPDX IDs as well as fuzzy-matching the text of input licenses to make a best-guess at the license ID.

This has so far been highly accurate and correctly identifies many hundreds of licenses, ensuring a compliant packaging process with less pain for the developers. Over time we’ll optimise and improve this process to ensure compliance for our developers rather than blocking them.

As of today we support the REUSE specification for expressing software licenses too!

Next on the list

The next steps are honest-to-goodness exciting for us. Or should I say.. exiting?

bill

Work formally begins now on Bootstrap Bill (Turner). Whilst we did successfully bootstrap Serpent OS and construct the Protosnek repository, the process for that is not reproducible as boulder has gone through massive changes in this time.

The new project will leverage boulder and a newly designed bootstrap process to eliminate all host contamination and bootstrap Serpent OS from stone.yml files, emitting an immutable bootstrap repository.

Layering support will land in moss and boulder to begin the infrastructure projects.

Build Submission (“Let’s Roll”)

The aim is to complete bill in a very short time so we can bring some initial infrastructure online to facilitate the automatic build of submitted build jobs. We’ll use this process to create our live repository, replacing the initial bootstrap repository from bill.

At this point all of the tooling we have will come together to allow us all to very quickly iterate on packaging, polish up moss and race towards installed systems with online updates.

A Word From The Founder

Jun 6, 2022

Ikey

Founder

Well well, it’s been a long time since I personally wrote a post.. :) So let’s keep this short and sweet, shall we? I’m returning to full time work on Serpent OS.

The 6th of July will be my last day at my current employment having tendered my 30 day notice today. Despite having enjoyment at my current position, the reality is that my passion and focus is Serpent OS.

I’m now in a transition process and will ramp up my efforts with Serpent OS. Realistically I need to reduce the outgoing costs of the project and with your help I can gain some level of financial support as we move through the next stages of development. Worst case, I will only take on any part-time or contractual gigs, allowing my primary focus to be Serpent OS.

I’ll begin accelerating works and enabling community contribution so we can get the derailed-alpha train back on the tracks.

I have absolute faith in this project, the community and our shared ability to deliver the OS and tooling. To achieve it will require far more of my time and I’m perfectly willing to give it.

Thank you all to everyone who has been supporting the project, it is now time to deliver. Not just another run of the mill distribution but a technically competent and usable distribution that is not only different but better.

Let’s do this in the most grassroots and enjoyable way possible =)

RELR Brings Smaller Files, More Performance?

Apr 5, 2022

SunnyFlunk

Distro Engineer

RELR is an efficient method of storing relative relocations (but is not yet available in glibc upstream). This has a significant reduction on file size often in the vicinity of 5% for libraries and even higher for PIE binaries. We also take a look at the performance impact on enabling RELR and that looks really good too! Smaller files with more performance - you can have your cake and eat it too!

Delicious Size Savings

Everyone enjoys smaller files, especially when it’s for free! RELR provides a very efficient method of storing relative relocations where it only requires a few percent compared to storing them in the .rela.dyn section which is what currently happens. However, it can’t store everything so the .rela.dyn section remains (though much smaller).

Here’s an example of the sections of libLLVM-13.so with and without RELR. Here we see the .rela.dyn section taking up a whole 2MB! When enabling RELR, .rela.dyn shrinks down to just under 100KB while adding a new section .relr.dyn which is just over 20KB! That’s nearly a 1.9MB file size reduction, so you’ll get smaller packages, smaller updates and it will be even faster to create delta packages from our servers. For reference, some of the biggest files have a .rela.dyn section over 10MB!

Section	Without RELR	With RELR
.dynstr	2,285,738	2,285,723
.rela.dyn	2,006,472	97,464
.relr.dyn	-	21,688
.text	28,708,290	28,708,386
.dynamic	592	608

Total	44,853,987	42,966,764

Smaller, But at What Cost?

While most of the discussion about RELR is around the size savings, there’s been very little in terms of the performance numbers of enabling RELR. For most things, it’s not going to make a noticeable difference, as it should only really impact loading of binaries. There’s one rule we have and that’s to measure everything! We care about every little detail where many 1-2% improvements can add up to significant gains.

First, we require a test to determine if we could detect changes between an LLVM built with RELR and one without. The speed of the compiler is vital to a distro, where lackluster performance of the build system hurts all contributors and anyone performing source based builds. clang in this example was built using a shared libLLVM so that it would load the RELR section and it’s large enough to be able to measure a difference in load times (if one exists). Building gettext was the chosen test (total time includes tarball extraction, configure, make and make install stages), rather than a synthetic binary loading test to reflect real world usage. The configure stage is very long when building gettext so clang is called many times for short compiles. Lets take a look at the results:

no RELR:
[Build] Finished: 1 minute, 57 secs, 80 ms, 741 μs, and 2 hnsecs
[Build] Finished: 1 minute, 57 secs, 691 ms, 586 μs, and 4 hnsecs
[Build] Finished: 1 minute, 56 secs, 861 ms, 31 μs, and 8 hnsecs

RELR:
[Build] Finished: 1 minute, 55 secs, 244 ms, 213 μs, and 8 hnsecs
[Build] Finished: 1 minute, 55 secs, 400 ms, 158 μs, and 8 hnsecs
[Build] Finished: 1 minute, 55 secs, 775 ms, 40 μs, and 8 hnsecs

RELR+startup patch:
[Build] Finished: 1 minute, 54 secs, 979 ms, 166 μs, and 8 hnsecs
[Build] Finished: 1 minute, 54 secs, 820 ms, and 675 μs
[Build] Finished: 1 minute, 54 secs, 713 ms, 440 μs, and 3 hnsecs

Here we see the base configuration was able to build gettext in 117.21s on average. When we enabled RELR in our LLVM build (all other packages were without RELR still), the average build time decreased by 1.74s! That does not sound like a lot, but the time spent loading clang would only be a portion of the total, yet still gives a 1-2% performance lift over the whole build. While we were reducing start up time, I ran another test, but this time adding a patch to reduce paths searched on startup as well as enabling RELR. This patch reduced the average build time by a further 0.63s!

That’s a 2.37s reduction in the build just from improving the clang binary’s load time.

What This Means - RELR by Default

So what actually is RELR? I can’t really do the topic justice, so will point you to a great blog post about RELR, Relative Relocations and RELR. It’s quite technical for the average reader, but definitely worth a read if you like getting into the details. To no surprise the author (Fangrui Song) started the initial push for getting RELR support upstream in glibc (at the time of this post the patch series has not yet been committed to glibc git).

What I can tell you, is that we’ve applied the requisite patches for RELR support and enabled RELR by default in boulder for builds. Our container has been rebuilt and all is working well with RELR enabled. More measurements will be done in future in the same controlled manner, particularly around PIE load times.

Caveats - The Hidden Details

The performance benchmark was quite limited in terms of being an optimal case for RELR as clang is called thousands of times in the build so on average improved load time by about 0.6-0.7ms. We can presume that using RELR on smaller files is unlikely to regress load times. It definitely gives us confidence that it would be about the same or better in most situations, but not noticeable or measurable in most use cases. Minimizing build times is a pretty significant target for us, so even these small gains are appreciated.

The size savings can vary between packages and not everything can be converted into the .relr.dyn section. The current default use of RELR is not without cost as it adds a version dependency on glibc. We will ensure we ship a sane implementation that minimizes or removes such overhead.

It was also not straight forward to utilize RELR in Serpent. The pending upstream glibc patch series included a patch which caused issues when enabling RELR in Serpent OS (patch 3/5). As we utilize two toolchains, gcc/bfd and clang/lld, both need to function independently to create outputs of a functional OS. However the part “Issue an error if there is a DT_RELR entry without GLIBC_ABI_DT_RELR dependency nor GLIBC_PRIVATE definition.” meant that glibc would refuse to load files linked by lld despite having the ability to load them. lld has supported RELR for some time already, but does not create the GLIBC_ABI_DT_RELR dependency that is required by glibc. I have added my feedback to the patch set upstream. lld now has support for this version dependency upstream if we ever decide to use it in future.

After dropping the patch and patching bfd to no longer generate the GLIBC_ABI_DT_RELR dependency either, I was finally able to build both glibc and LLVM with the same patches. With overcoming that hurdle, rebuilding the rest of the repository went without a hitch, so we are now enjoying RELR in all of our builds and is enabled by default.

There is even further scope for more size savings, by switching the rela.dyn section for the rel.dyn section (this is what is used for 32-bit builds and one of the reasons files are smaller!). lld supports switching the section type, but I don’t believe glibc will read the output as it expects the psABI specified section (something musl can handle though).

The Cost of Adding GLIBC_ABI_DT_RELR

A quick check of two equivalent builds (one adding the GLIBC_ABI_DT_RELR version dependency and one not), there was an increase of 34 bytes to the file’s sections (18 bytes to .dynstr and 16 bytes to .gnu.version_r). It also means having to validate that the GLIBC_ABI_DT_RELR version is present in the libc and that the file using RELR includes this version dependency. This may not sound like much but it is completely unnecessary! Note that the testing provided in this blog post is without GLIBC_ABI_DT_RELR.

Regardless of what eventuates, these negatives won’t ship in Serpent OS. This will allow for us to support files that include the version dependency (when appimage and other distros catch up) as it will still exist in libc, but we won’t have the version check in files, nor will glibc check that the version exists before loading for packages built by boulder.

Making Deltas Great Again! (Part 1)

Feb 11, 2022

SunnyFlunk

Distro Engineer

In Optimising Package Distribution we discussed some early findings for implementing binary deltas in Serpent OS. While discussing the implementation we have found the requirements to be suboptimal for what we were after. We provide a fresh look at the issue and what we can do to make it useful in almost all situations without the drawbacks.

Deltas Have a Bad Reputation

I remember back in the early 2000s on Gentoo when someone set up a server to produce delta patches from source tarballs to distribute to users with small data caps such as myself. When requesting the delta, the job would be added to the queue (which occasionally could be minutes) and then created a small patch to download. This was so important at the time to reduce the download size that the extra time was well worth it!

Today things are quite different. The case for deltas has reduced for users as internet speeds have increased. Shrinking 20MB off a package size may be a second reduction for some, but 10 seconds for others. The largest issue is that deltas have typically pushed extra computational effort onto the users computer in compensation for the smaller size. With a fast internet connection that cost is a real burden where deltas take longer to install than simply downloading the full package.

What’s so Bad About the Old Method?

The previous idea of using the full payload for deltas was very efficient in terms of distribution, but required changes in how moss handles packages to implement it. Having the previous payload available (and being fast) means storing the old payloads on disk. This increases the storage requirements for the base system, although that can be reduced by compressing the payloads to reduce disk usage (but increasing CPU usage at install time).

To make it work well, we needed the following requirements:

Install all the files from a package and recreate the payload from the individual files. However, Smart System Management allows users to avoid installing unneeded locale files and we would not be able to recreate the full payload without them.
Alternatively we can store the full payloads on disk. Then there becomes a tradeoff from doubling storage space or additional overhead from compressing the payloads to reduce it.
Significant memory requirements to use a delta when a package is large.

In short we weren’t happy with having to increase the disk storage requirements (potentially more than 2x) or the increase in CPU time to create compressed payloads to minimize it. This was akin to the old delta model, of smaller downloads but significantly slower updates.

Exploring an Alternative Approach

Optimal compression generally benefits from combining multiple files into one archive than to compress each file individually. With zstd deltas, since you read in a dictionary (the old file), you already have good candidates for compression matching. The real question was simply whether creating a delta for each file was a viable method for delta distribution (packaged into a single payload of course).

As the real benefit of deltas is reducing download size, the first thing to consider is the size impact. Using the same package as the previous blog post (but with newer versions of QtWebEngine and zstd) we look at what happens when you delta each file individually. Note that the package is quite unique in that the largest file is 76% of the total package size and that delta creation time increased a lot for files larger than 15-20MB at maximum compression.

	Full Tarball	Individual Files	Full Tarball —zstd=chainLog=30	Individual Files —zstd=chainLog=30
Time to create	134.6s	137.6s	157.9s	150.9s
Size of delta	30.8MB	29.8MB	28.3MB	28.6MB
Peak Memory	1.77GB	1.64GB	4.77GB	2.64GB

Quite surprisingly, the delta sizes were very close! Most surprising was that without increasing the size of the chainLog in zstd, the individual file approach actually resulted in smaller deltas! We can also see how much lower the memory requirements were (and they would be much smaller when there isn’t one large file in the package!). Our locale and doc trimming of files will still work nicely, as we don’t need to recreate the locale files that aren’t installed (as we still don’t want them!).

The architecture of moss allows us to cache packages, where all cached packages are available for generating multiple system roots including with our build tool boulder without any need for the original package file. Therefore any need to retain old payloads or packages is no longer required or useful, eliminating the drawbacks of the previous delta approach. The memory requirements are also reduced as the maximum memory requirement scales with the size of the largest file, rather than the entire package (which is generally a lot bigger). There are many packages containing hundreds of MBs of uncompressed data and a few into the GBs. But the largest file I could find installed locally was only 140MB, and only a handful over 100MB. This smaller increase in memory requirements is a huge improvement and the small increase in memory use to extract the deltas is likely to go unnoticed by users.

Well everything sounds pretty positive so far, there must be some drawback?

The Impact on Package Cache Time

As the testing method for this article is quite simplistic (bash loops and calls to zstd directly), the additional overhead from creating deltas for individual files I estimated to be about 20ms compared to a proper solution. The main difference from the old delta method is how we extract the payloads and recreate the files of the new package. Using the full package you simply extract the content payload and split it into its corresponding files. The new approach requires two steps, extracting the payload (we could in theory not compress it) and then applying patches to the original files to recreate the new ones. Note that times differ slightly from the previous table due to minor variations between test runs.

	Normal Package	Individual Delta File Package
Time to Delta Files	-	148.0s (137 files)
Time to Compress Payload	78.6s	4.0s
Size of Uncompressed Payload	165.8MB	28.9MB
Size of Compressed Payload	51.3MB	28.6MB
Instructions to Extract Payload	2,876.9m (349ms)	33.1m (34ms)
Instructions to Recreate Files	-	1,785.6m (452ms)

Total instructions to Extract	2,876.9m (349ms)	1,818.7m (486ms)

What’s important to note is that is this reflects a worst case scenario for the delta approach, where all 137 files were different between the old and new version of the package. Package updates where files remain unchanged allows us to omit them from the delta package altogether! So the delta approach not only saves time downloading files, but also requires fewer CPU instructions to apply the update. It’s not quite a slam dunk though as reading the original file as a dictionary results in an increase in elapsed time of extraction (though the extra time is likely much less than the time saved downloading 20MB less!).

In Part 2 we will look at some ways we can tweak the approach to balance the needed resources for creating delta packages and to reduce the time taken to apply them.

Note: This was intended to be a 2 part series as it contains a lot of information to digest.
However, Part 2 was committed and follows below.

There’s more than one way to create a delta! This post follows on from the earlier analysis of creating some basic delta packages. Where this approach, and the use of zstd, really thrives is that it gives us options in how to manage the overhead, from creating the deltas to getting them installed locally. Next we explore some ideas of how we can minimize the caching time of delta files.

Improving Delta Cache Time

To get the best bang for your buck with deltas, it is essential to reduce the size of the larger files. My experience in testing was that there wasn’t a significant benefit from creating deltas for small files. In this example, we only create delta files when they are larger than a specific size while including the full version of files that are under the cutoff. This reduces the number of delta files created without impacting the overall package size by much.

Only Delta Files Greater Than	Greater than 10KB	Greater than 100KB	Greater than 500KB
Time to Delta Files	146.1s (72 files)	146.3s (64 files)	139.4s (17 files)
Time to Compress Payload	3.9s	4.0s	8.3s
Size of Uncompressed Payload	28.9MB	29.3MB	42.4MB
Size of Compressed Payload	28.6MB	28.7MB	30.5MB
Instructions to Extract Payload	37.8m (36ms)	34.7m (29ms)	299.1m (66ms)
Instructions to Recreate Files	1,787.7m (411ms)	1,815.0m (406ms)	1,721.4m (368ms)

Total instructions to Extract	1,825.5m (447ms)	1,849.7m (435ms)	2,020.5m (434ms)

Here we see that by not creating deltas for files under 100KB, it barely impacts the size of the delta at all, while reducing caching time by 50ms compared to creating a delta for every file (486ms from the previous blog post). It even leads to up to a 36% reduction in CPU instructions to undertake caching through the delta than using the full package. In terms of showing how effective this delta technique really is, I chose one of the worst examples and I would expect that many deltas would be faster to cache when there’s files that are exact matches between the old and new package. The largest file alone took 300ms to apply the delta, where overheads tend to scale a lot when you start getting to larger files.

There are also some steps we can take to make sure that caching a delta is almost always faster than the full package (solving the only real drawback to users), only requiring Serpent OS resources to create these delta packages.

Creating More Efficient Deltas

For this article, all the tests have been run with zstd --ultra -22 --zstd=chainLog=30…until now! The individual file delta approach is more robust at lower compression levels to keep package size small while reducing how long they take to create. Lets take a look at the difference while also ensuring --long is enabled. This testing combined with the results above for only creating deltas for files larger than 10KB.

Individual Delta File Package	zstd -12	zstd -16	zstd -19	zstd -22
Time to Delta Files	6.7s	113.9s	142.3s	148.3s
Time to Compress Payload	0.5s	3.2s	5.3s	4.0s
Size of Uncompressed Payload	41.1MB	30.6MB	28.9MB	28.9MB
Size of Compressed Payload	40.9MB	30.3MB	28.6MB	28.6MB
Instructions to Extract Payload	46.5m (35ms)	51.2m (28ms)	42.6m (33ms)	37.8m (36ms)
Instructions to Recreate Files	1,773.7m (382ms)	1,641.5m (385ms)	1,804.2m (416ms)	1,810.9m (430ms)

Total instructions to Extract	1,820.2m (417ms)	1,692.7m (413ms)	1,846.8m (449ms)	1,848.7m (466ms)

Compression levels 16-19 look quite interesting where you start to reduce the time taken to apply the delta as well and only seeing a small increase in size. For comparison, at -19 it only took 9s to delta the remaining 39.8MB of files when excluding the largest file (it was 15s at -22). While the time taken between 19 and 22 was almost the same, at -19 it took 27% fewer instructions to create the deltas than at -22 (-16 uses 64% fewer instructions than -22). It will need testing across a broader range of packages to see the overall impact and to evaluate a sensible default.

As a side effect of reducing the compression level, you also get another small decrease in the time to cache a package. The best news of all is that these numbers are already out of date. Testing was performed last year with zstd 1.5.0, where there have been notable speed improvements to both compression and decompression that have been included in newer releases. Great news given how fast it is already! Here’s a quick summary of how it all ties together.

Delta’s Are Back!

This blog series has put forward a lot of data that might be difficult to digest…but what does it mean for users of Serpent OS? Here’s a quick summary of the advantages of using this delta method on individual files when compared to fetching the full packages:

Package update sizes are greatly reduced to speed up fetching of package updates.
moss architecture means that we have no need to leave packages on disk for later use, reducing the storage footprint. In fact, we could probably avoid writes (except for the extracted package of course!) by downloading packages to tmpfs where you have sufficient free memory.
Delta’s can be used for updating packages for packaging and your installed system. There’s no need for a full copy of the full original package for packaging. A great benefit when combined with the source repository.
Delta’s are often overlooked due to being CPU intensive while most people have pretty decent internet speeds. This has a lot to do with how they are implemented.
With the current vision for Serpent OS deltas they will require fewer CPU instructions to use than full packages, but may slightly increase the time to cache some packages (but we are talking ms). But we haven’t even considered the reduction in time taken to download the delta vs the full package which more than makes up the difference!
The memory requirements are reduced compared to the prior delta approach, especially if you factor in extracting the payload in memory (possibly using tmpfs) as part of installation.

Getting More Bang for Your Buck

There’s still plenty more work to be done for implementing delta’s in Serpent OS and they likely aren’t that helpful early on. To make delta packages sustainable and efficient over the long run, we can make them even better and reduce some wastage. Here are some more ideas in how to make deltas less resource intensive and better for users:

As we delta each file individually, we could easily use two or more threads to speed up caching time. Using this package as an example, two threads would reduce the caching time to 334ms, the time the largest file took to recreate plus the time to extract the payload. Now the delta takes less time and CPU to cache than the full package!
zstd gives us options to tradeoff some increase in delta size to reduce the resources needed to create delta packages. This testing was performed with --ultra -22 --zstd=chainLog=30 which is quite slow, but produces the smallest files. Even reducing the compression level to -16 --long didn’t result in a large increase in delta size.
We always have the option to not create deltas for small packages to ease the burden, but in reality the biggest overhead is created from large files.
When creating deltas, you typically generate them for multiple prior releases. We can use smart criteria when to stop delta generation from earlier releases for instance if they save less than 10% total size or less than 100KB. A delta against an earlier release will almost always be larger than versus a more recent release.

Even Better than These Numbers Suggest

While the numbers included have been nothing short of remarkable, they don’t quite reflect how good this approach will be. The results shown lack some of the key advantages of our delta approach such as excluding files that are unchanged between the two packages. Other things that will show better results are:

When package updates include minimal changes between versions (and smaller files), we would expect the average package to be much closer in elapsed time than indicated in these results.
A quick test using a delta of two package builds of the same source version resulted in a 13MB delta (rather than the 28.6MB delta we had here). On top of that it took 62% fewer CPU instructions and less time (295ms) than the full package to extract (349ms) without resorting to multi-threading.
Delta creation of the average package will be quite fast where the largest files are < 30MB. In our example, one file is 76% of the package size (126MB) but can take nearly 95% of the total creation time!
We will be applying features to our packages that will reduce file sizes (such as the much smaller RELR relocations and identical code folding), making the approach even better, but that will be for a future blog post.

Can Hardly Contain Myself, Plus a Bonus

Jan 20, 2022

SunnyFlunk

Distro Engineer

One of the core steps for building a package is setting up a minimal environment with only the required (and stated) dependencies. Currently we have been building our stones in an systemd-nspawn container, where the root contains every package that’s been built so far. This makes the environment extremely difficult to reproduce!

Today we announce moss-container, a simple but flexible container creator that we can integrate for proper containerized builds.

Versatile For Many Use Cases

Containers have a multitude of uses for a Linux distro, but our immediate use case is for reproducible container builds for boulder. However, we have plans to use moss-container for testing, validation and benchmarking purposes as well. Therefore it’s important to consider all workloads, so features like fakeroot and networking can be toggled on or off depending on what features are needed.

moss-container takes care of everything, the device nodes in the /dev tree, mounting directories as tmpfs so the environment is left in a clean state, and mounting the /sys and /proc special file-systems. These are essential for a fully functioning container where programs like python and even clang won’t work without them. And best of all, it’s very fast so fits in well with the rest of our tooling!

The next step is integrating moss-container into boulder, so that builds become reproducible across machines, and makes it much easier for users to run builds on their host machines.

moss Now Understands Repositories

Previously (but not covered in the blogs) work was also done on moss so that it can understand and fetch stone packages from an online repo. This ties in nicely with the moss-container work and is a requirement for finishing up a proper build process for Serpent OS. We are now one step closer to having a full distribution cycle from building packages and pushing those packages as system updates!

Container with functioning device nodes

Check Out The Development

In case you’ve missed it, ikey has been streaming some of the development of the tooling on his Twitch channel. DLang is not as commonly used as other languages, so check it out to see the advantages it brings. Streams are typically announced on twitter, or give him a follow to see when he next goes live!

Bonus Content Refresh

This year we’ve had a considerable number of new visitors and interest in Serpent OS. Unfortunately the content on the website had been a bit stale and untouched for some time. There was some confusion and misunderstanding over some of the terms and content. Some of the common issues were:

Subscriptions is a loaded term relating to software
Subscriptions only referred to a fraction of the smart features
Seemed targeted at advanced users with too many technical terms
Lack of understanding around what moss and boulder do
That features would add complexity when the tools were actually removing the complexity

The good news is that a good chunk of it has been redone, including two new pages for our core tools boulder and moss. Subscriptions has been renamed to Smart System Management to reflect its broader nature (which you can read about here).

Much of the content has also had a refresh or a rewrite, so if you’ve seen it before, it will likely be a lot easier to digest now. But this isn’t the final state of the content, as more features will need to be added and there’s still a few rough edges (and I like to rewrite things every once in awhile). Many ideas have been raised by our community in the matrix channel, so a shout-out to the good folks we have hanging out there.

Performance Corner: Small Changes Pack a Punch

Dec 14, 2021

SunnyFlunk

Distro Engineer

Here we have another round of changes to make packages smaller and show just how much we care about performance and efficiency! Today we are focusing mainly on moss-format changes to reduce the size of its payloads. The purpose of these changes is to reduce the size of packages, DB storage for transactions and memory usage for moss. These changes were made just before we moved out of bootstrap, so we wouldn’t have to rebuild the world after changing the format. Lets get started!

Squeezing Out the Last Blit of Performance

Blitting in Serpent OS is the process of setting up a root, using hardlinks of files installed in the moss store to construct the system /usr directory. Some initial testing showed buildPath using about 3% of the total events when installing a package with many files. As a system is made up of over 100,000 files, that’s a lot of paths that need to be calculated! Performance is therefore important to minimize time taken and power consumed.

After running a few benchmarks of buildPath, there were a few tweaks which could improve the performance. Rather than calculating the full path of the file, we can reuse the constant root path for each file instead, reducing buildPath events by 15%. Here we see some callgrind data from this small change (as it was difficult to pickup in the noise).

Before:
   48,766,347 ( 1.50%)  /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath
   34,241,805 ( 1.05%)  /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath
   14,363,684 ( 0.44%)  /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath

After:
   40,357,800 ( 1.25%)  /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath
   29,523,966 ( 0.91%)  /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath
   12,814,283 ( 0.40%)  /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath

Only Record What’s Needed!

Our Index Payload is used for extracting the Content Payload into its hashed file names. As it has one job (and then discarded), we could hone in on including only the information we needed. Previously an entry was a 32 byte key and storing a 33 byte hash. We have now integrated the hash as a ubyte[16] field, cut other unneeded fields so that we can fit the whole entry into 32 bytes. That’s a 51% reduction in the size of the Index Payload and about 25% smaller when compressed.

Before: Index [Records: 3688 Compression: Zstd, Savings: 60.61%, Size: 239.72 KB]
After:  Index [Records: 3688 Compression: Zstd, Savings: 40.03%, Size: 118.02 KB]

Too Many Entries Makes for a Large Payload

One of the bugbears about the Layout Payload, was the inclusion of Directory paths for every single directory. This is handy in that directories can be created easily before any files are included (to ensure they exist), but it comes at a price. The issue was twofold, the extra entries made inspecting the file list take longer and also made the Layout Payload larger than it needed to be. Therefore directories are no longer included as Layout entries with the exceptions of empty directories and directories with different permissions. Lets compare nano and glibc builds before and after the change:

nano
Before: Layout [Records: 174 Compression: Zstd, Savings: 80.58%, Size: 13.78 KB]
After:  Layout [Records:  93 Compression: Zstd, Savings: 73.49%, Size:  9.15 KB]

glibc
Before: Layout [Records: 7879 Compression: Zstd, Savings: 86.88%, Size: 755.00 KB]
After:  Layout [Records: 6813 Compression: Zstd, Savings: 86.24%, Size: 689.20 KB]

A surprisingly large impact from a small change, with 1,000 fewer entries for glibc and cutting the Layout Payload size of nano by a third. What it shows is that it’s hugely beneficial where locales are involved and % size reduction increases where you have fewer files. To give an example of how bad it could be, the KDE Frameworks package ktexteditor would have produced 300 entries in the Layout Payload, where 200 of those would have been for directories! I’d estimate a 50% reduction in the Layout Payload size for this package! Here’s an example of how a locale file used to be stored (where only the last line is added now).

  - /usr/share/locale [Directory]
  - /usr/share/locale/bg [Directory]
  - /usr/share/locale/bg/LC_MESSAGES [Directory]
  - /usr/share/locale/bg/LC_MESSAGES/nano.mo -> ec5b82819ec2355d4e7bbccc7d78ce60 [Regular]

This will be exceptionally useful for keeping the LayoutDB slimmer and faster as the number of packages grows.

Cutting Back 1 Byte at a Time

Next we removed recording the timestamps of files, which for reproducible builds, is often a number of little relevance as you have to force them all to a fixed number. As moss is de-duplicating for files, there’s a second issue where two packages could have different timestamps for the same hash. Therefore it was considered an improvement to simply exclude timestamps altogether. This improved install time as we no longer overwrite the timestamps and made the payload more compressible due to replacing it with 8 bytes of padding. Unfortunately we weren’t quite able to free up 16 bytes to reduce the size of each entry, but will be something to pursue in future.

Another quick improvement was reducing the lengths of paths for each entry. moss creates system roots and switches between them by changing the /usr symlink. Therefore, all system files need to be installed into /usr or they will not make up your base system. Therefore we have no need to store /usr in the Payload so we strip /usr/ from the paths (the extra / gives us another byte off!) which we recreate on install. This improved the uncompressed size of the Payload, with only a minor reduction when compressed.

Combined these result in about a 5% decrease in the compressed and uncompressed size of the Layout Payload.

Before: Layout [Records: 6812 Compression: Zstd, Savings: 86.24%, Size: 689.10 KB]
After: Layout [Records: 6812 Compression: Zstd, Savings: 86.20%, Size: 655.00 KB]

Storing the Hash Efficiently

Using the same method for the Index Payload, we are now storing the hash as ubyte[16], but not directly in the Payload Entry. This gives us a sizeable reduction of 17 bytes per entry which is the most significant of all the Layout Payload changes. As the extra space was unneeded, it compressed well so only resulted in a small reduction in the compressed Payload size.

Before: Layout [Records: 6812 Compression: Zstd, Savings: 86.20%, Size: 655.00 KB]
After: Layout [Records: 6812 Compression: Zstd, Savings: 83.92%, Size: 539.26 KB]

Planning payload changes

Hang On, Why am I Getting Faster Installation?

As a side-effect of small code rewrites to implement these changes, we’ve seen a nice decrease in time to install packages. There are fewer entries to iterate over with the removal of directories and buildPath is now only called once for each path. It goes to show that focusing on the small details leads to less, more efficient and faster code. Here we find that we have now essentially halved the number of events related to buildPath with all the changes resulting in about a 5% reduction in install time. Note that for this test, over 80% of time is spent in zstd decompressing the package which we haven’t optimized (yet!). Here’s another look the buildPath numbers factoring in all the changes:

Before:
   48,766,347 ( 1.50%)  /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath
   34,241,805 ( 1.05%)  /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath
   14,363,684 ( 0.44%)  /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath

After:
   25,145,625 ( 0.79%)  /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath
   17,967,216 ( 0.57%)  /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath
    7,761,417 ( 0.24%)  /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath

Summing it All Up

It was a pretty awesome weekend of work (a few weeks ago now), making some quick changes that will improve Serpent OS a great deal for a long time to come. This also means we have integrated all the quick format changes so we won’t have to rebuild packages while bringing up the repos.

Here’s a quick summary of the results of all these small changes:

51% reduction in the uncompressed size of the Index Payload
25% reduction in the compressed size of the Index Payload
29-50% reduction in the uncompressed size of the Layout Payload (much more with fewer files and more locales)
12-15% reduction in the compressed size of the Layout Payload
5% faster package installation for our benchmark (800ms to 760ms)

These are some pretty huge numbers and even excluded the massive improvements we made in the previous blog!

What if We Include the Previous Changes?

I’m glad you asked, cause I was curious too! Here we see a before and after with all the changes included. For the Layout Payload we see a ~45% reduction in the compressed and uncompressed size. For the Index Payload we have reduced the uncompressed size by 67% and the compressed size by 56%. Together resulting in halving the compressed and uncompressed size of the metadata of our stone packages!

Before:
Payload: Layout [Records: 5441 Compression: Zstd, Savings: 83.13%, Size: 673.46 KB] (113.61 KB compressed)
Payload: Index [Records: 2550 Compression: Zstd, Savings: 55.08%, Size: 247.35 KB] (111.11 KB compressed)

After:
Payload: Layout [Records: 4718 Compression: Zstd, Savings: 83.62%, Size: 374.42 KB] (61.33 KB compressed)
Payload: Index [Records: 2550 Compression: Zstd, Savings: 39.85%, Size: 81.60 KB] (49.01 KB compressed)

Now we proceed towards bringing up repos to enjoy our very efficient packages!

Out of the Bootstrap - Towards Serpent OS

Dec 2, 2021

SunnyFlunk

Distro Engineer

The initial stone packages that will seed the first Serpent OS repo have now been finalized! This means that work towards setting up the infrastructure for live package updates begins now. We plan on taking time to streamline the processes with a focus of fixing issues at the source. In this way we can make packaging fast and efficient so we can spend time working on features rather than package updates.

The End of Bootstrap

Bootstrapping a distribution involves building a new toolchain and many packages needed to support it. For us bootstrap was getting us to the point where we have built stone packages that we can use to start an initial repository with full working dependencies. This has been enabled by integrating dependencies into moss, creating the first repo index. Of note is that it is already enabled for 32bit support, so we have you covered there. While this is the end of bootstrap, the fun has only just begun!

The first install from the bootstrap

Where to Next?

The next goal is to make Serpent OS self-hosting, where we can build packages in a container and update the repo index with the newly built packages. It is essentially a live repository accessible from the internet. There’s still plenty of improvements to be made with the tooling, but will soon enable more users to participate in bringing Serpent OS into fruition.

Becoming More Inclusive

While there’s a strong focus in Serpent OS on performance, the decision has been made to lower the initial requirements for users. Despite AVX2 being an older technology, there are still computers sold today that don’t support it. Because of this (and already having interested users who don’t have newer machines), the baseline requirement for Serpent OS will be x86_64-v2, which only requires SSE4.2.

It was always the plan to add support for these machines, just later down the track. In reality, this makes a lot more sense, as there will be many cases where building 2 versions of a package provides little value. This is where a package takes a long time to build and doesn’t result in a notable performance improvement. We will always need the x86_64-v2 version of a package to be compatible with the older machines. With this approach we can reduce the build server requirements without a noticeable impact to users as only a few packages you use will be without extra optimizations (and probably don’t benefit from them anyway).

I want to make it clear that this will be temporary, with impactful x86_64-v3+ packages rolling out as soon as possible. This change paves the way to integrate our technology stack taking care of your system for you and increases its priority. Users meeting the requirements of the x86_64-v3+ instruction set (this includes additional instructions beyond x86_64-v3) will automatically start installing these faster packages as they become available. Our subscriptions model will seamlessly take care of everything for you behind the scenes so you don’t need to read a wiki or forum to learn how to get these faster packages. We can utilize the same approach in future for our ARM stack, offering more optimized packages where it provides the most benefit.

Note that from the bootstrap, most packages built in under 15s and only three took longer than 2 minutes.

Trying Out Some Tweaks From the Get Go

While the project is young is a great time to test out new technologies. The use of clang and lld open up new possibilities to reduce file sizes, shorten load times and decrease memory usage. Some of these choices may have impacts on compatibility, so testing them out will be the best way to grasp that. Making sure that you can run apps like Steam is vital to the experience, so whatever happens we will make sure it works. The good news is that due to the unique architecture of Serpent OS, we can push updates that break compatibility with just a reboot, so if we ever feel the need to change the libc, we can make the change without you having to reinstall! More importantly, we can test major stack updates by rebooting into a staged update and go straight back to the old system, regardless of your file system.

Speed Packaging!

In the early days of the repository, tooling to make creating new packages as simple as possible is vital for efficiency. Spending some time automating as much of the process as possible will take weeks off bringing up Serpent OS. By making packaging as easy as possible will also help users when creating their own packages. While it would be faster to work around issues, the build tooling upgrades will benefit everyone.

The other way we’ll be speeding up the process is by holding back some of the tuning options by default. LTO for instance can result in much longer build times so will not initially be the default. The same is true for debug packages, where it slows down progress without any tangible benefit.

Things Are Happening!

We hope you are as excited as we are!

It All Depends

Nov 23, 2021

Ikey

Founder

It all depends.. it really does. On shared libraries.. interpreters.. pkg-config providers and packages. It’s the same story for all “package managers”, how do we ensure that the installed software has everything it needs at runtime?

Our merry crew has been involved in designing and building Linux distributions for a very, very long time, so we didn’t want to simply repeat history.

Using updated moss

Thanks to many improvements across our codebase, including moss-deps, we automatically analyse the assets in a package (rapidly too) to determine any dependencies we can add without requiring the maintainer to list them. This is similar in nature to RPM’s behaviour.

As such we encode dependencies into our (endian-aware, binary) format which is then stored in the local installation database. Global providers are keyed for quick access, and the vast majority of packages will not explicitly depend on another package’s name, rather, they’ll depend on a capability or provider. For subpackage dependencies that usually depend on “NEVRA” equality (i.e. matching against a name with a minimum version, release, matching epoch and architecture), we’ll introduce a lockstep dependency that can only be resolved from its origin source (repo).

Lastly, we’ll always ensure there is no possibility for “partial update” woes. With these considerations, we have no need to support >= style dependencies, and instead rely on strict design goals and maintainer responsibility.

First and foremost!

The rapid move we’re enjoying from concept, to prototype, and soon to be fully fledged Linux distribution, is only possible with the amazing community support. The last few months have seen us pull off some amazing feats, and we’re now executing on our first public milestones. With your help, more and more hours can be spent getting us ready for release, and would probably help to insulate my shed office! (Spoiler: its plastic and electric heaters are expensive =))

The Milestones

We have created our initial milestones that our quite literally our escape trajectory from bootstrap to distro. We’re considerably closer now, hence this open announcement.

v0.0: Container (`systemd-nspawn`)

Our first release medium will be a systemd-nspawn compatible container image. Our primary driver for this is to allow us to add encapsulation for our build tool, boulder, permitting us to create distributable builder images to seed our infrastructure and first public binary repository.

v0.1: Bootable “image”

Once our build infra is up and running (honestly a lot of work has been completed for this in advance) we’ll work towards our first 0.1 image. This will initially target VM usage, with a basic console environment and tooling (moss, boulder, etc).

And then..

We have a clear linear path ahead of us, with each stage unlocking the next. During the development of v0.0 and v0.1 we’ll establish our build and test infrastructure, and begin hosting our package sources and binaries. At this point we can enter a rapid development cycle with incremental, and considerable improvements. Such as a usable desktop experience and installer.. :)

Recent changes

I haven’t blogged in quite a while, as I’ve been deep in the trenches working on our core features. As we’ve expressed before, we tend to work on the more complex systems first and then glue them together after to form a cohesive hole. The last few days have involved plenty of glue, and we now have distinct package management features.

Refactor

Replaced defunct InstallDB with reusable MetaDB for local installation of archives as well as forming the backbone of repository support.
Added ActivePackagesPlugin to identify installed packages
Swapped non cryptographic hash usage with xxhash

Dependencies

Introduced new Transaction type to utilise a directed acyclical graph for dependency solving.
Reworked moss-deps into plugins + registry core for all resolution operations.
Locally provided .stone files handled by CobblePlugin to ensure we depsolve from this set too.
New Transaction set management for identifying state changes and ensuring full resolution of target system state.
Shared library and interpreter (DT_INTERP) dependencies and producers automatically encoded into packages and resolved by depsolver.

Package Installation

We handle locally provided .stone packages passed to the install command identically to those found in a repository. This eliminates a lot of special casing for local archives and allows us to find dependencies within the provided set, before looking to the system and the repositories.

Install

Dependency resolution is performed now for our package installation and is validated at multiple points, allowing a package like nano to depend on compact automatic dependencies:

  Dependency(DependencyType.SharedLibraryName, "libc.so.6(x86_64)");

Note our format and database are binary and endian aware. The dependency type only requires 1 byte of storage and no string comparisons.

List packages

Thanks to the huge refactor, we can now trivially access the installed packages as a list. This code will be reused for a list available command in future.

Example list installed output:

                   file (5.4) - File type identification utility
             file-32bit (5.4) - Provides 32-bit runtime libraries for file
       file-32bit-devel (5.4) - Provides development files for file-32bit
             file-devel (5.4) - Development files for file
                   nano (5.5) - GNU Text Editor

ListInstalled

Inspect archives

For debugging and development purposes, we’ve moved our old “info” command to a new “inspect” command to work directly on local .stone files. This displays extended information on the various payloads and their compression stats.

For general users - the new info command displays basic metadata and package dependencies.

Info

Package Removal

Upon generating a new system state, “removed” packages are simply no longer installed. As such no live mutation is performed. As of today we can now request the removal of packages from the current state, which generates a new filtered state. Additionally we remove all reverse dependencies, direct and transitive. This is accomplished by utilising a transposed copy of the directed acyclical graph, identifying the relevant subgraph and occluding the set from the newly generated state.

Remove

Lastly

The past few weeks have been especially enjoyable. I’ve truly had a fantastic time working on the project and cannot wait for the team and I to start offering our first downloads, and iterate as a truly new Linux distribution that borrows some ideas from a lot of great places, and fuses them into something awesome.

Keep toasty - this train isn’t slowing down.

Performance Corner: Faster Builds, Smaller Packages

Nov 5, 2021

SunnyFlunk

Distro Engineer

Performance Corner is a new series where we highlight to you some changes in Serpent OS that may not be obvious, but show a real improvement. Performance is a broad term that also covers efficiency, so things like making files smaller, making things faster or reducing power consumption. In general things that are unquestionably improvements with little or no downside. While the technical details may be of interest to some, the main purpose is to highlight the real benefit to users and/or developers that will make using Serpent OS a more enjoyable experience. Show me the numbers!

Here we focus on a few performance changes Ikey has been working on to the build process that are showing some pretty awesome results! If you end up doing any source builds, you’ll be thankful for these improvements. Special thanks to ermo for the research into hash algorithms and enabling our ELF processing.

Start From the Beginning

When measuring changes, it’s always important to know where you’re starting from. Here are some results from a recent glibc build, but before these latest changes were incorporated.

Payload: Layout [Records: 5441 Compression: Zstd, Savings: 83.13%, Size: 673.46 KB]
Payload: Index [Records: 2550 Compression: Zstd, Savings: 55.08%, Size: 247.35 KB]
Payload: Content [Records: 2550 Compression: Zstd, Savings: 81.46%, Size: 236.72 MB]
 ==> 'BuildState.Build' finished [4 minutes, 6 secs, 136 ms, 464 μs, and 7 hnsecs]
 ==> 'BuildState.Analyse' finished [21 secs, 235 ms, 300 μs, and 2 hnsecs]
 ==> 'BuildState.ProducePackages' finished [25 secs, 624 ms, 996 μs, and 8 hnsecs]

The build time is a little high, but a lot of that is due to a slow compiler on the host machine. But analysing and producing packages was also taking a lot longer than it needed to.

The Death of moss-jobs in boulder

In testing an equivalent build outside of boulder, the build stages were about 5% faster. Testing under perf, the jobs system was a bit excessive for the needs of boulder, polling for work when we already know the times when parallel jobs would be useful. Removing moss-jobs allowed for simpler codepaths using multiprocessing techniques from the core language. This work is integrated in moss-deps and the excess overhead of the build has now been eliminated.

Before:
 ==> 'BuildState.Build' finished [4 minutes, 6 secs, 136 ms, 464 μs, and 7 hnsecs]
 ==> 'BuildState.Analyse' finished [21 secs, 235 ms, 300 μs, and 2 hnsecs]
After:
[Build] Finished: 3 minutes, 53 secs, 386 ms, 306 μs, and 4 hnsecs
[Analyse] Finished: 8 secs, 136 ms, 22 μs, and 8 hnsecs

The new results reflect a 26s reduction in the overall build time. But only 13s of this relates to the moss-jobs removal. The other major change is making the analyse stage parallel in moss-deps (a key part of why we wanted parallelism to begin with). Decreasing the time from 21.2s to 8.1s is a great achievement despite it doing more work as we’ve also added ELF scanning for dependency information in-between these results.

New Hashing Algorithm

One of the unique features in moss is using hashes for file names which allows full deduplication within packages, the running system, previous system roots and for source builds with boulder. Initially this was hooked up using sha256, but it was proving to be a bit of a slowdown.

Enter xxhash, the hash algorithm by Yann Collet for use in fast decompression software such as lz4 and zstd (and now in many places!). This is seriously fast, with the potential to produce hashes faster than RAM can feed the CPU. The hash is merely used as a unique identifier in the context of deduplication, not a cryptographic verification of origin. XXH3_128bit has been chosen due to it having an almost zero probability of a collision across 10s of millions of files.

The benefit is actually two-fold. First of all, the hash length is halved from sha256, so there’s savings in the package metadata. This shouldn’t be understated as hash data is generally not as compressible as typical text and there are packages with a lot of files! Here the metadata for the Layout and Index payloads has reduced by 232KB! That’s about a 25% reduction with no other changes.

Before:
Payload: Layout [Records: 5441 Compression: Zstd, Savings: 83.13%, Size: 673.46 KB]
Payload: Index [Records: 2550 Compression: Zstd, Savings: 55.08%, Size: 247.35 KB]
After:
Payload: Layout [Records: 5441 Compression: Zstd, Savings: 86.66%, Size: 522.97 KB]
Payload: Index [Records: 2550 Compression: Zstd, Savings: 60.02%, Size: 165.75 KB]

Compressed this turns out to be about a 89KB reduction in the package size. For larger packages, this probably doesn’t mean much but could help a lot more with delta packages. For deltas, we will be including the full metadata of the Layout and Index payloads, so the difference will be more significant there.

The other benefit of course is the speed and the numbers speak for themselves! A further 6.4s reduction in build time removing most of the delay at the end of the build for the final package. This will also improve speeds for caching or validating a package.

Before:
 ==> 'BuildState.Analyse' finished [21 secs, 235 ms, 300 μs, and 2 hnsecs]
After:
[Analyse] Finished: 1 sec, 688 ms, 681 μs, and 8 hnsecs

With these changes combined, building packages can take 12x less time in the analyse stage, while reducing the size of the metadata and the overall package. We do expect the analyse time to increase in future as we add more dependency types, debug handling and stripping, but with the integrated parallel model, we can minimize the increase in time.

Building

We’re Not Done Yet

The first installment of Performance Corner shows some great wins to the Serpent OS tools and architecture. This is just the beginning and there will likely be a follow up soon (you may have also noticed that it takes too long to make the packages), and there’s a couple more tweaks to further decrease the size of the metadata. Kudos to Ikey for getting these implemented!

Optimal File Locality

Oct 4, 2021

SunnyFlunk

Distro Engineer

File locality in this post refers to the order of files in our content payload. Yes that’s right, we’re focused on the small details and incremental improvements that combined add up to significant benefits! All of this came about from testing the efficiency of content payload in moss-format and how well it compared against a plain tarball. One day boulder was looking extremely inefficient and then retesting the following day was proving to be extremely efficient without any changes made to boulder or moss-format. What on Earth was going on?

Making Sure you Aren’t Going Crazy!

To test the efficiency our content payload, the natural choice was to compare it to a tarball containing the same files. When first running the test the results were quite frankly awful! Our payload was 10% larger than the equivalent tarball! It was almost unbelievable in a way, so the following day I repeated the test again only this time the content payload was smaller than the tarball. This didn’t actually make sense, I made the tarball with the same files, but only changed the directory it was created from. Does it really matter?

File Locality Really Matters!

Of course it does (otherwise it would be a pretty crappy blog post!). When extracting a .stone package it creates two directories, mossExtract where the sha256sum named files are stored and mossInstall where those files are hardlinked to their full path name. The first day I created the tarball from mossInstall and the second day I realised that creating the tarball from mossExtract would provide the closest match to the content payload since it was a direct comparison. When compressing the tarballs to match the .stone compression level, the tarball compressed from mossInstall was 10% smaller, despite the uncompressed tarball being slightly larger.

Compression Wants to Separate Apples and Oranges

In simplistic terms, the way compression works is comparing data that it’s currently reading versus data that it’s read earlier in the file. zstd has some great options like --long that increases the distance in which these matches can be made at the cost of increased memory use. To limit memory use while making compression and decompression fast, it takes shortcuts that reduce the compression ratio. For optimal compression, you want files that are most similar to each other to be as close as possible. You won’t get as many matches from a text file to an ELF file as you would from a similar looking text file.

Spot the Difference

Files in mossExtract are listed in their sha256sum order, which is basically random, where files in mossInstall are ordered by their path. Sorting files by path actually does some semblance of sorting where binaries are in /usr/bin and libraries are in /usr/lib bringing them closer together. This is in no way a perfect order, but is a large improvement on a random order (up to 10% in our case!).

Our glibc package has been an interesting test case for boulder, where an uncompressed tarball of the install directory was just under 1GB. As boulder stores files by their sha256sum, it is able to deduplicate files that are the same even when the build hasn’t used symlinks or hardlinks to prevent the wasted space. In this case, the deduplication reduced the uncompressed size of the payload by 750MB alone (that’s a lot of duplicate locale data!). In the python package, it removes 1,870 duplicate cache files to reduce the installation size.

As part of the deduplication process boulder would sort files by sha256sum to remove duplicate hashes. If two files have the same sha256sum, then only one copy needs to be stored. It also felt clean with the output of moss info looking nice where the hashes are listed in alphabetical order. But it was having a significant negative impact on package sizes so that needed to be addressed by resorting the files by path order (a simple one-liner), making the content payload more efficient than a tarball once again.

Compression Level	sha256sum Order	File path Order
1	72,724,389	70,924,858
6	65,544,322	63,372,056
12	49,066,505	44,039,782
16	45,365,415	40,785,385
19	26,643,334	24,134,820
22	16,013,048	15,504,806

Testing has shown that higher compression levels (and enabling --long) is more forgiving of a suboptimal file order (3-11% smaller vs only 2-5% smaller when using --long). The table above is without --long so the difference is larger.

Hang On, Why Don’t You…

There’s certainly something to this and sorting by file order is a first step. In future we can consider creating an efficient order for files to improve locality. Putting all the ELF, image or text files together in the payload will help to shave a bit off our package sizes at only the cost to sort the files. However, we don’t want to go crazy here, the biggest impact on reducing package sizes will be using deltas as the optimal package delivery system (and there will be a followup on this approach shortly). The moss-format content payload is quite simple and contains no filenames or paths in it. Therefore it’s effectively costless to switch around the order of files, so we can try out a few things and see what happens.

An Academic Experiment

To prove the value of moss-format and the content payload, I tried out some crude sorting methods and their impact on compression for the package. As you want similar files chunked together, it divided the files into 4 groups, still sorted by their path order in their corresponding chunk:

gz: gzipped files
data: non-text files that weren’t ELF
elf: ELF files
text: text files (bash scripts, perl etc)

Path order vs optimal order

As the chart shows, you can get some decent improvements from reordering files within the tarball when grouping files in logical chunks. At the highest compression level, the package is reduced by 0.83% without any impact on compression or decompression time. In the compression world, such a change would be greatly celebrated!

Also important to note was that just moving the gzipped files to the front of the payload was able to capture 40% of the size improvement at high compression levels, but had slightly worse compression at levels 1-4. So simple changes to the order (in this case moving non-compressible files to the edge of the payload) can provide a reduction in size at the higher levels that we care about. We don’t want to spend a long time analyzing files for a small reduction in package size, so we can start off with some basic concepts like this. Moving files that don’t compress a lot such as already compressed files, images and video to the start of payload meaning that the remaining files are closer together. We also need to test out a broader range of packages and the impact any changes would have on them.

Food For Thought?

So ultimately the answer to the original question (was moss-format efficient?), the answer is yes! While there are some things that we still want to change to make it even better, in its current state package creation time was faster and overheads were lower than with compressing an equivalent tarball. The compressed tarball at zstd -16 was 700KB larger than the full .stone file (which contains a bit more data than the tarball).

The unique format also proves its worth in that we can make further adjustments to increase performance, reduce memory requirements and reduce package sizes. What this experiment shows is that file order really does matter, but using the basic sorting method of filepath gets you most of the way there and is likely good enough for most cases.

Here are some questions we can explore in future to see whether there’s greater value in tweaking the file order:

Do we sort ELF files by path order, file name or by size?
Does it matter the order of chunks in the file? (i.e. ELF-Images-Text vs Images-Text-ELF)
How many categories do we need to segregate and order?
Can we sort by extension? (i.e. for images, all the png files will be together and the jpegs together)
Do we simply make a couple of obvious changes to order and leave zstd to do the heavy lifting?

Unpacking the Build Process: Part 2

Sep 20, 2021

SunnyFlunk

Distro Engineer

Part 2 looks at the core of the build process, turning the source into compiled code. In Serpent OS this is handled by our build tool boulder. It is usually the part of the build that takes the longest, so where speed ups have the most impact. How long it takes is largely down to the performance of your compiler and what compile flags you are building with.

This post follows on from Part 1.

Turning Source into Compiled Code

The steps for compiling code are generally quite straight-forward:

Setting up the build (cmake, configure, meson)
Compiling the source (in parallel threads)
Installing the build into a package directory

This will build software compiled against packages installed on your system. It’s a bit more complicated when packaging as we first set up an environment to compile in (Part 1). But even then you have many choices to make and each can have an impact on how long it takes to compile the code. Do you build with Link Time Optimizations (LTO) or Profile Guided Optimizations (PGO), do you build the package for performance or for the smallest size? Then there’s packages that benefit considerably from individual tuning flags (like -fno-semantic-interposition with python). With so many possibilities, boulder helps us utilize them through convenient configuration options.

What Makes boulder so Special?

As I do a lot of packaging and performance tuning, boulder is where I spend most of my time. Here are some key features that boulder brings to make my life easier.

Ultimate control over build C/CXX/LDFLAGS via the tuning key
Integrated 2 stage context sensitive PGO builds with a single line workload
Able to switch between gnu and llvm toolchains easily
Rules based package creation
Control the extraction locations for multiple upstream tarballs

boulder will also be used to generate and amend our stone.yml files to take care of as much as possible automatically. This is only the beginning for boulder as it will continue to be expanded to learn new tricks to make packaging more automated and able to bring more information to help packagers know when they can improve their stone.yml, or alert them that something might be missing.

Serpent OS is focused on the performance of produced packages, even if that means that builds will take longer to complete. This is why we have put in significant efforts to speed up the compiler and setup tools in order to offset and minimize the time needed to enable greater performance.

Why do You Care so Much About Setup Time?

My initial testing focused on the performance of clang as well as the time taken to run cmake and configure. This lays the foundation for all future work in expanding the Serpent OS package archives at a much faster pace. On the surface, running cmake can be a small part of the overall build. However, it is important in that it utilizes a single thread, so is not sped up by adding more CPU cores like the compile time is. With a more compile heavy build, our highly tuned compiler can build the source in around 75s. So tuning the setup step to run in 5s rather than 10s actually reduces the overall build time by an additional 6%!

There are many smaller packages where the setup time is an even higher proportion of the overall build and becomes more relevant as you increase the numbers of threads on the builder. For example, when building nano on the host, the configure step takes 13.5s, while the build itself takes only 2.3s, so there’s significant gains to be had from speeding up the setup stage of a build (which we will absolutely be taking advantage of!).

A Closer Look at the clang Compiler’s Performance

A first cut of the compiler results were shared earlier in Initial Performance Testing, and given the importance to overall build time, I’ve been taking a closer look. In the post I said that "At stages where I would have expected to be ahead already, the compile performance was only equal" and now I have identified the discrepancy.

I’ve tested multiple configurations for the clang compiler and noticed that changing the default standard C++ library makes a difference to the time of this particular build. The difference in the two runs is compiling llvm-ar with the LLVM libraries of compiler-rt/libc++/libunwind or the GNU libraries of libgcc/libstdc++. And just to be clear, this is increasing the time of compiling llvm-ar with libc++ vs libstdc++ and not to do with the performance of either library. The clang compiler itself is built with libc++ in both cases as it produces a faster compiler.

Test using clang	Serpent LLVM libs	Serpent GNU libs	Host
`cmake` LLVM	5.89s	5.67s	10.58s
Compile -j4 `llvm-ar`	126.16s	112.51s	155.32s
`configure` gettext	36.64s	36.98s	63.55s

The host now takes 38% longer than the Serpent OS clang when building with the same GNU libraries and is much more in line with my expectations. Next steps will be getting bolt and perf integrated into Serpent OS to see if we can shave even more time off the build.

What remains unclear is whether this difference is due to something specifically in the LLVM build or whether it would translate to other C++ packages. I haven’t noticed a 10% increase in build time when performing the full compiler build with libc++ vs libstdc++.

Unpacking the Build Process: Part 1

Aug 25, 2021

SunnyFlunk

Distro Engineer

While the build process (or packaging as it’s commonly referred to) is largely hidden to most users, it forms a fundamental and important aspect to the efficiency of development. In Serpent OS this efficiency also extends to users via source based builds for packages you may want to try/use that aren’t available as binaries upstream.

The build process can be thought of in three distinct parts, setting up the build environment, compiling the source and post build analysis plus package creation. Please note that this process hasn’t been finalized in Serpent OS so we will be making further changes to the process where possible.

Setting up the Build Environment

Some key parts to setting up the build environment:

Downloading packages needed as dependencies for the build
Downloading upstream source files used in the build
Fetching and analyzing the latest repository index
Creating a reproducible environment for the build (chroot, container or VM for example)
Extracting and installing packages into the environment
Extracting tarballs for the build (this is frequently incorporated as part of the build process instead)

While the focus of early optimization work has been on build time performance, there’s more overhead to creating packages time than simply compiling code. Now the compiler is in a good place, we can explore the rest of the build process.

Packaging is More than just Compile Time

There’s been plenty of progress in speeding up the creation of the build environment such as parallel downloads to reduce connection overhead and using zstd for the fast decompression of packages. But there’s more that we can do to provide an optimal experience to our packagers.

Some parts of the process are challenging to optimize as while you can download multiple files at once to ensure maximum throughput, you are still ultimately limited by your internet speed. When packaging regularly (or building a single package multiple times), downloaded files are cached so become a one off cost. One part we have taken particular interest in speeding up is extracting and installing packages into the environment.

Eight seconds (for a small number of dependencies) that don't need to be endlessly repeated

The Dynamic Duo: boulder and moss

Installing packages to a clean environment can be the most time consuming part of setting up the build (excluding fetching files which is highly variable). Serpent OS has a massive advantage with the design of moss where packages are cached (extracted on disk) and ready to be used by multiple roots, including the creation of clean build environments for boulder. Having experienced a few build systems in action, setting up the root could take quite some time with a large number of dependencies (even getting over a minute). moss avoids the cost of extracting packages entirely every build by utilizing its cache!

There are also secondary benefits to how moss handles packages via its caches where disk writes are reduced by only needing to extract packages a single time. But hang on, won’t you be using tmpfs for builds? Of course we will have RAM builds as an option and there are benefits there too! When extracting packages to the RAM disk, it consumes memory which can add up to more than a GB before the build even begins. moss allows for us to start with an empty tmpfs so we can perform larger builds before exhausting the memory available on our system.

Another great benefit is due to the atomic nature of moss. This means that we can add packages to be cached as soon as they’re fetched while waiting for the remaining files to finish downloading (both for boulder and system updates). Scheduling jobs becomes much more efficient and we can have the build environment available in moments after the last file is downloaded!

moss allows us to eliminate one of the bigger time sinks in setting up builds, enabling developers and contributors alike to be more efficient in getting work done for Serpent OS. With greater efficiency it may become possible to provide a second architecture for older machines (if the demand arises).

Part 1?

Yes, there’s plenty more to discuss so there will be more follow up posts showing the cool features Serpent OS is doing to both reduce the time taken to build packages and in making packages easier to create so stay tuned!

A Rolling Boulder Gathers No Moss

Aug 10, 2021

Ikey

Founder

We actually did it. Super pleased to announce that moss is now capable of installing and removing packages. Granted, super rough, but gotta start somewhere right?

Transactional system roots + installation in moss

OK let’s recap. A moss archive is super weird, and consists of multiple containers, or payloads. We use a strongly typed binary format, per-payload compression (Currently zstd), and don’t store files in a typical archive fashion.

Instead a .stone file (moss archive) has a Content Payload, which is a compressed “megablob” of all the unique files in a given package. The various files contained within that “megablob” are described in an IndexPayload, which simply contains some IDs and offsets, acting much like a lookup table.

That data alone doesn’t actually tell us where files go on the filesystem when installed. For that, we have a specialist Layout Payload, encoding the final layout of the package on disk.

As can be imagined, the weirdness made it quite difficult to install in a trivial fashion.

Databases

Well, persistence really. Thanks to RocksDB and our new moss-db project, we can trivially store information we need from each package we “precache”. Primarily, we store full system states within our new StateDB, which at present is simply a series of package ID selections grouped by a unique 64-bit integer.

Additionally we remember the layouts within the LayoutDB so that we can eventually recreate said layout on disk.

Precaching

Before we actually commit to an install, we try to precache all of the stone files in our pool. So we unpack the content payload (“megablob”), split it into various unique files in the pool ready for use. At this point we also record the Layouts, but do not “install” the package to a system root.

Blitting

This is our favourite step. When our cache is populated, we gather all relevant layouts for the current selections, and then begin applying them in a new system root. All directories and symlinks are created as normal, whereas any regular file is hardlinked from the pool. This process takes a fraction of a second and gives us completely clean, deduplicated system roots.

Currently these live in /.moss/store/root/$ID/usr. To complete the transaction, we update /usr to point to the new usr tree atomically assuming that a reboot isn’t needed. In future, boot switch logic will update the tree for us.

Removal

Removal is quite the same as installation. We simply remove the package IDs from the new state selections (copied from the last state) and blit a new system root, finally updating the atomic /usr pointer.

Removal

Tying it all together

We retain classic package management traits such as having granular selections, multiple repositories, etc, whilst sporting advanced features like full system deduplication and transactions/rollbacks.

When we’re far enough along, it’ll be possible to boot back to the last working transaction without requiring an internet connection. Due to the use of pooling and hardlinks, each transaction tree is only a few KiB, with files shared between each transaction/install.

On the list..

We need some major cleanups, better error handling, logging, timed functions, and an eventloop driven process to allow parallel fetching/precaching prior to final system rootfs construction.

It’s taken us a very long time to get to this point, and there is still more work to be done. However this is a major milestone and we can now start adding features and polish.

Once the required features are in place, we’ll work on the much needed pre alpha ISO :) If you fancy helping us get to that stage quicker, do check out our OpenCollective! (We won’t limit prealpha availability, don’t worry :))

Moss DB Progress

Aug 3, 2021

Ikey

Founder

I’ll try to make this update as brief as I can but it’s certainly an important one, so let’s dive right into it. The last few weeks have been rough but work on our package manager has still been happening. Today we’re happy to reveal another element of the equation: moss-db.

moss-db is an abstract API providing access to simplistic “Key Value” stores. We had initially used some payload based files as databases but that introduced various hurdles, so we decided to take a more abstract approach to not tie ourselves to any specific implementation of a database.

Our main goal with moss-db is to encapsulate the RocksDB library, providing sane, idiomatic access to a key value store.

High level requirements

At the highest level, we needed something that could store arbitrary keys and values, grouped by some kind of common key (commonly known as “buckets”). We’ve succeeded in that abstraction, which also required us to fork a rocksdb-binding to add the Transform APIs required.

Additionally we required idiomatic range behaviours for iteration, as well as generic access patterns. To that affect we can now foreach a bucket, pipe it through the awesomely powerful std.algorithm APIs, and automatically encode/decode keys and values through our generic APIs when implementing the mossdbEncode() and mossdbDecode() functions for a specific type.

In a nutshell, this was the old, ugly, hard way:

  /* old, hard way */
  auto nameZ = name.toStringz();
  int age = 100;
  ubyte[int.sizeof] ageEncoded = nativeToBigEndian(ageEncoded);
  db.setDatum(cast(ubyte[]) (nameZ[0 .. strlen(nameZ)]), ageEncoded);

And this is the new, shmexy way:

    db.set("john", 100);
    db.set("user 100", "bobby is my name");

    auto result = db.get!int("john");
    if (result.found)
    {
        writeln(result.value);
    }

    auto result2 = db.get!string("user 100");
    if (result2.found)
    {
        writeln(result2.value);
    }

It’s quite easy to see the new API lends itself robustly to our needs, so that we may implement stateful, strongly typed databases for moss.

Next Steps

Even though some APIs in moss-db may still be lacking (remove, for example) we’re happy that it can provide the foundation for our next steps. We now need to roll out the new StateDB, MetaDB and LayoutDB, to record system states, package metadata, and filesystem layout information, respectively.

With those 3 basic requirements in place we can combine the respective works into installation routines. Which, clearly, warrants another blog post … :)

For now you can see the relevant projects on our GitLab project.

Initial Performance Testing

Aug 2, 2021

SunnyFlunk

Distro Engineer

With further progress on boulder, we can now build native stone packages with some easy tweaks such as profile guided optimizations (PGO) and link time optimizations (LTO). That means we can take a first look at what the performance of the first cut of Serpent OS shows for the future. The tests have been conducted using benchmarking-tools with Serpent OS measured in a chroot on the same host with the same kernel and config.

One of the key focuses for early in the project is on reducing build time. Every feature can either add or subtract from the time it takes to produce a package. With a source/binary hybrid model, users will greatly benefit from the faster builds as well. In terms of what I’ve targeted in these tests is the performance of clang and testing some compiler flag options on cmake.

Clang Shows its Promise

clang has always been a compiler with a big future. The performance credentials have also been improving each release and are now starting to see it perform strongly against its GNU counterpart. It is common to hear that clang is slow and produces less optimized code. I will admit that most distros provide a slow build of clang, but that will not be the case in Serpent OS.

It is important to note that in this comparison the Host distro has pulled in some patches from LLVM-13 that greatly improve the performance of clang. Prior to this, their tests actually took 50% longer for cmake and configure but only 10% longer for compiling. boulder does not yet support patching in builds so the packages are completely vanilla.

Test using clang	Serpent	Host	Difference
`cmake` LLVM	5.89s	10.58s	79.7%
Compile -j4 `llvm-ar`	126.16s	155.32s	23.1%
`configure` gettext	36.64s	63.55s	73.4%

Based on the results during testing, the performance of clang in Serpent OS still has room to improve and was just a quick tuning pass. At stages where I would have expected to be ahead already, the compile performance was only equal (but cmake and configure were still well ahead).

GCC Matters Too!

While clang is the default compiler in Serpent OS, there may be instances where the performance is not quite where it could be. It is common to see software have more optimized code paths where they are not tested with clang upstream. As an example, here’s a couple of patches in flac (1, 2) that demonstrate this being improved. Using benchmarking-tools, it is easy to see where gcc and clang builds are running different functions via perf results.

In circumstances where the slowdown is due to hitting poor optimization paths in clang, we always have the option to build packages using gcc, where the GNU toolchain is essential for building glibc. Therefore having a solid GNU toolchain is important but small compile time improvements won’t be noticed by users or developers as much.

Test using gcc	Serpent	Host	Difference
`cmake` LLVM	7.00s	7.95s	13.6%
Compile `llvm-ar`	168.11s	199.07s	18.4%
`configure` gettext	45.45s	51.93s	14.3%

An OS is More Than Just a Compiler

While the current bootstrap exists only as a starting point for building the rest of Serpent OS, there are some other packages we can easily test and compare. Here’s a summary of those results.

Test	Serpent	Host	Difference
Pybench	1199.67ms	1024.33ms	-14.6%
`xz` Compress Kernel (-3 -T1)	42.67s	46.57s	9.1%
`xz` Compress Kernel (-9 -T4)	71.25s	76.12s	6.8%
`xz` Decompress Kernel	8.03s	8.18s	1.9%
`zlib` Compress Kernel	12.60s	13.17s	4.5%
`zlib` Decompress Kernel	5.14s	5.21s	1.4%
`zstd` Compress Kernel (-8 -T1)	5.77s	7.06s	22.3%
`zstd` Compress Kernel (-19 -T4)	51.87s	66.52s	28.3%
`zstd` Decompress Kernel	2.90s	3.08s	6.3%

State of the Bootstrap

From my experiences with testing the bootstrap, it is clear there’s some cobwebs in there that require some more iterations of the toolchain. There also seems to be some slowdowns in not including all the dependencies of some packages. Once more packages are included, naturally all the testing will be redone and help influence the default compiler flags of the project.

It’s not yet clear the experience of using libc++ vs libstdc++ with the clang compiler. Once the cobwebs are out and Serpent OS further developed, the impact (if any) should become more obvious. There are also some parts not yet included in boulder such as stripping files, LTO and other flags by default that will speed up loading libraries. At this stage this is deliberate until integrating outputs from builds (such as symbol information).

But this provides an excellent platform to build out the rest of the OS. The raw speed of the clang compiler will make iterating and expanding the package set a real joy!

Hang On, What’s Going on With Python?

Very astute of you to notice! python in its current state is an absolute minimal build of python in order to run meson. However, I did an analyze run in benchmarking-tools where it became obvious that they were doing completely different things.

Apples and oranges comparison

For now I’ll simply be assuming this will sort itself out when python is built complete with all its functionality. And before anyone wants to point the finger at clang, you get the same result with gcc.

Boulder Keeps On Rolling

Jul 27, 2021

SunnyFlunk

Distro Engineer

Squirrelling away in the background has been some great changes to bring boulder closer to its full potential. Here’s a quick recap of some of the more important ones.

Boulder hard at work

Key Changes to Boulder

Fixed a path issue that prevented manifests from being written for 32bit builds
Added keys to control where the tarballs are extracted to
- This results in a greatly simplified setup stage when using multiple upstreams
More customizations to control the final c{,xx}flags exported to the build
Added a key to run at the start of every stage so definitions can be exported easily in the stone.yml file
Fixed an issue where duplicate hash files were being included in the Content Payload
- This resulted in reducing the Content Payload size by 750MB of a glibc build with duplicate locale files
Finishing touches on profile guided optimization (PGO) builds - including clang’s context-sensitive PGO
- Fixed a few typos in the macros to make it all work correctly
- Profile flags are now added to the build stages
- Added the llvm profile merge steps after running the workload
- Recreate a clean working directory at the start of each PGO phase

With all this now in place, the build stages of boulder are close to completion. But don’t worry, there’s plenty more great features to come to make building packages for Serpent OS simple, flexible and performant. Next steps will be testing out these new features to see how much they can add to the overall stage4 performance.

Let There Be Databases

May 18, 2021

Ikey

Founder

We haven’t been too great on sharing progress lately, so welcome to an overdue update on timelines, progress, and database related shmexiness.

Emerging DB design

OK, so you may remember moss-format, our module for reading and writing moss binary archives. It naturally contains much in the way of binary serialisation support, so we’ve extended the format to support “database” files. In reality, they are more like tables binary encoded into a single file, identified by a filepath.

The DB archives are currently stored without compression to ensure 0-copy mmap() access when loading from disk, as a premature optimisation. This may change in future if we find the DB files taking up too much disk space.

So far we’ve implemented a “StateMetaDB”, which stores metadata on every recorded State on the system, and right now I’m in the progress of implementing the “StateEntriesDB”, which is something akin to a binary encoded dpkg selections file with candidate specification reasons.

Next on the list is the LayoutsDB (file manifests) and the CacheDB, for recording refcounts of every cached file in the OS pool.

Integration with Serpent ECS

An interesting trial we’re currently implementing is to hook the DB implementation up to our Entity Component system from the Serpent Engine, in order to provide fast, cache coherent, in memory storage for the DB. It’s implemented using many nice DLang idioms, allowing the full use of std.algorithm APIs:

    auto states()
    {
        import std.algorithm : map;

        auto view = View!ReadOnly(entityManager);
        return view.withComponents!StateMetaArchetype
            .map!((t) => StateDescriptor(t[1].id, t[3].name, t[4].description,
                    t[1].type, t[2].timestamp));
    }
    ...

  /* Write the DB back in ascending numerical order */
  db.states
    .array
    .sort!((a, b) => a.id < b.id)
    .each!((s) => writeOne(s));

Tying it all together

Ok, so you can see we need basic DB types for storing the files for each moss archive, plus each cache and state entry. If you look at the ECS code above, it becomes quite easy to imagine how this will impact installation of archives. Our new install code will simply modify the existing state, cache the incoming package, and apply the layout from the DB to disk, before committing the new DB state.

In essence, our DB work is the current complex target, and installation is a <50 line trick tying it all together.

  /* Pseudocode */

  State newState...

  foreach (pkgID; currentState.filter!((s) => s.reason == SelectionReason.Explicit))
  {
    auto fileSet = layoutDB.get(pkgID);
    fileSet.array.sort!((a, b) => a.path < b.path).each!((f) => applyLayout(f));
    /* Record into new state */
    ...
  }

Til next time -

Ikey

Moss Unlocked

Mar 20, 2021

Ikey

Founder

Well, it’s not all doom and gloom these days. We’ve actually made some significant progress in the last few days, so it seems a good time to share a progress update.

Extracting content from moss archives

moss can now extract

Oh yeah, that totally happened. So, we can now successfully build moss packages from boulder and then extract them to disk once again with moss. This might sound totally uninteresting, but it demonstrates that our format is actually working as intended.

Admittedly the code is super rough within moss and somewhat proof of concept, however we’re able to extract the contents of the moss archive and rebuild the layout on disk.

What makes extraction difficult?

Well, quirky new format for one. A moss archive currently consists of 4 “payloads”, or major sections:

MetaPayload

Contains all package information, with strongly typed keys.
IndexPayload

Contains the IDs of all unique files (hash) and their offsets within the ContentPayload
LayoutPayload

A sequence of specialised structs describing the final “layout” of the package on disk, with attributes, paths, and for regular files, the ID of the file in the ContentPayload to build this file from.
ContentPayload

A binary blob containing every unique file from the package, in an order described by the IndexPayload. The files are stored sequentially with no gaps.

Additionally, each payload is independently compressed using zstd. In order to extract files to disk, we must first decompress ContentPayload to a temporary file. Next, we blit each file from the “megablob” to the cache store, using the IndexPayload to understand the offsets. Finally, we apply the instructions in LayoutPayload to construct the final layout on disk, hardlinking the cache assets into their final locations, setting attributes, etc.

Net result? Deduplication on a per package basis, and a system-wide deduplication policy allowing sharing of identical assets on disk between multiple packages. This will also power our core update mechanism, whereby each update is atomic, and is simply the difference on disk from the previous version, permitting a powerful rollback mechanism.

Room for improvement

Multiple

There are areas where we’re doing things inefficiently, and we’ll certainly improve that in future revisions of the important. For example, IndexPayload actually wastes some bytes by storing redundant information that can be calculated at runtime. Additionally, we want to use the zstd C APIs directly to gain the level of control we actually need. We’re also going to wrap the copy_file_range syscall to make extraction of the content payload more efficient and not rely on userspace copies.

However, we’re working towards a prealpha, and some inefficiencies are OK. Our first port of call will be a prealpha image constructed from .stone files, produced by boulder, installed by moss. This will validate our toolchain and tooling and serve as a jumping off point for the project.

Stay tuned, there is a whole bunch of awesome coming now that moss is officially unlocked and progressing.

And finally

I want to thank everyone who is currently supporting the project. I also want to personally thank you for your understanding of the setbacks of real life, given the difficult times myself and my family have been going through. I hope it is clear that I remain committed to the project and it’s future, which is why we’re transparently run and funded via OpenCollective.

Despite the rough times, work continues, and awesome people join our ranks on a regular basis. Stability is on the immediate horizon and work with Serpent OS grows exponentially. You can be part of our journey, and help us build an amazing community and project that outlives us all.

Optimising Package Distribution

Mar 16, 2021

SunnyFlunk

Distro Engineer

Getting updates as fast as possible to users has made deltas a popular and sought after feature for distributing packages. Over the last couple of days, I’ve been investigating various techniques we can look at to support deltas in moss.

Trade-offs Between Packages and Deltas

Minimising the size of updates is particularly valuable where files are downloaded many times and even better if they’re updated infrequently. With a rolling release, packages will be updated frequently, so creating deltas can become resource intensive, especially if supporting updates over a longer period of time. Therefore it’s important to get the right balance between compression speed, decompression memory and minimising file size.

	Package priorities	How best to meet these needs
Developers	Creation speed	Quickly created packages
Users	File size and update speed	Size minimised deltas

From the users point of view, minimising file size and upgrade time are important priorities, but for a developer, the speed at which packages are created and indexed is vital to progression. Deltas are different to packages in that they aren’t required immediately, so there’s minimal impact in taking longer to minimise their size. By getting deltas right, we can trade-off the size of packages to speed up development, while users will not be affected and fetch only size optimised deltas.

Test Case - QtWebEngine

QtWebEngine provides a reasonable test case where the package is a mix of binaries, resources and translations, but above average in size (157.3MB uncompressed). The first trade-off for speed over size has already been made by incorporating zstd in moss over xz, where even with max compression zstd is already 5.6% larger than using xz. This is of course due to the amazing decompression speeds where zstd is magnitudes faster.

Compression levels with zstd

With maximum compression, large packages can take over a minute to compress. With a moderate increase in size, one can reduce compression time by 2-10x. While making me happier as a developer, it does create extra network load during updates.

Full Package	zstd -16 -T0	zstd -16	zstd -19 -T0	zstd -19	zstd -22	xz -9
Time to create	5.4s	26.8s	27.8s	56.0s	70.6s	66.5s
Size of package	52.6MB	52.6MB	49.2MB	49.2MB	48.4MB	45.9MB

Deltas to the Rescue!

There are two basic methods for deltas. One simple method is to include only files that have been changed since the earlier release. With reproducible builds, it is typical to create the same file from the same inputs. However, with a rolling release model, files will frequently have a small change from dependency changes and new versions of the package itself. In these circumstances the delta size starts to get closer to the full package anyway. As a comparison to other delta techniques, this approach resulted in a 38.2MB delta as it was a rebuild of the same version at a different time so the resources and translations were unchanged (and therefore omitted from the delta).

An alternative is a binary diff, which is a significant improvement when files have small changes between releases. bsdiff has long been used for this purpose and trying it out (without knowing much about it) I managed to create a delta of 33.2MB, not a bad start at all.

To highlight the weakness of the simple method, when you compare the delta across a version change, the simple delta was only a 7% reduction of the full package (as most files have changed), while using an optimal binary diff, it still managed to achieve a respectable 31% size reduction.

A New Contender

While looking into alternatives, I happened to stumble across a new feature in zstd which can be used to create deltas. As we already use zstd heavily it should make integration easier. --patch-from allows zstd to use the old uncompressed package as a dictionary to create the newer package. In this way common parts between the releases will be reused in order to reduce the file size. Playing around I quickly achieved the same size as bsdiff, and with a few tweaks was able to further reduce the delta by a further 23.5%! The best part is that it has the same speedy decompression as zstd, so it will recreate most packages from deltas in the blink of an eye!

Delta	only changed files	bsdiff	zstd -19	zstd -22	zstd -22 —zstd=chainLog=30
Time to create	60.8s	153.0s	85.5s	111.6s	131.8s
Size of delta	38.2MB	33.2MB	33.3MB	28.5MB	25.4MB

Next Steps

There’s certainly a lot of information to digest, but the next step is to integrate a robust delta solution into the moss format. I really like the zstd approach, where you can tune for speed with an increase in size if desired. With minimising on delta size, users can benefit from smaller updates while developers can benefit from faster package creation times.

Some final thoughts for future consideration:

zstd has seen many improvements over the years, so I believe that ratios and performance will see incremental improvements over time. Release 1.4.7 already brought significant improvements to deltas (which are reflected in this testing).
The highest compression levels (--ultra) are single threaded in zstd, so delta creation can be done in parallel to maximise utilisation.
Over optimising the tunables can have a negative impact on both speed and size. As an example, --zstd=targetLength=4096 did result in a 2KB reduction in size at the same speed, but when applied to different inputs (kernel source tree), it not only made it larger by a few hundred bytes, but added 4 minutes to delta creation!
Memory usage of applying deltas can be high for large packages (1.74GB for the kernel source tree) as it has to ingest the full size of the original uncompressed package. It is certainly possible to split up payloads with some delta users even creating patches on a per file basis. It is a bit more complicated when library names change version numbers each release with only the SONAME symlink remaining unchanged.
There’s always the option to repack packages at higher compression levels later (when deltas are created). This solves getting the package ‘live’ ASAP and minimises the size (eventually), but adds some complication.

Moss Format: Read Write Support

Feb 17, 2021

Ikey

Founder

It’s been 8 days since our last blogpost and a lot of development work has happened in that time. Specifically, we’ve completely reworked the internals of the moss-format module to support read/write operations.. Which means installation is coming soon (TM)

Development work on moss-format

So, many commits have been made to the core repositories, however the most important project to focus on right now is moss-format, which we used to define and implement our binary and source formats. This module is shared between boulder, our build tool, and moss, our package manager.

We’ve removed the old enumeration approach from the Reader class, instead requiring that it processes data payloads in-place, deferring reading the content payload streams. We’ve also enforced strong typing to allow safe and powerful APIs:

        import moss.format.binary.payload.meta : MetaPayload;

        auto reader = new Reader(File(argv[0], "rb"));
        auto metadata = reader.payload!MetaPayload();

Right now we can read and write our MetaPayload from and to the stream, allowing us to encode & decode the metadata associated with the package, with strong type information (i.e. Uint64, String, etc.)

Next Steps

We need to restore the IndexPayload, LayoutPayload and ContentPayload definitions. The first two are simply data payloads and will largely follow the design of the newly reimplemented MetaPayload. Then we restore ContentPayload support, and this will allow the next steps: unpack, install.

Many of the babysteps required are done now, which power our binary format. The design of the API is done in a way which will allow powerful manipulation via the std.algorithm and std.range APIs, enabling extremely simple and reliable installation routines.

It might seem odd that we’ve spent so much time on the core format, however I should point out that the design of the format is central to the OS design. Our installation routine is not unpacking of an archive.

With our binary format, the stream contains multiple PayloadHeaders, with fixed lengths, type, tag, version, and compression information. It is up to each Payload implementation to then parse the binary data contained within. Currently our Payloads are compressed using ZSTD, though other compression algorithms are supported.

So, we have the MetaPayload for metadata. Additionally, we encode all unique files in the package in a single compressed payload, the ContentPayload. The offset to each unique file (by hash) is stored within the IndexPayload, and the filesystem layout is encoded as data within the LayoutPayload.

In order to unpack a moss package, the ContentPayload blob will be decompressed to a temporary location. Then, each struct within the IndexPayload can be used to copy portions (copy_file_range) of the blob to the cache store for permanence. We skip each Index if its already present within the cache. Finally, the target filesystem is populated with a view of all installed packages using the LayoutPayload structs, creating a shallow installation using hardlinks and directories.

The net result is deep deduplication, atomic updates, and flexibility for the user. Once we add transactions it’ll be possible to boot an older version of the OS using the deduplication capabilities for offline recovery. Additionally there is no requirement for file deletion, rename or modification for an update to succeed.

Nutshell

Huge progress. Major excitement. Such wow. Soon alphas.

Unlocking Moss

Feb 9, 2021

Ikey

Founder

Wait, what? Another blog post? In the same WEEK? Yeah totally doing that now. So, this is just another devlog update but there have been some interesting updates that we’d like to share.

LDC present in the bootstrap

Thanks to some awesome work from Peter, we now have LDC (The LLVM D Lang Compiler) present in the stage3 bootstrap. To simplify the process we use the official binary release of LDC to bootstrap LDC for Serpent OS.

In turn, this has allowed us to get to a point where we can now build moss and boulder within stage3. This is super important, as we’ll use the stage3 chroot and boulder to produce the binary packages that create stage4.

Some patching has taken place to prevent using ld.gold and instead use lld to integrate with our toolchain decisions.

Reworking the Payload implementation

Originally our prototype moss format only contained a ContentPayload for files, and a MetaPayload for metadata, package information, etc. As a result, we opted for simple structs, basic case handling and an iterable Reader implementation.

As the format expanded, we bolted deduplication in as a core feature. To achieve this, we converted the ContentPayload into a “megablob”, i.e. every unique file in the package, one after the other, all compressed in one operation. We then store the offsets and IDs of these files within an IndexPayload to allow splitting the “megablob” into separate, unique assets. Consequently, we added a LayoutPayload which describes the final file system layout of the package, referencing the unique ID (hash) of the asset to install.

So, while the format grew into something we liked, the code supporting it became very limiting. After many internal debates and discussions, we’re going to approach the format from a different angle on the code front.

It will no longer be necessary/possible to iterate payloads in sequence within a moss archive, instead we’ll preload the data (unparsed) and stick it aside when reading the file, winding it to the ContentPayload segment if found. After initial loading of the file is complete, the Reader API will support retrieval (and lazy unpacking ) of a data segment. In turn this will allow code to “grab” the content, index and layout payloads and use advanced APIs to cache assets, and apply them to disk in a single blit operation.

In short, we’ve unlocked the path to installing moss packages while preserving the advanced features of the format. The same new APIs will permit introspection of the archives metadata, and of course, storing these records in a stateful system database.

Thank you for the unnecessary update

Oh you’re quite welcome :P Hopefully now you can see our plan, and that we’re on track to meet our not-28th target. Sure, some code needs throwing away, but all codebases are evolutionary. Our major hurdle has been solved (mentally) - now it’s just time to implement it and let the good times roll.

Cleanups Complete

Feb 8, 2021

Ikey

Founder

Well, we’re officially back to working around the clock. After spending some time optimising my workflow, I’ve been busy getting the entire codebase cleaned up, so we can begin working towards an MVP.

Lots and lots of work

Codebase hygiene

Since Friday, I’ve been working extensively on cleaning up the codebase for the following projects:

boulder
moss
moss-core
moss-format

As it now stands, 1 lint issue (a non-issue, really) exists across all 4 projects. A plethora of issues have been resolved, ranging from endian correctness in the format to correct idiomatic D-lang integration and module naming.

Bored

Granted, cleanups aren’t all that sexy. Peter has been updating many parts of the bootstrap project, introducing systemd 247.3, LLVM 11.0.1, etc. We now have all 3 stages building correctly again for the x86_64 target.

And then…

Yikes, tough audience! So we’ve formed a new working TODO which will make its way online as a public document at some point. The first stage, cleanups, is done. Next is to restore feature development, working specifically on the extraction and install routines. From there, much work and cadence will be unlocked, allowing us to work towards an MVP.

TODO

You keep saying MVP..

I know, it makes me feel all … cute and professional. At the moment we’re cooking up a high level description of how an MVP demonstration could be deployed. While most of the features are ambiguous, our current ambition is to deploy a preinstalled QCOW2 Serpent OS image as a prealpha to help validate moss/boulder/etc.

It’ll be ugly. Likely slow. Probably insecure as all hell. But it’ll be something. It’ll allow super basic interaction with moss, and a small collection of utilities that can be used in a terminal environment. No display whatsoever. That can come in a future iteration :)

ETA? ~~Definitely not by the 28th of February~~. When it’s done.

Build System Functional

Oct 11, 2020

Ikey

Founder

Wow, has it been a hectic few weeks, and it definitely shows: last time we blogged it was about re-bootstrapping with glibc. Feels like ancient news already! So, what’s new in the world of Serpent OS? Apart from yours truly now being proud parent to a beautiful baby girl, work has resumed on the development of Moss, our package manager. And it builds stuff. Awesomely. Let’s quickly catch up with those updates and see where we’re headed next.

Look at all the buildiness

It lives!

We’re now able to build the majority of a moss package. Notice I’ve made a distinction there. So, we’re able to run the build instructions, and use all of the metadata, configuration and macros available. The only thing we’re not actually doing is dumping the built files into a binary package.

We’re able to do the build-system part rather well, now. Right now we support the following features in the build system:

Multiple, profile-based architecture support
Automatic -m32 profile based cross compilation support for 32-bit libraries on 64-bit OS
Basic Profile Guided Optimisation via the workload key. Set a workload, optimise accordingly.
LLVM Context Sensitive Profile Guided Optimisation. This is default for LLVM with the workload key, and comes for free, with multiple build stages.
Profile based tuning options, such as optimize for speed, size, disabling hardening, etc.
Trivially switch between gnu and llvm toolchain in stone.yml, with profiles knowing the right flags to use with tuning options.
Recursive macro support in build scripts, defined on per-profile level
Architecture specific profiles support in stone.yml, i.e. profiles -> aarch64 -> setup: to avoid if/else spaghetti.

Next on the agenda

Now that the huge amount of scaffolding work has been done, we can actually turn the results of builds into installable binary packages using our moss.format.binary module. We’ll add some magic sauce to have automatic subpackages + inter-package dependencies, along with the expected automatic runtime dependencies + such. Cue some linting, et voila, build tool that’s a pleasure to work with.

Once we have all those packages building, we’ll need a way to install them. Luckily some scaffolding is in place for this already, and it won’t take much effort to support moss install somepkg.stone. Then we throw a few dozen packages in the mix, add some dependency handling, repo support, Bob’s your uncle, and your aunt is downloading exclusive early-access images from our Open Collective as soon as they’re available. :O

Results Of The Experiment

Sep 22, 2020

Ikey

Founder

import FigureScreenshotOne from ”@/components/ui/FigureScreenshotOne.astro” import featuredImage from ”./Featured.webp”

It seems like only yesterday we announced to the world a Great Experiment. It was in fact 2 months ago, and a whole lot of work has happened since that point. A few take-homes are immediately clear, the primary one being the need to be a community-oriented Linux distribution.

To quote ourselves 2 months ago:

 If the experiment is a success, which of course means having tight controls on scope and timescale,
 then one would assume the primary way to use Serpent OS would be through some downstream repackaging
 or reconfiguration, i.e. basing upon the project, to meet specific user demand.

It turns out that so far the experiment has been successful, and being forkable is still at the very heart of our goals. Others have joined us on our journey, and expressed the same passion in our goals as we have. A community has formed around the project, with individuals sharing the same ambitions for a reliable, rolling operating system with a powerful package manager.

An open community

Over the past 2 months we’ve transformed from set of ideas into a transparent, community-first organisation with clear leadership and open goals. I’ve stepped into the Benevolent Dictator For Life position, and Peter has taken on daily responsibilities for the project running. Aydemir is now our treasurer on Open Collective, and many individuals contribute to our project.

Transactional package management

No need to rehash this, but the defining feature of Serpent OS has clearly become moss, something initially not anticipated when we started. A read-only root filesystem, transactional operations, rollbacks, deduplicating throughout and atomic updates. Combine that with a rolling release model, stateless policy and ease-of-use, the core feature-set is already powerful.

Deferral of musl integration

In recent weeks we’ve been working on libwildebeest and libc-support, primarily as a stop-gap to provide glibc compatibility without using glibc. While musl has many advantages, it is clear to us now that writing another libc through our support projects isn’t what we originally planned. With that in mind we’re adopting glibc and putting our musl works under community ownership, until such time as reevaluation shows that musl is what is needed in Serpent OS. Note the primary motivator here is investing our efforts where it makes sense, and obtaining the best results in the most manageable fashion for our users.

Our new toolchain configuration will be as follows:

LLVM/clang as primary toolchain
libc++ as C++ library
glibc as C library
gcc+binutils (ld.bfd) provided to build glibc, elfutils, etc.
Permitting per-package builds using GCC, i.e. kernel to alleviate Clang IAS issues.

Thus our toolchain will in fact be a hybrid GNU/LLVM one. This will allow both source and binary compatibility with the majority of desktop + server Linux distributions, facilitating choice of function for our users.

It should be noted this decision has been made after much discussion internally, on our IRC, on our OpenCollective, etc. Our bootstrap-scripts is being improved to support both glibc and musl, so that the decision can continuously be reviewed. If we reach a position whereby musl inclusion once again makes sense, thanks to atomic updates from moss, it will be possible to switch.

Final thoughts

Initially Serpent OS emerged as a collective agreement on IRC as a set of notions as opinions. Over the past few months those opinions have solidified into tangible ideas, and a sense of community. In keeping with what is right for the community, our messaging has been reworked.

It is fair to say our initial stance appeared quite hostile, as a bullet-point list of exhaustion with past experiences. As we’ve pivoted to being a fully community-oriented distribution, we’ve established our goals of being a reliable, flexible, open, rolling Linux distribution with powerful features imbued by the package manager, and an upstream-first approach.

As such we’ve agreed to not let our own pet-peeves interfere with the direction of the project, and instead enable users to do what they wish on Serpent OS, be it devops, engineering, browsing, gaming, you name it. We’re a general purpose OS with resilience at the core.

Our focus is on the usability and reliability of the OS - thus our efforts will be invested in areas such as the package manager, hardware enabling, the default experience, etc.

So, strap yourself in, as we’re fond of saying. Development of Serpent OS is about to accelerate rapidly.

Source Format Defined

Sep 21, 2020

Ikey

Founder

Following quickly on the heels of yesterday’s announcement that the binary format has been defined, we’ve now implemented the initial version of our source format. The source format provides metadata on a package along with instructions on how to build the package.

The next step of course, is to implement the build tool, converting the source specification into a binary package that the end user can install. With our 2 formats defined, we can now go ahead and implement the build routines.

Very trivial package recipe

A YAML based format

The eagle-eyed among you will already see this is a derivation of the package.yml format I originally created while at Solus. Minor adaptations to the format have been made to support multiple architectures via the profiles key, and package splitting behaviour has now been grouped under a packages key to make the structure more readable.

In package.yml, one would have to redefine subpackage summaries as a key in a list of the primary summary key, such as:

    rundeps:
        - primary-run-dep
        - dev: secondary-run-dep
    summary:
        - Some Summary
        - dev: Some different summary

We’ve opted to group “Package Definition” behaviour into core structs, which are allowed to appear in the root-level package and subpackages:

    summary: Top-level summary
    packages:
        - dev:
            summary: Different summary
            rundeps: Different rundeps

Multiple architecture support

In keeping with the grouping behaviour, we’re baking multiple architecture configurations into the YML file. A common issue encountered with the older format was how to handle emul32:

    setup: |
        if [[ -z "${EMUL32BUILD}" ]]; then
            %configure --some-emul32-option
        else
            %configure
        fi

    build: |
    %make

Our new approach is to group Build Definitions into the root level struct, which may then individually be overridden for each architecture. For example:

    profiles:
        - ia32:
            setup: |
        %configure --some-emul32-option
    setup: |
    %configure

Permutations

Differences

As you can see it is highly similar to package.yml - which is a great format. However, with our tooling and aims being slightly different, it was time to reevaluate the spec and bolster it where appropriate. We’re happy to share our changes, but in the interest of not causing a conflict between the 2 variants, we’ll be calling ours “stone.yml”.

Our main motivation came from the tooling, which is written in the D language. With D we were able to create a strongly typed parser and explicit schema, and with a struct-based approach it made it more trivial to group similar definitions.

Other than that, we have the same notions with the format, intelligent automatic package splitting, ease of developer experience, etc.

Moss Format Defined

Sep 20, 2020

Ikey

Founder

The core team have been hard at work lately implementing the Moss package manager. We now have an initial version of the binary format that we’re happy with, so we thought we’d share a progress update with you.

Development work on moss

Explaining the format

Briefly, the binary container format consists of 4 payloads:

Meta (Information on the package)
Content (a binary blob containing all files)
Index (indices to files within the binary blob)
Layout (How to apply the files to disk)

Each payload is verified internally using a CRC64-ISO, and contains basic information such as the length of the payload both compressed and uncompressed, the compression algorithm used (zstd and zlib supported) as well as the type and version of the payload. All multiple-byte values are stored in Big Endian order (i.e. Network Byte Order).

Payloads

Internally the representation of a Payload is defined as a 32-byte struct:

    @autoEndian uint64_t length = 0; /* 8 bytes */
    @autoEndian uint64_t size = 0; /* 8 bytes */
    ubyte[8] crc64 = 0; /* CRC64-ISO */
    @autoEndian uint32_t numRecords = 0; /* 4 bytes */
    @autoEndian uint16_t payloadVersion = 0; /* 2 bytes  */
    PayloadType type = PayloadType.Unknown; /* 1 byte  */
    PayloadCompression compression = PayloadCompression.Unknown; /* 1 byte */

We merge all unique files in a package rootfs into the Content payload, and compress that using zstd. The offsets to each unique file (i.e. the sha256sum) are stored within the Index payload, allowing us to extract relevant portions from the “megablob” using copy_file_range().

These files will become part of the system hash store, allowing another level of deduplication between all system packages. Finally, we use the Layout payload to apply the layout of the package into a transactional rootfs. This will define paths, such as /usr/bin/nano, along with permissions, types, etc. All regular files will actually be created as hard links from the hash store, allowing deduplication and snapshots.

The Meta payload consists of a number of records, each with strongly defined types (such as String or Int64) along with the tag, i.e. Name or Summary. The entire format is binary to ensure greater resilience and a more compact representation. For example, each metadata key is only 8 bytes.

    @autoEndian uint32_t length; /** 4 bytes per record length*/
    @autoEndian RecordTag tag; /** 2 bytes for the tag */
    RecordType type; /** 1 byte for the type */
    ubyte[1] padding = 0;

TLDR that for me…

Binary format that is self deduplicating at several layers, permitting fast transactional operations.

Up Next

Before we work any more on the binary format, we now need to pivot to the source format. Our immediate goal is to now have moss actually build packages from source, with resulting .stone packages. Once this step is complete we can work on installation, upgrades, repositories, etc, and race to becoming a self hosting distribution.

Note, the format may still change before it goes into production, as we encounter more cases for optimisation or improvement.

Defining Moss

Sep 10, 2020

Ikey

Founder

Over the past few weeks, throughout the entire bootstrap process, we’ve been deliberating on what our package manager is going to look like. We now have a very solid idea on what that’ll be, so it’s time for a blog post to share with the community.

Initial moss prototype CLI

Old, but new

The team has been very clear in wanting a traditional package management solution, whereby repositories and packages compose the OS as a whole. However, we also want modern features, such as being stateless by default. A common theme is wanting to reduce the complex iterations of an OS into something that is sanely versioned, but also flexible, to ensure a consistent, tested experience with multiple end targets.

Additionally, the OS must be incredibly easy for contributors and team members to maintain, with intelligent tooling and simple, but powerful formats.

Atomic updates

One of the most recent software update trends of recent years is atomic updates. In essence this allows applying an update in a single operation, and importantly, reversing an update in a single operation, without impacting the running system.

This is typically achieved using a so-called A/B switch, which is what we will also do with moss. We won’t rely on any specific filesystem for this implementation, instead relying on a smart layout, pivot_root and a few other tricks.

Primarily we’ll update a single /usr symlink to point to the current OS image, with / being a read-only faux rootfs, populated with established mountpoints and symlinks. Mutation will be possible only via moss transactions, or in world-writable locations (/opt, /usr/local, /etc …)

Deduplication

The moss binary package format will be deduplicated in nature, containing hash-keyed blobs in an zstd compressed payload. Unique blobs will be stored in a global cache, and hard-linked into their final location (i.e. /moss/root/$number/usr/...) to deduplicate the installed system too. This allows multiple system snapshots with minimal overhead, and the ability to perform an offline rollback.

Implications

We’ll need to lock kernels to their relevant transactions (or repo versions) to prevent untested configurations. Additionally the boot menu will need to know about older versions of the OS that are installed, so they can be activated in the initrd at boot. This will require us doing some work with clr-boot-manager to achieve our goals.

Difference from existing solutions

We’re trying to minimise the iterations of the OS to what is available in a given version of the package repositories. Additionally we wish to avoid extensive “symlink farms” as we’re not a profile-oriented distribution. Instead we focus on deduplication, atomic updates and resolution performance.

Subscriptions

Keeping a system slim is often a very difficult thing to achieve, without extensive package splitting and effort on the user’s part. An example might be enabling SELinux support on a system, or culling the locales to only include the used ones.

In Serpent OS (and moss, more specifically) we intend to address this through “subscriptions”. Well defined names will be used by moss to filter (or enable) certain configurations in packages + dependencies. In some instances this will toggle subpackages/dependencies and in others it will control which paths of a package are actually installed to disk.

Going further with this concept, we will eventually introduce modalias based capabilities to automatically subscribe users to required kernel modules, allowing slim or fullfat installations as the user chooses. This in turn takes the burden of maintenance away from developers + users, and enables an incredibly flexible, approachable system.

Mandatory reboots

Where possible we will limit mandatory reboots, preferring an in-place atomic update to facilitate high uptime. However, there are situations where a reboot is absolutely unavoidable, and the system administrator should plan some downtime to handle this case.

Certain situations like a kernel update, or security fix to a core library, would require a reboot. In these instances, the atomic update will be deferred until the next boot. In most situations, however, reboots will not be mandatory.

Until the next time

Well, we’ve given a brief introduction to our aims with moss and associated tooling, and you can get more information by checking the moss README.md.

The takeaway is we want a package-based software update mechanism that is reliable and trusted, and custom-built to handle Serpent OS intricities, with a simple approach to building and maintaining the distribution.

For now, we’re gonna stop talking, and start coding.

Stage3 Complete

Sep 7, 2020

Ikey

Founder

Another week, another milestone completed. We’re pleased to announce that we’ve completed the initial Stage3 bootstrap. While we have some parallel works to complete for Stage 4, we can now begin to work on the exciting pieces! We’re now bootstrapped on ARMv8(-a) and x86_64

Build

Announcing Moss

Our immediate focus is now to implement our package manager: moss. Currently it only has a trivial CLI and does absolutely nothing. We will now shift our attention to implement this as a core part of the Stage 4 bootstrap. Moss is so-called as we’re a rolling release, a rolling stone gathers no moss. The package manager is being implemented in the D Programming Language.

Moss does not aim to be a next generation package management solution, it instead inspired by eopkg, RPM and swupd, aiming to provide a reliable modern package management and update solution with stateless design at the core. Core features and automatic dependencies will be managed through capability subscriptions. The binary format will be versioned and deduplicated, with multiple internal lookup tables and checksums. Our chief focus with moss is a reliable system management tool that is accessible even in situations where internet access is limited.

Ongoing Works

In order to complete stage3, we provided linker stubs in libwildebeest to ensure we can build. Obviously this introduces runtime errors in systemd and our stage3 isn’t bootable, just chrootable. This will be resolved when systemd is properly packaged by moss in stage4.

Docker Image

We now have a test container available on Docker Hub. You can install and run the simple bash environment by executing the following command with a Docker-enabled user:

    docker run -it --rm serpentos/staging:latest

Currently we only have an x86_64 image, but may experiment with multiarch builds later in stage4.

IMPORTANT: The staging image is currently only a dump of the stage3 tree with minor cleaning + tweakups. It is not, in any way, shape or form, representative of the final quality of Serpent OS. Additionally zero performance work or security patching has been done, so do not use in a production environment.

The image is provided currently as a means to validate the LLVM/musl toolchain.

Future ISO

We cannot currently say when we’ll definitely have an ISO, however we do know that some VM-specific images will arrive first. After that we’ll focus on an installer (package selection based) and a default developer experience. All we can say is strap in, and enjoy the ride.

Looking Stage4 In The Eye

Aug 28, 2020

Ikey

Founder

Well, we’ve made an awful lot of progress in these last few days. It wasn’t that long ago that we introduced some of the new projects required to get stage3 off the ground.

Building systemd the easy way

libc-support

Our libc-support project has been growing, thanks primarily to the contributions of Jouni Roivas. We now have initial working versions of getent and getconf. The getconf program is considered feature-complete for our current requirements, and the focus is now on cleaning up getent making it more modular and easy to maintain in the long run.

libwildebeest

We began work on libwildebeest to quickly unlock building systemd. Remember, our ambition with this project is to provide a sane, centralised way of maintaining source compatibility with various projects that rely on features currently only available in the GNU toolchain + runtime. This is our alternative vision to patching every single package that fails to build against our LLVM+musl toolchain, ensuring our work scales.

Right now, libwildebeest implements some missing APIs, and indeed, some replacement APIs, for when GNU behaviours are expected. It does so in a way that doesn’t impact the resulting binary’s license, or any of the system ABI. We provide some pkg-config files with explicit cflags and libs fields set, such as:

      -L${libdir} -lwildebeest-ftw -Wl,--wrap=ftw -Wl,--wrap=nftw -I${includedir}/libwildebeest --include=lwb_ftw.h

Our headers take special care to mask the headers provided by musl to avoid redefinitions, and instruct the linker to replace calls to these functions with our own versions, i.e.:

int __wrap_ftw(const char *dir, int (*funcc)(const char *, const struct stat *, int),
               int descriptors)

It then becomes trivial to enable wildebeest for a package build, i.e.:

        export CFLAGS="${CFLAGS} $(pkg-config --cflags --libs libwildebeest)"

Right now - we’ve only provided stubs in libwildebeest to ensure we can build our packages. Our next focus is to actually implement those stubs using MIT licensed code so that applications and libraries can rely on libwildebeest to provide a basic level of GNU compatibility in a reliable fashion.

Until such point as all the APIs are fully and safely implemented, it would be highly ill-advised to use libwildebeest in any project. We’ll announce stability in the coming weeks.

systemd

We’ve made great progress in enabling systemd in Serpent OS. Where libwildebeest is in place, it now enables our currently required level of source compatibility to a point where systemd is building with networkd, resolved and a number of other significant targets enabled.

In a small number of cases, we’ve had to patch systemd, but not in the traditional sense expected to make it work with musl.

The only non-upstreamable patching we’ve done (in combination with libwildebeest enabling) was to the UAPI headers, as the musl provided headers clash with the upstream kernel headers in certain places (if_ether.h, if_arp.h) - but this is a tiny cost to bear.

The other patches, were simply portability fixes, ensuring all headers were included:

It should be noted both of these (very trivial) pull requests were accepted and merged upstream, and will be part of the next systemd release.

Next On The Agenda

Our major ticket items involve fleshing out stage3 with some missing libraries to further enable systemd, rebuilds of util-linux to be systemd-aware, and continue fleshing out dbus, systemd and libwildebeest support to the point we have a bootable disk image.

At that point we’ll move into stage4 with package management, boot management, and a whole host of other goodies. And, if there is enough interest, perhaps some early access ISOs!

After having engaged in discussions with a variety of developers using musl as their primary libc, we’ve catalogued common pain points. We therefore encourage developers to contribute to our libwildebeest and libc-support projects to complete the tooling and experience around musl-based distributions.

Our aim for Serpent OS is a full fat experience, which means we have large ticket items on our horizon, such as NSS/IDN integration, performance improvements, increasing the default stack size, along with source compatibility for major upstream projects.

Until the next blog post, you can keep up to date on our IRC channel. Join #serpentOS on freenode!

Stage 3 Progress

Aug 20, 2020

Ikey

Founder

Well, it’s been a few days since we last spoke, so now it’s time for a quick roundup. Long story short, we’re approaching the end of the stage3 bootstrap.

Fully functional chroot

In an effort to simplify our bootstrap process, we dropped the newly-introduced stage2.5 and came up with a new strategy for stage3. In order to make it all work nicely, we bind-mount the stage2 resulting runtime at /serpent within the stage3 chroot environment, executing the /serpent/usr/bin/bash shell.

In order to make this work, we build an intermediate musl package for libc.so in the very start of stage3, with all subsequent builds being performed in chroots. Part of the build is done on the host, i.e. extraction and patching, minimising the tool requirements for the chroot environment. The configuration, build and install is performed from within the initially empty chroot environment, replacing all the /serpent/usr/bin tools and /serpent/usr/lib libraries.

Mild Blockers

As we move further through stage3, towards a fully usable chroot environment, we’ve encountered a small number of blockers. Now, we could solve them by using existing patchwork and workarounds, but most have not and will not be accepted upstream. Additionally it is incredibly hard to track the origin and history of most of these, making security rather more painful.

libc-support

We’re going to start working on a project to flesh out the musl runtime with some missing utilities, written with a clean-room approach. These will initially include the getconf and getent tools, which will be written only with Linux in mind, and no legacy/BSD support.

These will be maintained over at our GitHub

libwildebeest

As a project we strive for correctness in the most pragmatic way. Some software, such as systemd, is heavily reliant on GNU GCC/GLibc extensions. In some software there are feasible alternatives when using musl, however in a good number of cases, functionality required to build certain software is missing and has no alternative.

Over time we’ll try to work with upstreams to resolve those issues, but we’re working on an interim solution called ‘libwildebeest’. This will provide source compatibility for a limited number of software packages relying on so-called ‘GNUisms’. Binary compatibility is not an aim whatsoever, and will not be implemented. This convenience library will centralise all patchwork on packages that need more work to integrate with musl, until such time as upstreams have resolved the remaining issues.

Additionally it will help us track those packages needing attention in the distribution, as they will have a build-time dependency on libwildebeest. We do not intend to use this project extensively.

This will be maintained over at our GitHub

A Word On Init Systems

Recently we’ve had many queries regarding the init system, as there is an expectation that due to our use of musl/llvm we also dislike systemd or wish to be a small OS, etc. There is a place in the world for those projects already, and we wish them much success. However from our own stance and goals, systemd has already “won the battle” and actually fits in with our design.

If it is possible in future with package manager considerations and packaging design, then we may make it possible to swap systemd for a similar set of packages. However, we only intend at this time to support systemd/udev/dbus directly in Serpent OS and leave alternatives to the community.

Other News

Just a quick heads up, we’ve been talking to the cool folks over at fosshost.org and they’ve agreed to kindly provide us with additional hosting and mirroring. This will allow us to build scale in from the very start, ensuring updates and images are always available. Once the new server is up and running we’ll push another blogpost with the details and links.

While initially we intended to avoid public bug trackers, the rate of growth within the project and community have made it highly apparent that proper communication channels need establishing. Therefore we will be setting up a public Phabricator instance for reporting issues, security flaws, and contributing packaging.

Much of our website is in much need of update, but our current priority is with building the OS. Please be patient with us, we’ll have it all sorted out in no time.

Where We At?

Well, stage3 completes fully, builds the final compiler, which has also been verified. A usable chroot system is produced, built using musl, libc++, libunwind, clang, etc. Some might say that stage3 is complete, however we wish to avoid circular dependency situations. We’ll mark stage3 as complete once we’ve integrated an initial slimmed down build of systemd and appropriately relinked the build.

As soon as this stage is done, we’ll proceed with stage4. This is the final stage where we’ll add package management and define the OS itself, with global flags, policies, etc.

With the speed we’re moving at, that really isn’t too far away.

And finally..

I personally wish to thank the Serpent OS team as a whole for the commitment and work undertaken of late. Additionally I want to thank the growing community around Serpent OS, primarily residing in our IRC channel (#serpentOS on freenode) and our Twitter account. Your input has been amazing, and it’s so refreshing to have so many people on the same page. Stay awesome.

Stage2 Complete

Aug 16, 2020

Ikey

Founder

Just in case you thought we were sleeping behind the wheel, we’ve got another blogpost for your viewing pleasure. In a nutshell, we completed stage2 bootstrap.

Complete build-target for ARMv8

In order to simplify life, we greatly reduced the size of the stage2 build component. This decision was taken to better support cross-compilation in the face of software that is distinctly cross-compilation unfriendly.

A support stage, stage2.5 will be added which will chroot into a copy of stage2, and natively compile a small handful of packages required to complete stage3, also within the chroot environment.

For cross-compilation, we’ll be relying on qemu-static to complete 2.5 and 3. However, at this point in time, we have the following:

Working cross-compilation of the entire bootstrap
Complete LLVM based toolchain: clang, llvm, libc++, lib++abi, libunwind
Entirety of stage2 built with musl libc.
Working, minimal, chroot environment as product of stage2, with working compiler (C & C++)

x86_64

This is a major milestone for the project, as it is an early indication that we’re self hosting.

Multiple Architecture Support

At this point in time, we now have build support for two targets: x86_64 and ARMV8a. Our intent is to support haswell and above, or zen and above, for the x86_64 target.

With our ARMv8 target, we’re currently looking to support the Pine Book Pro, if we can manage to get hold of some testing hardware. It will likely be some time after full x86_64 support that we’d officially support more hardware, however it is very important that our bootstrap-scripts can trivially target multiple platforms.

ARMv8

An interesting change when cross-compiling for other architectures, is the chicken & egg situation with compiler-rt and other LLVM libraries. When we detect cross-compilation, we’ll automatically bootstrap compiler-rt before building musl, and then cross-compile libc++, libc++abi and libunwind to ensure stage1 can produce native binaries for the target with correct linkage.

Next Steps

As we’ve mentioned, we’ll push ahead with 2.5 and 3, which will complete the initial Serpent OS bootstrap, producing a self-hosting, self-reliant rootfs. This is the point at which we can begin to bolt-on package management, boot management, stateless configuration utilities, etc.

Our initial focus is x86_64 hardware with UEFI, and as we gain access to more hardware we can enable support for more targets, such as ARMv8a. Our bootstrap-scripts will always remain open source, as will all processes and tooling within Serpent OS, or anything used to build and deploy Serpent OS.

This will make it much easier in future to create custom spins of Serpent OS for different configurations or targets, without derailing the core project. It should therefore be the simplest thing in the world to fork Serpent OS to one’s liking or needs.

If you want to support our work, you can jump onto our IRC channel (#serpentOS on freenode) or support us via the Team page.

Stage1 Complete

Aug 8, 2020

Ikey

Founder

Short and sweet, stage1 of the bootstrap is complete. As I indicated on the Lispy Snake blog, I’m still in the process of settling into new accommodation. This is going well, but still awaiting proper broadband connectivity. Work has begun, however, and we’re now moving onto stage2 of the bootstrap.

This is handled via our bootstrap-scripts project and can be run by anyone on a relatively modern Linux distribution.

Validating stage1 cross-compiler

Next on the list is completing stage2, which we’ve already started on. This is simply a cross-compiled chroot environment with all the basic bits in place to build stage3, sanitizing and cleansing the toolchain.

Even though it is early days, we can already, automatically produce a working cross-compiler that targets LLVM’s libc++, the musl libc.so, and x86_64-serpent-linux-musl host triplet.

Probably a dull update for most, but an update it is.

Welcome: Project Manager

Jul 7, 2020

Ikey

Founder

The Serpent OS team wishes to formally welcome Aydemir Ulaş Şahin to the core team in the primary capacity of Project Manager. Aydemir will help us to coordinate and manage the project goals and infrastructure, being responsible for certain “keys to the kingdom”.

Aydemir has extensive experience with Linux, both as a user and developer, having worked in the media industry for many years. Additionally he will be contributing to the core system, code and packaging.

Our initial trio has been announced, and you can always access information on the current team at the new Team page. Now that much of our groundwork has been laid, we can get on with building the project. Check progress at the new Roadmap page.

Welcome Peter

Jul 6, 2020

Ikey

Founder

Recently we did reveal on Twitter that long time friend and colleague-of-many-projects, Peter O’Connor, has formally joined the core team for Serpent OS. Many of you know him as sunnyflunk. He will be working on core toolchain optimisations, benchmarking, and eventually Plasma inclusion.

Peter has long wanted to get the most out of his computing experience, starting from Gentoo back in the 2000’s to ensure there was no wasted resources on features he didn’t use. The desire to over-tinker and use unstable software often led to poor outcomes. While the itch laid dormant for over a decade, it has always been there.

Peter O’Connor (sunnyflunk)

There’s not much about Linux that currently excites me, but I’ve always wanted to see what’s possible with a full LLVM/Clang stack. There’s never been a better time to find out, but it can be difficult to push the envelope when limited with backwards compatibility and supporting older CPU architectures. After experimenting with some tool-chain customisations the chance to test them out more broadly and with a new tool-chain was too hard to pass up. This is the research project I’ve always wanted to undertake and looking forward to using it as my daily driver

We’d like to officially extend our welcome to Peter, as the ramp up gets in place. It should be noted, he is partly to blame for Serpent OS becoming manifest, as the core team got together after discussions on IRC. He has some fantastic ideas, and is a brilliant distribution engineer with proven success.

As an aside, we’re working on improving the website to incorporate a team page, ready for the third team member reveal.

The Great Experiment

Jul 1, 2020

Ikey

Founder

So as many have come to realise, we had to rush out a website super quick yesterday as the cat was already out of the bag. One thing that should also be clarified, is our approach to development.

We’re A Lab

Primarily (but not exclusively) Serpent OS (future: Serpent Linux) is an endeavour worked on by the Lispy Snake, Ltd crew. So we’re all about creating awesome technology and trying to further the playing field.

Serpent OS is not, however, owned by Lispy Snake. It is a contributor-based open source project, with specific roles. Those will be clarified over the coming week, but there is a dedicated Project Manager. Hint: It’s not me.

Measuring Success

So, how does an experimental project.. measure success? Quite simply - if others adopt our methodologies or technology, we’ve succeeded. We’re really not looking at “number of installs” or “size of userbase” as a metric, as our chief aim is to build technology.

Options for Longevity

If the experiment is a success, which of course means having tight controls on scope and timescale, then one would assume the primary way to use Serpent OS would be through some downstream repackaging or reconfiguration, i.e. basing upon the project, to meet specific user demand.

It Should Be Fun

This is the heart and soul of all Linux development. Developers should enjoy the involvement and seeing the end result. If we build something more in the end, then that’s a journey we can create together.

First Post

Jun 30, 2020

Ikey

Founder

In a nutshell, this post wasn’t meant to be coming out for a few weeks. As it turns out, we had to get ahead of the cat coming out of the bag, and ensure clarity of goals and ambition.

We’ve got an awful lot of work ahead of us across the next few months and weeks, and we strongly suggest checking out the About page which should answer most questions.

TL;DR This is not your typical user-facing Linux distribution.

So go ahead, check out the About page.

Posts will follow in the coming weeks, introducing the core team. It should be pointed out, internally, we’re in a stage1 bootstrap phase, with stage2 ready. Tooling is next on the agenda.

If you’re looking for a modern, lightweight, user-friendly, privacy-focused Linux desktop distribution, then you’re in the wrong place.

Otherwise, stay tuned for the updates. We know our website is insanely basic, but it will be improved when it becomes a priority. You’re more than welcome to join us on #serpentOS on freenode to become part of the community discussion in the mean time.

Til next time.

news

How we view the project and our roles in creating it

NVIDIA driver issues for certain GPUs

Why have monthly ISOs anyway?!

We actually delivered quite a lot in the last couple of days!

oxidize Window Manager theming

ISO refresh

Next Steps

Supporting the project

Thank You!

What’s new in the distro

Infrastructure and Tooling Updates

boulder cache subcommands with clean and size options

boulder recipe automation

Faster moss state remove

Improved moss search

Continuation of our Versioned Repository feature set

Wider Project Updates

New Branding: A Fresh Look for AerynOS

New Wallpapers: Bringing Nature to Your Desktop

Python stack upgrade

Kernel updates

ISO refresh

Next Steps

Supporting the project

Thank You!

AerynOS: March 2026 project update

What’s new in the distro

Desktop Updates

GNOME

KDE Plasma

Wayland Compositor Environments

Infrastructure and Tooling Updates

Use CDN for the default volatile stream

moss: Search for binaries

moss: Enable removal of ranges of states

boulder: Add support for control files

Wider Project Updates

Recipe repository REUSE compliance

Framework hardware sponsorship

ISO refresh

Next Steps

Python upgrade and repository rebuild

Versioned Repositories, phase2

Supporting the project

Thank You!

AerynOS: February 2026 project update

What’s new in the distro

Desktop Updates

Cosmic

Gnome

KDE Plasma

Wayland Compositor Environments

Infrastructure and Tooling Updates

Boulder and recipe updates

Summit queue visualization

Boulder --verify flag for manifest checks

Moss blitting speeds

Trigger speeds

Wider Project Updates

Website redesign

Documentation

Donation methods

ISO refresh

Next Steps

Package addition soft-freeze

Invasive toolchain and full-repo rebuilds

General recipes repository housekeeping

Versioned Repos, phase2 work

Supporting the project

Thank You!

AerynOS: January 2026 project update

What’s new in the distro

Desktop Updates

Cosmic

Cosmic context

Gnome

KDE Plasma

New Wayland Compositor Environments

Why not use our system-model?

Faster `moss state remove`

Improved `moss search`

Boulder `--verify` flag for manifest checks

Parallelize the `moss state verify` command

moss `system-model` prototype feature