Blog

We are firmly in the final quarter of 2025 and what a year it’s been so far. In our last blog post at the end of September, we mentioned that we were delaying the release of our next ISO into October to give our Gnome 49 stack (and the wider extensions ecosystem) more time to mature.

Coming into the final week of October, the team made the decision to transition from clang’s libc++ to GNU libstdc++ and work through the associated rebuild requirements across our repository.

We also mentioned in our September blog post that we had reached the point of having stable build infrastructure. However, over the course of October, we had an old problem re-appear, which proved somewhat vexing to solve initially.

After the issue was debugged, and a fixed version of boulder was deployed to the build infrastructure, we used the transition back to libstdc++, which included hundreds and hundreds of rebuilds, and the landing of the KDE Plasma 6.5 stack to verify that that we have put this particular issue behind us.

In addition to the build infrastructure related boulder fixes, the team has also found the time to hash out a design for, and prepare an early prototype PR of, the moss declarative system-model feature, as well as landing a few small user experience improvements and correctness fixes to both moss and boulder.

With that all said and done, we are ready with our 2025.10 GNOME Live ISO refresh for you to enjoy along with this blog post.

Whats new in the distro

Package / stack updates for this month include:

KDE Plasma 6.5.1
Cosmic Beta3
Gnome 49.1
Linux 6.16.12
Mesa 25.2.5
KDE Frameworks 6.19
KDE Gear 25.08.2
llvm 21.1.4
uutils-coreutils 0.3.0
sudo-rs 0.2.9
ffmpeg 8.0
pipewire 1.4.9
Wine 10.17
nodejs 22.21.0
zed 0.206.6
virt-manager 5.1.0
bash 5.3.3
scx-scheds 1.0.16
vim 9.1.1829
systemd 257.10
uv 0.9.5
perl-module-build: Add at v0.4234
libdovi: Add at v3.3.2
openconnect: Add a v9.12

… along with sundry additions and updates.

Desktop Updates

Cosmic

Our Cosmic environment has seen additional testers and contributors coming on-board to support Bryan who we highlighted in the previous blog post. Our Cosmic packaging team’s approach is to use System76’s repo-release repository rather than waiting for git tags. The team are running roughly on a bi-weekly update cycle keeping us up to date with Cosmic’s development where we had previously fallen behind. At the time of this blog post, our versions are tagged at Cosmic Beta3, however the code is somewhere between Cosmic Beta3 and Beta4, and Beta4 is being worked on as we write this.

Known issues

We are currently experiencing an issue with sudo-rs in Cosmic Terminal and the GNOME Ptyxis terminal emulator in our Cosmic session, however the issue doesn’t present with e.g. the Kitty terminal emulator, which can therefore be used as a workaround for the issue. We are tracking this issue in our bug tracker.

Additional details about Cosmic can be found on System76’s website.

Gnome

The Gnome packaging team has updated our Gnome environment to 49.1. The team has also expanded the number of available Gnome packages in our repository, and we now include Showtime and Gnome-contacts in our gnome pkgsets accordingly. For clarity, these were added in September but were not mentioned in our previous blog post.

Gnome continues to work very well on AerynOS and remains as our default live installation environment.

KDE Plasma

Reilly Brogan has done a fantastic job landing KDE Plasma 6.5, KDE Gear 25.08.2 and KDE Frameworks 6.19.0 into our repository over the last month. The overall KDE Plasma experience continues to improve with users providing largely very positive feedback on its performance and fluidity on AerynOS.

Over the last couple of months, KDE Plasma has now also been promoted as a recommended installation option alongside Gnome.

Distribution Updates

Switching back to GNU libstdc++ from LLVM libcxx

The team decided to switch back to the GNU libstdc++ library from the LLVM libcxx C++ library as a defensive measure. This has resulted in us having to carry fewer patches across the stack, and has resolved a few bugs as a bonus. In particular, this has squashed an annoying bug related to the Firefox Widevine DRM plugin that in turn previously made certain video conferencing tools crash.

Packaging License Matching

New this month, when packagers use the boulder recipe new invocation to create a new recipe from an upstream source archive, boulder will attempt to identify the applicable license of the package, and add it to the autogenerated skeleton stone.yaml recipe file.

This is a nice quality of life and user experience feature for our packagers. The AerynOS ethos is to try to help lessen the burden on packagers by automating things that are simple in nature, yet time-consuming. This in turn, we hope, will enable packagers to spend more of their time on the parts of the packaging process that genuinely benefit from a human touch in terms of building a distro that they and users enjoy.

Parallelize the `moss state verify` command

Until recently, when employing the moss state verify command to ensure the integrity of all moss states, boulder was limited to running on a single thread. The team updated the moss state verify command to run certain tasks in parallel (using rayon) which has significantly reduced the time to complete the verification process.

The moss state verify command currently does not have any indication of progress such as a progress bar. As such, a user may be unsure if their system is frozen. Whilst parallelizing performance-sensitive aspects of this task significantly reduces the time to completion, we have an issue raised to add a progress bar to the command for an improved user experience.

Infrastructure Stabilization

As it turned out, the issue we kept seeing (and have kept seeing since May) where build tasks would hang for no apparent reason, was actually related to the boulder threading code leaving threads running at the point where we called the clone2 syscall to enter a user-namespace container for build isolation purposes – fearless concurrency indeed…

However, as these issues only manifested for us when boulder was invoked by the build infrastructure, and even then only rarely, it was not a problem we could trigger on demand.

After some head-scratching and debugging sessions where we attached debuggers to live builds that had happened to hang, we finally found the root cause, and in turn were able to simplify our threading code significantly by always using a dedicated, explicit runtime that gets automatically shut down once the parallel (rayon) or threaded (tokio) work unit goes out of scope.

This in turn now ensures that all threads have been shut down as required, before entering the cloned user-namespace container.

Massive thanks to Cory Forsstrom for the work he put into solving this problem very elegantly over a couple of Pull Requests.

moss `system-model` prototype feature

The system-model feature is inspired by the feature of the same name from the Conary system and package manager. However, in practice, it will likely end up more like a hybrid of the Conary feature and e.g. the simpler and more limited Gentoo (and FreeBSD) ‘world’ set.

We are still hoping to be able to, when the time comes, offer versioned system-models that match our planned versioned repository work.

The goal of the moss system-model feature is that it will eventually enable us to define, and switch between, system installs declaratively and seamlessly. This feature is slated to be suitable for use on both live systems, recovery initramfs environments, and for installing new systems.

It is important to note that the system-model feature will be an opt-in feature, and that it will continue to be possible to use moss in an imperative fashion, where the system state is defined by the sum total of historical moss operations executed on the command line.

The design we have settled on makes it possible in the future to import (“resolve”) a declarative system-model to an imperatively defined system, just as it will be possible in the future to export (“derive”) a declarative system-model of any given imperatively created moss system state.

We hope that, as we evolve and flesh out the system-model feature work, this will give both users and system integrators the flexibility they need to define their systems in ways that matches their requirements and preferences.

Wider Project updates

Donations

During October, we have made changes to our donation accounts and updated our site accordingly. Most importantly, the primary way to now provide donations to AerynOS is via our Ko-fi page.

Due to the way Ko-Fi works, we have had to cancel all existing recurring donations as the money would not automatically flow to our new accounts. We have sent messages out to our donors and are hoping that they will want to sign-up again to provide continued donations towards the project.

All donations received by AerynOS are going towards paying our running costs, reimbursing ermo for his project hardware investments, and building a buffer for acquiring new hardware once the current hardware reaches its End of Life.

Taken together, these three budget items result in a target income of around €500 per month, with a third of that projected to go to each of the outlined items.

Due to the cancellations mentioned above, we are currently down to €25 of new/committed monthly recurring donations.

Rest assured however, that this will not impact the project’s long term viability as we have the ability as a team to cover these costs. However, if you are able to provide any sort of donation, it would be greatly appreciated.

Sponsor aerynOS

As a side note, we have also moved away from taking donations in USD by default and switched to EUR as this is the currency we are primarily spending in for our project costs. This does not preclude us from accepting USD based donations however.

Presenting the 2025.10 ISO refresh

Whilst we provide a bootable ISO image with a live GNOME environment, its main purpose is to serve as a vehicle for our ‘lichen’ installer which is able to install all of our ‘GNOME’, ‘KDE Plasma’, ‘Cosmic’ and ‘Terminal Only’ versions of AerynOS. Given that ‘lichen’ is a net installer, it will always pull the latest packages from our repository at the time of installation and as such, it requires an active internet connection. This means there is less of a need for regular ISO updates, as we have carefully designed our installation routine and our pkgsets to not be tightly coupled to the packages that happened to be available at the time of the live ISO creation.

Gnome Install

That said, with the addition of some significant updates in the Gnome stack, we felt it was a good opportunity to land a new live ISO for you all to try out. The link for our 2025.10 ISO can be found at our download page.

Next steps

Going into November, the team will continue to execute on our development plans related to improving our infrastructure in terms of both capability and user experience, just as we will be continuing to work on improving our moss and boulder tools via their planned workstreams, particularly the moss system-model related workstream.

On a final note, we are progressing on the documentation front, and work is steadily trickling into our documentation site. Thank you to everyone who has contributed!

If you are interested in our work or generally want to hang out, the best place to join us is in our Matrix¹ space. Alternatively, our Forums can be found on GitHub

aerynOS now uses a Zulip server for community discussion. ↩

September 2025 project update

Oct 1, 2025 - 5 min read - 899 words

As we reach the end of September, it’s time for another monthly update. For this month, we are deferring our technical blog post into October but wanted to provide wider project updates.

Kernels

AerynOS is currently running on 6.16.8. While we are aware that Linux 6.17 was released on 28th September 2025, the AerynOS team takes the view that for larger software stacks, the wise thing to do is to hold back for a few point releases after a major version update to ensure there are no unexpected issues.

Even though AerynOS is a rolling release distribution, we don’t aim to be on the bleeding edge just for the sake of being on the bleeding edge.

Gnome 49

The team updated to Gnome 49 within a couple of weeks following its release meaning it’s now available in our repository. Given the way Gnome updates and how extensions then follow suit, we are aware that many extensions have not yet been updated for compatibility with Gnome 49.

As such, we are holding back our next live installer ISO release until a larger number of extensions have had time to go through their own update cycles.

Infra stabilization

2025 has seen a massive lift in our infra code with the rewrite to Rust taking place in Q2, and its stabilization being pursued from the end of Q2 and up until now.

In this period, the team has been incrementally refining the code, and resolved bugs as they have arisen. This is work that average users will not have seen or been aware of as it only affects workflows behind the scenes. However, this has been extremely important work that has increasingly allowed us to become more confident in our new infra foundations.

We are now at the point where we are confident in calling our new infrastructure “stable”, with no bugs having presented themselves in several weeks and hundreds of packages built.

It cannot be overstated how much better the state of our infra is compared to the start of the year. In turn, this now allows us to focus our efforts in other areas of our technology stack to similarly work through bugs, implement features and generally refine our code base.

Cosmic maintainership

A couple of months ago we had a new contributor, Bryan Hyland, join us and begin working on the Cosmic stack. Bryan’s contributions have been invaluable to the team and we thank him for his engagement.

Unfortunately, due to personal reasons, he is having to take a temporary step back from the project. He has shared his methodology for providing updates to the Cosmic packages and we already have willing contributors from our community picking up the in-flight tasks.

We know he will be back once his circumstances improve, but in the meantime, if you are curious about supporting Cosmic on AerynOS, please do get in touch in our Matrix¹ packaging channel as it’s always good to have several people capable of maintaining each of our DE stacks.

Web development

One area in which we often receive feedback is our website. Given we are in an alpha state, it has not been our highest priority to improve it. We do have a couple of interested contributors wanting to work on our website to help improve the overall look and feel of the project, though.

Hence, if you are interested in website design and want to lend a hand, please feel free to join our Matrix¹ webdev channel as we will likely begin work on this area in parallel to our core development work.

Documentation

Similarly to our web development work, we are seeing increasing numbers of contributors willing and able to help with our documentation. Outside of our core tooling development, this is arguably our second most important workstream and another area in which we often receive feedback.

If you are interested in supporting our documentation drive, feel free to join us in our Matrix¹ documentation channel.

Holidays

NomadicCore has been leading our social media presence for the last few months. Someone (unwisely) gave him permission to go on holiday for the next three weeks, which means that our social media presence is likely to be less active than usual in the interim.

Please do not take this as any indication of the project slowing down. If anything, we have an increasing number of developers working on different areas of our tooling stack and are seeing our development and packaging activity picking up.

October plans

We hope to release out next ISO in October, once:

We have landed the Cosmic Beta packages and have had a few weeks to shake out any bugs
Gnome extensions have had more time to update for compatibility with Gnome 49
We are a few point releases into the Linux 6.17.x kernel cycle and there are no confirmed bugs we need to address

We are also aware of the planned release of KDE Plasma 6.5 on the 21st of October 2025. Similarly to Linux 6.17.x, we are not planning / trying to be first out of the gate with a large stack update. This will likely land at a later point after a few weeks / a point release or two.

That’s it for this monthly roundup. Thanks for reading, and we hope to see you around in our community.

aerynOS now uses a Zulip server for community discussion. ↩ ↩² ↩³

August 2025 project update

Aug 31, 2025 - 12 min read - 2376 words

Closing out August 2025, we are proud to announce our third release of the year. This release follows an intense development period where the team has reevaluated its priorities / timelines and refocused efforts on “delivering core Linux distribution tooling that will simplify our ability to scale out over time”.

We have documented some of our progress in our last two blog posts and spent the last two months further progressing towards these goals. We have implemented a basic version of virtual packages (Package Sets), continued our hardware (and VM) enablement efforts and have selectively been growing our repository where we feel it’s beneficial to our users.

What’s new Distro-wise

Whilst not an exhaustive list, some of the top line repository updates include:

GNOME 48.4
Plasma 6.4.4
Sway 1.11
Cosmic Alpha 7
Linux 6.15.11
Mesa 25.2.1
LLVM 20.1.8
uutils-coreutils 0.1.0
sudo-rs 0.2.8
ffmpeg 7.1.1
fastfetch 2.51.1 (adds AerynOS logo)
Waydroid: Add at 1.5.4
openvpn: Add at 2.6.14
protontricks: Add at 1.13.0
winetricks: Add at 20250102

In addition, we fixed a subtle issue with our PATH configuration that mostly affected our console logins. With this fix, we have made our login experience fully stateless. We have also enabled sulogin for a single-user root shell to diagnose and repair boot failures.

Package sets

AerynOS is transitioning to a package set model for core packages installed on a user’s system.

Package sets are a collection of packages that are related or used together for a specific purpose. In AerynOS, they are used for consolidating our base system packages and for each of our offered Desktop Environments / Window Managers.

Each Desktop Environment offered by AerynOS has an associated package set (usually “recommended”). Depending on the environment, we may optionally offer a “minimal” and/or “full” solution with less or more packages to better suit our users requirements.

The package set model we have implemented is a stepping stone technology, not the final solution we are looking to implement. It introduces the basic premise of virtual sets of packages and is a precursor to our “system-model” work that will allow for exact reproduction of a user’s installed system.

Our installation experience with lichen

To fully integrate the new package set model within the AerynOS project, we have adapted lichen to install Desktop Environments based on the associated package set. The TUI (Text User Interface) prompts guide a user to select which DE they want, and depending on the DE, the version we have curated for the install (recommended for the DE’s, minimal for sway) with moss determining what dependencies are required for a successful install based on that package set.

Lichen is a network installer meaning that it downloads the latest set of packages from the AerynOS repository for installation. Tweaks to our package sets don’t necessarily require a new ISO. Whilst the “live environment” may not be up to date, the user will get a fully up-to-date installation without requiring a post-install update step.

lichen usability updates

In its current state, lichen requires that a user pre-format their disk prior to attempting to install AerynOS. Given its nature of being a network installer, lichen also needs an active internet connection to complete an install. Before, if either of these prerequisites were not met, the installer would output a very unhelpful Rust code error. We have added the appropriate prompts to guide users to ensure they have an active internet connection and to remind them that they need to pre-format their disk.

Whilst we could fix the pre-format requirement, a conscious decision has been made to keep this “anti-usability” feature as a barrier to entry for “beginner Linux users”. This may seem very counter intuitive, however whilst we are in an alpha state, we need to be careful not to position ourselves as a “beginner Linux distro” that could attract many support requests. We need to focus our time on developing our tooling and infrastructure.

Do note that in time, we will fix this issue and become more beginner friendly!

Virtual Machine usage and hardware enablement

Significant work has taken place to enable virtual machine support, both with AerynOS as the host and as the guest.

For host support, we have packaged virt-manager into our repository. The team utilised VMs for testing package set configurations and other potential breaking changes over the last month. It has sped up our development pace as VMs are more disposable by their nature.

For guest support, we have enabled a significant amount of hardware in our kernel and specifically enabled HyperV (based on user requests). We are also actively seeking user feedback for other VM environments. If you try AerynOS in other VM solutions and have any problems, you can report those issues here.

By better supporting both VM host and guest scenarios, we hope to unblock potential contributors from exploring our distribution and tooling.

Please note that we still classify AerynOS as alpha status project. We do not recommend anyone install it on hardware required for “production environments”! Our key goal is to enable the hardware / software that developers and contributors may need to make the transition to and/or explore AerynOS.

Other updates

Reilly Brogan has added scx-scheds to our repository and set scx_flash as our default scheduler. Scx_flash is a scheduler that is focused on ensuring fairness among tasks and performance predictability. More details about it can be found on the sched-ext website.

For our use case, this is helpful as it allows an AerynOS system to still be responsive whilst heavy tasks such as building packages are happening in the background.

Whilst this has been implemented in our ISO (ie. for new installs), it will not retroactively apply for existing installs. If you want to transition to this, you will need to install scx-scheds.

What’s new DE-wise

There has been considerable effort around our DE provision this year, some of which is yet to materialise. We are happy to report that we are now offering KDE Plasma in our repository and it will be installable from our ISOs going forwards.

In addition, we have created a console only installation option for more advanced users and for our own testing purposes.

KDE Plasma

It’s fair to say that KDE Plasma has been one of the biggest requests we have received and we are happy to report that it is now available in our repository. Reilly Brogan has done a fantastic job packaging up the latest 6.4.4 version into our repository with both sddm and plasma-login-manager offered as login managers.

A running bug tracker can be found here to report any issues. Please do test it out and help us find and resolve any undocumented bugs that remain.

We are far enough in our bring up and testing process for KDE Plasma that we are comfortable offering it as an installation option in our ISO’s going forwards.

Cosmic

Within the AerynOS repository, Cosmic DE sits at the release tag of Alpha 7. Given the significant pace of development of Cosmic, we are looking to move to a more frequent update cycle to incorporate bug fixes and new feature releases tracking System76’s repo-release repository. Whilst still in flux, we are looking at a bi-weekly update frequency to balance maintainer burden with keeping the DE up to date. The first of these updates bring Cosmic DE packages to their most recent versions will land in the AerynOS repository in the coming days.

Following recent engagement efforts, we have been seeing new users and contributors checking out AerynOS and specifically our Cosmic spin. Through this additional testing, we are seeing more active engagement on keeping Cosmic updated and bugs squashed. Given its alpha status, we don’t expect a fully bug free user experience so please bear this in mind if choosing Cosmic. We are classifying our Cosmic spin as a “Technical Preview” given that both Cosmic and AerynOS are currently in alpha status.

Moving forward, we are looking to package up more of the available Cosmic applets and generally polish the Cosmic experience on AerynOS. If you have an interest in packaging and specifically in the Cosmic Desktop, feel free to get in touch as we could always use more support in testing and improving upon our DE experience.

Sway

We have had Sway within our repository since early last year but did not really highlight its inclusion. Sway has been updated to v1.11 and we have also included Waybar and a few other packages to make ricing Sway a nicer experience on AerynOS.

We debated including Sway as an installable option from our ISO, however we have made the decision to defer this for a future release. We have created an initial “minimal” package set for Sway which includes the bare minimum to get started on ricing it. However, it has not yet been validated to a level that we are comfortable shipping it to users, even in our alpha state.

It remains in our repository and will continue being worked on as we progress into the second half of the year. In time, we would also like to develop a couple of pre-configured Sway configs as additional package sets so users not already familiar with Sway can jump right in without having as much background experience.

Console-only

In addition to the other environments, we have created a very minimal package set that will boot into the Linux console without any Desktop Environment. Users can use this console-only option as a starting point to configure a system install exactly to their requirements with only the packages they wish to have included or as the basis for a new DE/WM to be included within AerynOS.

Given the way we have layered DE/WM package sets over our base package sets, a user is able to install any of the other DE/WM options on top of this console only solution. This has been very helpful for the AerynOS team in testing our different offerings.

Gnome

Other than updating to the latest 48.4 version, there is not else much to say on GNOME. To its credit, it has been working smoothly on AerynOS so there has not been any major work required.

It remains our default option for our ISO live environment and should only require updating to new versions as they release. If you do happen to discover an “undocumented feature”, please feel free to report it here.

What’s new tooling-wise

Moss state diff

Joey Riches has delivered a new command “moss state diff” which allows users to check the differences between two states. This is very useful when you want to revert back to an older state.

Each state is identified as a number and there was no way to understand what a state had. With this new command, you will be able to inspect the state and see if it’s the right state that you are looking for.

The command requires you provide two state numbers and it will return back the differences in package versions and new / removed packages between the two states you have specified.

Moss search-file

We have also landed another new moss command “moss search-file”. This works similar to “moss search” however it works at the file level and enables users to ask moss to which package any given installed file under /usr belongs.

What we are working on

Packagekit and appstream generator work

Joey Riches has picked up and continued his previous packagekit integration for moss to integrate into our various DE software centres (GNOME Software, KDE Discover and Cosmic Store).

Until now, users could only install AerynOS .stone packages through the terminal. This integration is a significant usability upgrade though we still recommend our users have familiarity of how to interact with moss via command line.

Alongside packagekit, we now also have appstream meta data hosted on our dotdev site. Work is on-going with both packagekit and appstream but the groundwork is completed. We can build from this point towards fully developed software centre integration.

Once this lands in our repository, we will take another step towards making AerynOS a more user friendly distribution.

Documentation improvements

Outside of the code development, there is a renewed focus on our documentation site. This is a continuing and incremental exercise with improvements coming across the board.

Over the last few months, we have improved the FAQ page, added more information on how to update packages on an AerynOS system and added additional information around the Desktop Environments we offer. We have also added specific background detail about how AerynOS is different to other distributions on our Philosophy page.

We will continue fleshing out our documentation in the coming months, with a specific focus on how to contribute, both to the project itself and how to create packages and submit them for inclusion in our repository.

Some of the feedback we have received is that documentation is fragmented and/or not yet created. This is a frustration we can remove through our documentation efforts and we have new contributors helping out in this aspect too.

We are always looking for more support so if you have any interest in getting involved with our documentation efforts, please feel free to reach out on Matrix¹ and specifically engage with NomadicCore.

Next steps

Our focus for the second half of the year remains similar to what we have detailed in our previous two blog posts.

We are working towards versioned repositories which will allow the team to deliver new features to our os-tooling (moss and boulder) in a seamless fashion. Versioned repositories are a prerequisite and gateway to future features that we will deliver in AerynOS hence its prioritisation.

For the os-tooling, we are adding structured logging for better insight and reporting, improving error handling and ensuring we deliver more helpful message output and looking towards adding JSON output for all of this for nicer parsing of “structured output” across process barriers. We continue to add low hanging fruit features

Download AerynOS

The link for our latest iso can be found at our download page.

aerynOS now uses a Zulip server for community discussion. ↩

Development update: os-tools

Jul 11, 2025 - 13 min read - 2601 words

In our recent mid-year blog post, we mentioned that it would be the first in a short series of posts providing updates on the various work streams we have been actively progressing during the last few months. Whilst that post focused primarily on our infrastructure, in this one, we will shift our focus towards the work we have been doing around our os-tools.

Short overview

To recap, our os-tools consist of Moss and Boulder. Whilst also originally written in DLang, initial ports of these were built in Rust during the latter half of 2023. Though we made the odd improvement here and there during 2024, through Q2 2025 we set out to review the code, develop an improvement plan and then put that into action.

The TL;DR is:

Moss: Existing PRs reviewed, refined and merged (including a PR enabling faster package installation via parallel blitting). The code received various bug fixes and refactors for correctness and maintainability.
Boulder: Similarly, existing PRs reviewed, refined and merged including a few smaller features and packaging macros being added. Since Boulder uses the moss crate, it now builds packages slightly quicker due to faster buildroot creation. The odd bugfix was also made.
User Experience: Improved error reporting features were added to both Moss and Boulder to improve the troubleshooting experience for users and packagers. More to be done on this front.

We will be continuing our os-tools work over the coming months with a specific focus on tidying up the code, improving documentation, and ensuring better error handling and status reporting throughout the codebase. This will come in particularly handy when we do the feature work to make us able to produce JSON output as the final part of the alpha2 os-tools milestone.

Overall, this work will help users when they come across unexpected errors and also be beneficial when onboarding new developers.

The JSON output feature, in contrast, is largely targeted at convenient machine parsing of structured output for automation and integration purposes, which we are banking on will come in handy for future development work currently in the planning stages.

Introducing NomadicCore

Before going any further, you may have noticed my name as one of the authors of our mid-year update blog post. If you hang around our Matrix¹ rooms, you will likely already know me but I thought it prudent to provide a formal introduction.

I first became aware of SerpentOS about three years ago but only joined the Matrix¹ chat rooms in September 2023. I’m not actually a developer or have any coding experience, however, I am interested in open source projects and Linux distributions that can help me get the most out of my hardware. I liked what I saw with SerpentOS and over the course of 2024, started getting involved, trying to help out where I could.

Earlier this year, I ended up formally joining the team, around the time of the AerynOS rebrand, taking more of a support/communications role and providing feedback from an “average user” perspective of what I think might be important.

My focus will mainly be around working on our documentation, writing blog posts and engaging on our various social media platforms and Matrix¹ rooms. I’m looking forward to getting stuck in and helping support a Linux distribution I want to use on my various devices.

A deeper dive into our os-tools work

Since their port to Rust, our os-tools have been working well enough. However, we are self-aware enough to know that our initial porting efforts left room for improvement, both in code quality and performance.

The following subsections outline some of the os-tools work we have been doing throughout Q2.

Refactor work

Both tarkah and new contributor, Jonas Platte, have been working on refactoring our existing codebase. To increase the available insight and diagnostic information, we have decided to align around the use of the tracing crate given that important parts of our code base are asynchronous, for which tracing is particularly suited.

For error handling, Jonas suggested that we move away from thiserror towards snafu. Whilst thiserror suited our requirements during our initial porting work, snafu offers some nice quality of life features and forces us to be more explicit about handling different types of errors, which we hope will yield better longer term maintainability. Moving over to snafu requires a little more upfront work to get high quality error output, but we believe that the reward will be worth it once the transition is completed across all of our code base.

Along with this refactor work, tarkah, Jonas and ermo are also improving the documentation within the codebase itself. With the infrastructure code having been ported to Rust, there is now greater scope to reuse and consolidate code between the various tooling crates.

Keeping our dependencies up to date

One aspect of managing our tooling is ensuring that our codebase remains up to date. Part of this effort is also to ensure that we are updating our code’s dependencies to their own respective latest versions to benefit from bug fixes and performance improvements. Whilst this is an on-going task, some of our dependencies had been allowed to get a little stale. Through multiple commits, Jonas has systematically been updating the dependencies in our os-tools repo.

Part of this upgrade work also involved being able to lock dependencies for Rust packages as a way to ensure robustness of the Moss and Boulder builds we use in production.

Moss: Parallel blitting

Long-time contributor Joey Riches identified a parallelization improvement in Moss’s blitting process which was merged after several months of local testing.

In our testing, the code showed significant speedups across all three of our supported file systems (XFS, ext4 and F2FS). The previous single-threaded blitting made using ext4 and F2FS particularly slow, to the point that we did not recommend users use either filesystem as the basis of an AerynOS install.

However, blitting speeds with the new parallel approach – particularly with a “cold” kernel VFS cache – have significantly improved. Whilst ext4 and F2FS are still not as performant as XFS for our use case, they are at least more serviceable as the basis of an AerynOS install than they used to be. By way of an example, I saw a ~2x blitting speed improvement on my Gen4 NVMe SSD using XFS with the new parallel blitting code.

It’s worth restating that, to our knowledge, the moss approach to atomic updates, is the only one of its kind (at least in the Linux space) where users do not have to rely on containerization or A/B system swaps to deliver package updates. Eliminating download speeds as a variable, Moss is capable of atomically installing/updating hundreds of packages on your system in a matter of seconds to tens of seconds on SSD drives, and the installed/upgraded applications are ready to use next time the application is opened. No reboots and no messing with container permissions necessary.

Given that boulder also needs to blit files when it creates buildroots, the code has also had a positive impact on reducing package build times. This will be more evident on larger package builds and will have a cumulative impact, the more package work you end up doing.

Moss: Sync available before installed packages

As we were testing the upgrade path from our old packages.aerynos.com/volatile repository to our new, CDN-backed cdn.aerynos.dev/unstable repository, we ran into some unexpected small niggles related to how packages are resolved.

While tarkah’s fix to Moss was relatively small in terms of code, it served to ensure that updates would install properly on the first go when syncing to the new repo.

Consequently, we synced this bug-fix to the Moss version in the old repository to ensure that users will be able to seamlessly upgrade to the new rolling unstable repository.

Moss: Add index output directory option

As we were preparing the process for syncing the packages in our volatile build-server repository to our new downstream rolling unstable repository on our public-facing server, we ran into an issue with the existing moss index code path.

Before this issue was fixed, the stone.index file would be unconditionally written next to the actual package .stone files. This was useful when indexing local repos, but not as useful when indexing actual stone pool/ directories and sub-directories.

In the end, this was another small feature with somewhat large consequences, in that this enabled us to do the actual manual indexing in a way that is identical to how our infrastructure organizes things when indexing.

This in turns made it possible to make sure that the new rolling unstable repo presents with the same URI “pattern” as our volatile build repo:

❯ moss lr
 - oldrepo = https://packages.aerynos.com/volatile/x86_64/stone.index [0]
 - unstable = https://cdn.aerynos.dev/unstable/x86_64/stone.index [5]
 - volatile = https://build.aerynos.dev/volatile/x86_64/stone.index [10]
 - local = file:///home/ermo/.cache/local_repo/x86_64/stone.index [100]

Boulder: Fix up phase timing in end-of-build report

When Boulder successfully completes a package build, it emits a report detailing how long each phase of the build process took.

ermo noticed that the output was wrong when the time exceeded an hour. For example 136m would be formatted as a rather silly-looking 1h76m instead of 2h16m. He fixed this in the following commit.

Boulder: Improve cache hit rates when updating packages

Boulder is designed to cache files and hash them as part of its build process. By hashing files, it uniquely identifies each file and stores this for later utilization. When a package is updated from one version to the next, where the package has files that have not changed, we have the opportunity to reuse the cache from a prior build (as long as the cache has not been purged for space saving).

Given the way boulder previously cached files, in named directories based on the package source names, there was a high likelihood that new caches would have to be built because the source file names contain version numbers, which by design will always change.

Reilly implemented a change to boulder so that our ccache entries will persist over version number updates improving the hit rate and therefore performance on boulder package builds.

Boulder: Tweak how we use sh-compatible shells

For user-facing recipes, our recipe snippets are now always interpreted by bash.

However, testing with hyperfine has previously shown that dash is ~20% faster to start up during ./configure runs when compared to bash.

To reap the benefits of faster dash process startup times, ermo and Reilly implemented a change to our GNU autotools macros to use dash as the default shell.

Having said that, there are certain packages that just expect and/or are hardcoded to require bash. So to cater to that use case, we have also added autotools macros that packagers can use to make Boulder execute GNU autotools with bash on a per-package basis.

This gives our tooling the “dash as /bin/sh” ./configure speed improvements by default, yet allows packagers to still successfully invoke ./configure et al. with bash, where doing so is necessary for the build to complete.

Boulder: Update build macros

On reviewing Boulder’s build macros, we found some low hanging fruit improvements to make to our cmake, ninja, and meson macros, which we landed back in April. Implementing and improving the various build macros available in AerynOS makes it easier and more convenient for packagers to package up applications with Boulder; either for their own personal use or for submission into the official repositories.

Boulder: Use bsdtar-static for unpacking

At Reilly’s initiative, we moved our decompression solution away from GNU tar to bsdtar-static. This change reduces the likelihood of compatibility issues and ensures Boulder package creation doesn’t rely on a dynamically linked system library, thus making it more of a reliable solution.

With this move, we have also added the ability to decompress tgz based source packages as part of the boulder build process.

Other notable work

Some of the work we have done has been aimed more at how we use our os-tools rather than the os-tools themselves.

Update build triple and fix up ARM AArch targets

With our recent transition from SerpentOS to AerynOS, we needed to update our build triple accordingly. This step has been completed in the background whilst allowing for seamless updates from older SerpentOS systems onto AerynOS based systems. This is part of a wider rebranding effort that is still on-going through our documentation site, repo READMEs and anywhere else we have an official presence.

In this same area, whilst AerynOS currently only supports x86_64 based devices, there is a desire to be able to target other system types longer term. One of our contributors has been experimenting with RISC-V so we have added preliminary support for this to aid their testing. Don’t expect to see AerynOS on RISC-V any time soon, but it’s great to see our distro becoming a sandbox for fun and experimenting on alternative systems.

Emul32 ELF Machine type migration

During the port of boulder from DLang over to Rust, there was a change in how we expressed the ISA for packages built for the x86 emul32 architecture target. The old DLang version of Boulder expressed it as x86 where the Rust elf crate expresses it as 386 (EM_386 for those of you familiar with ELF parsing internals).

Reilly took point in implementing a series of changes that retained full compatibility between packages originally built on the old DLang infrastructure, and newer packages built on the Rust infrastructure.

The end goal was to flush out the packages containing references to the x86 Emul32 ISA through the recent rebuild of our whole recipes repository. This was accomplished by first ensuring that all packages exposed both x86 and 386 provider patterns, and then subsequently dropping the code that wrote the x86 provider patterns during the second full repo rebuild, ensuring that packages only contained the 386 provider patterns.

In the end, this worked out nicely for us.

Current and near-future os-tools focus

We reviewed the open issues in our various GitHub repositories and the plethora of ideas we have for our tooling, and developed a high level set of milestones for our os-tools. For this milestone (os-tools alpha2), we want to focus on:

Adding structured logging for better insight and reporting
Improving error handling
Ensuring we deliver more helpful message output
Adding JSON output for the above, for nicer parsing of structured output across process barriers
Adding low hanging fruit features and fixing misfeatures as we review the code

As our readers may have surmised, we have slowly accumulated bits of technical debt here and there over the course of development. The os-tools alpha2 milestone is our chance to address that and make our code crisp, clean and ready for already-planned future development work, while we stabilize our new infra code in parallel to this effort.

A special thank you goes to Jonas for the work he has already done in terms of this sort of refactor work.

How to join in the fun

As already covered in this blog, we have been reviewing all open issues and PRs in our os-tools repository on GitHub for prioritization and to set internal milestones.

We are open to and actively looking for contributors who might be interested in looking through our code and providing feedback.

If you would like to try your hand at contributing, look out for issues marked “good first issue” and get in touch on Matrix¹.

We hope to see you there!

aerynOS now uses a Zulip server for community discussion. ↩ ↩² ↩³ ↩⁴

Mid Year Update

Jun 30, 2025 - 11 min read - 2164 words