Skip to content

news

71 posts with the tag “news”

More choice and developing tooling

It’s been a couple of months since our last update and there has been quite a lot going on. With the project having spent the last 18 months shoring up its foundations, we are now able to start shifting towards more open and collaborative development.

We have landed our new moss command-line structure, which is a pretty significant departure from old CLI organization. The commands for adding and removing both repositories and packages from the internal system-model have changed so early adopters will need to adjust their mental model of how moss operates. It’s a “breaking change” we are making now whilst aerynOS is still in development and better aligns the command convention to how moss is intended to work.

Budgie Desktop is now offered as an option for aerynOS early adopters, including as an install time option within our installer(s). Whilst we are still heavily in development, we are now at a point where we can offer multiple different Desktop Environments and a dozen Wayland Compositor options for those that want to “Build You Own Desktop”.

We have a new 2026.11 ISO being released this month along side this blog post. It includes our existing installer but also a secondary option lichen-installer. Whilst it’s not yet ready to be our default and only installer, is being added to the ISO for wider testing purposes. This newer installer takes care of drive partitioning and has a number of under the hood enhancements that will make aerynOS installation more robust. It is still Work in Progress and does not yet support disk encryption, dual booting or other features that might indicate aerynOS is daily driver ready. It does however pave the way for those features to be added in the future.

A smaller but important request was for fwupd, which we have also now landed in our repository to ease hardware firmware updates for early adopters using aerynOS.

Lastly, we have also been doing housekeeping on our web presence. We have consolidated our websites to a single primary aerynos.com site and updating the content to better reflect our current project status. This is not the full website rebrand work, that is still happening in the background. However, we felt the existing website needed to be updated to better reflect the current position of aerynOS in the meantime.

A community that works together, builds together

Whilst August is a holiday month around Europe, it’s safe to say it’s been a very busy month around the project, with a lot of progress made across various core tooling repositories.

Following on from our Versioned Repositories phase 2 work that we landed last month, we are developing moss on multiple fronts. tarkah has supplemented our current hardlink driver approach with a reflink strategy for filesystems that support it, along with working on an in-progress development sprint to eventually deliver an EROFS metadata-image based approach that will be fully filesystem agnostic. Fabio is working on moss’ command tree, reorganising the commands we already have but also looking forward to the command functionality we will eventually want to have available. Lastly, Jonathan is also working towards improved search and shell completion functionality that will make moss easier to interact with for both users and packagers alike.

Joey has delivered a number of performance-related improvements to boulder, making packaging quicker and also offering additional optimisation techniques such as BOLT for key packages.

On the distribution side, Reilly has led our expanding packaging team through a repository-wide rebuild to ensure ABI sanity and protect against potential bit rot. Ultimately, this means less risk of broken packages or systems for our early adopters.

In a separate work stream, staff member Bryan is working on a new TUI-based lichen installer that is substantially improved from the current lichen installer that we offer. We are developing it openly with updated versions being shared in our Zulip server. Bryan is taking on board feedback and iterating on the design and functionality with the aim of it being included in a future ISO. Key improvements include automatic disk formatting, system-model-driven installs, btrfs as a root partition option and generally a much more refined TUI navigation experience.

Staff members Alice and NomadicCore have worked on a new Discourse forum that will replace our GitHub Discussions forum. We are thankful to Discourse for sponsoring the project with this server, and to our community members who have been giving us feedback on the layout. It is now ready for wider use by our community.

Finally, we’re continuing to refine the project’s branding with a slight tweak to the project name from AerynOS to aerynOS along with tweaks to logo colours and fonts.

Solid foundations are leading to expanding horizons

It’s been several months since our last update in May, and a lot has happened around the AerynOS project in the meantime!

First and foremost, we have tested, landed and productised our Versioned Repositories phase 2 feature, which will serve to firm up AerynOS’ foundations and help us further develop our vision of how to build a Linux distribution.

On the distribution side, Reilly has landed experimental OpenZFS support for data drives alongside significant systemd packaging efforts bringing it from 257.13 to 261.2.

Community member Christian Bendiksen is continuing his work around his Gnist and Malm cli tools for conveniently declaring, applying, and sharing Wayland Window Manager themes, and community member lumi has worked on their aerbox tool, which natively makes use of moss’ system-model concept to quickly create containers gated by access control to network, sound, drives and other hardware.

On the kernel front, we have moved to 7.1.5 for our stable and gaming branches and 6.18.40 for LTS, following a period of elevated vulnerability disclosures in the Linux kernel that required rapid response from the team.

We have also landed NVIDIA driver 610.43.03 in our repositories and have been running a testing programme with community members over the last month that is already yielding positive results.

In our wider project activities, we continue to explore Codeberg as an alternative to GitHub, with our new website built with Hugo using the Hextra theme under active development over there. We are also releasing a new ISO, AerynOS 2026.08.

In terms of our code bases, we have contributors experimenting with improving the moss package search UX, and refactoring some of our shared low-level git and sqlite3 db functionality.

From a development story arc perspective, AerynOS has been deliberately kept under a very tight leash over the last year, as we have focused on strengthening our core tooling and infrastructure code bases. Whilst we are nowhere near “done” with our vision for our tooling, now that our ability to seamlessly deliver updates to our on-disk repository and file formats has been secured, you will see AerynOS deliberately transitioning into a more open and collaborative space for those of you who want to get involved with the project and work on areas that you find fun, interesting and engaging.

This shift in approach has already been evident over the last couple of months, as engagement around the project continues to grow through all of our social media platforms and particularly in our Zulip server.

A post we never want to have to make!

This is the post we never want to make, where we made errors with our previous ISO release and are having to take steps to fix the issue, including releasing an updated ISO.

Over the course of April, the team worked on systemd-presets as a workstream we didn’t actually talk about in our April blog post. We thought that all the work had been completed in such a fashion that users wouldn’t see any discernible difference whilst allowing us to move closer to “best-practice” approaches.

Unfortunately, due to the way systemd functions, it doesn’t natively support a global preset-all approach on first boot, one of the preset approaches we had recently adopted. In combination, our lichen installer was creating an /etc/machine-id file in new installs, which is what systemd uses to detect whether a machine is in its first boot or not.

As such, users on existing systems did not have an issue, but new installs could be broken and we did not detect this as part of our ISO testing process.

Thanks to user feedback, we were able to narrow down, identify and resolve the issue within 24 hours of the ISO release, and we had an updated ISO up for testing within our server by the end of the first day of the original release. The fix came via two commits:

  1. Lichen: Fix first boot
  2. Systemd: Preset-all user services on first-boot

Following additional testing with a wider audience on our Zulip server, we have updated the links on our download page to our new ISO so any users downloading the latest ISO won’t see these issues.

Rebranding, Upgrading, and Wallpapering: AerynOS’ April glow-up!

No, this isn’t a suspicious phishing attempt! AerynOS has officially gone through a rebrand 🎉

Over the course of April, we rolled out a new logo and refreshed colour palette across the entire project. It took a bit of time to thread this through every corner of the OS and our web presence, but we’re happy to say it’s now fully in place.

Alongside the new branding, we’ve also been collaborating with community member ziegenmelker5, licensing a selection of his photography for use as wallpapers (and the title image above). A handful of these have already landed in our artwork repository and will show up on user systems after the next sync. On top of that, the team has created an abstract wallpaper inspired by the new logo, so there’s a bit of something for everyone.

On the software development side, a big chunk of April was spent improving our core tooling, especially moss and boulder:

Boulder:

  • Added boulder cache size to show cache size of both boulder and moss.
  • Added boulder cache clean to free up space on system by deleting the cache.
  • Updated boulder recipe update to use ent to check for recipe updates and appropriately update the stone.yaml accordingly.

Moss:

  • moss state prune is now faster and shows a progress bar when removing states.

In addition, we’ve continued our reuse compliance work, extending it from our recipes repository into our os-tools repository. This brings us closer to full compliance across the board.

Lastly, we have expanded our kernel configurations and now offer an LTS, stable and gaming kernel, though switching away from the stable kernel isn’t yet a simple process.

March 2026 project update

Another month brings another project update. In some respects, March has felt somewhat quieter than usual for the project. However, in the background, things have been fairly busy on the development front, as we prepare for larger and more visible tooling updates to land in the months to come.

On the packaging front, notable package & stack updates include GNOME 50, KDE Plasma 6.6.3, LLVM 22.1.1, Wayland 1.25.0, QT6.11.0, FFmpeg 8.1 and mesa 26.0.3 which have kept our builders busy with a significant number of package builds and rebuilds. The team mentioned last month that there is a soft-freeze in place for our repository, in terms of accepting new package recipes. This is not a full freeze as packages with little or no reverse dependency rebuild chains are still being accepted where we believe they will prove useful.

The development work this month has centered around the features we offer in terms of automating recipe updates and how we can better support bootstrap builds on the boulder side, along with the usual clean-up and refactoring work across the moss and boulder code bases.

Lastly, we want to take a moment to thank Framework for their on-going hardware sponsorship of our project. This month, they have provided the project with an AMD AI 300 based Framework 16 that Joey Riches is now using as a dedicated device for AerynOS development.

February 2026 project update

February has been a busy month for the project with a lot of activity around our tooling and infrastructure. We have merged a number of smaller improvements, which have both led to an increase in useful features, correctness and maintainability. Moss has become significantly quicker in usage, boulder has seen improvements to help automate recipe creation & updates to a set standard, and our summit dashboard has seen improvements to better represent build queues live and dynamically in graphical form.

The website re-design is progressing nicely. Last month we shared that we are using the Hextra theme. Over the last month, we have coordinated with the Hextra developer around conformance to WCAG 2.2 Level AA (Web Content Accessibility Guidelines) and have been really impressed with their rate of development in this area.

Documentation has seen continued focus. We have reworked our FAQ section, the installation section has been re-written to make it clearer, and we have conducted a site-wide documentation spelling review to conform to American English.

We have also seen packagers becoming more active in helping flesh out the recipes repository and helping keep existing packages up-to-date.

During the last week of February, we have shared with our packaging community that we will likely be more focused on maintaining our existing set of packages and becoming more selective on adding new packages in the coming months, whilst our tooling capabilities mature to where they need to be for efficiently managing long term growth.

January 2026 project update

Kicking off 2026, activity across the AerynOS project has been moving at pace. As covered in our 2025 retrospective, last year the team laid the groundwork for the project’s future growth and development. Starting off 2026, the team is building upon this foundation to keep delivering progress across the project.

From a core tooling perspective, we are preparing for the next phase of our Versioned Repository feature set. During January, we have been refining our existing codebase to make future improvements easier to implement. Once delivered, moss will be able to update itself seamlessly through what would otherwise have been breaking changes to its codebase. This functionality will be very important for our long term goals as a rolling release distro where you never have to reinstall to receive the latest updates.

Progress has been made around our branding with efforts towards a new website design and a healthy discussion and proposals for a new AerynOS logo.

You may recently have seen that AerynOS is taking a stronger stance against the use of LLMs and generative AI. Our LLM policy can be found on GitHub and will be replicated on our website in due course.

Additional progress has been made project-wide on our documentation and we have also made improvements to our CDN setup for improved package delivery.

Our efforts seem to be received well, as we are seeing an influx of new members joining our Zulip server and getting involved with AerynOS, whether through general discussion or through packaging efforts. We have also seen a jump in our followers across our various social media accounts and generally a more positive reception of what AerynOS is trying to achieve in the Linux space.

2025 in retrospect

2025 has been a year of significant change for the AerynOS project, not just in terms of development itself but also in name and in the staff working on the project.

We know many people will be following along, waiting for our beta and/or stable releases but for now we want to take a look back at 2025 and summarize what we have delivered and how that is positioning us strongly for 2026.

The TL;DR summary:

  • We changed our name from Serpent OS to AerynOS. Aeryn is a different form of Erin, which means Ireland in Irish Gaelic.
  • The project founder, Ikey, stepped away from the project in mid April. A few weeks later, he sent an invitation to project co-founder and co-architect, Rune Morling (ermo), to be his Github account successor.
  • After these events, ermo asked the team if they would be interested in carrying on working on the project under his stewardship while waiting for Ikey to potentially make contact, and the team voted unanimously in favour of doing so.
  • Since then, the team has continued to deliver per the project roadmap:
    • We completed the transition of all core tooling to Rust.
      • The new Rust-based infra has now delivered thousands of package builds while proving to be much more stable and robust than the previous implementation.
      • Leveraging Rust has enabled us to confidently deliver new tooling solutions like versioned repositories and a system-model approach to declaratively describe system installs.
    • We improved upon the Cosmic DE offering whilst delivering new KDE Plasma and Console-based install options
    • We onboarded several new staff members onto the project over the course of the year with differing strengths and interests
    • We reworked our cloud hosting setup to deliver more build capabilities at lower cost base
    • We renewed our media strategy through increased blog posts and engagement across social media platforms

November + early December 2025 project update

As we near the end of the year, the team has been reflecting on how best to position itself going into 2026. Over the course of November and early December, the team has made a number of changes, some of which have been public facing which you may have already seen if you’ve been following our socials.

On the public side, the two most visible changes has been our transition away from Matrix to Zulip for our instant messaging chat platform, and our transition to netcup for our server requirements. In the background, the team has also changed email hosting providers, however no one would really have noticed any difference there.

Packaging updates are progressing nicely with a nice rhythm for Cosmic and KDE stack updates, which along with fixes to reported issues is all helping to improve the overall experience for the people testing and checking out AerynOS.

On the infrastructure side of things, we have landed simple auto-pruning logic into our Vessel repository manager. This sort of automation ensures that we keep our repository lean and mean.

Things are progressing nicely in terms of getting phase 1 of our Versioned Repositories infra feature ready for production use. It was designed to from the outset to mesh nicely with our Moss system-model feature. Internal testing on a private infra instance has been running smoothly with these new features so far.

October 2025 project update

We are firmly in the final quarter of 2025 and what a year it’s been so far. In our last blog post at the end of September, we mentioned that we were delaying the release of our next ISO into October to give our Gnome 49 stack (and the wider extensions ecosystem) more time to mature.

Coming into the final week of October, the team made the decision to transition from clang’s libc++ to GNU libstdc++ and work through the associated rebuild requirements across our repository.

We also mentioned in our September blog post that we had reached the point of having stable build infrastructure. However, over the course of October, we had an old problem re-appear, which proved somewhat vexing to solve initially.

After the issue was debugged, and a fixed version of boulder was deployed to the build infrastructure, we used the transition back to libstdc++, which included hundreds and hundreds of rebuilds, and the landing of the KDE Plasma 6.5 stack to verify that that we have put this particular issue behind us.

In addition to the build infrastructure related boulder fixes, the team has also found the time to hash out a design for, and prepare an early prototype PR of, the moss declarative system-model feature, as well as landing a few small user experience improvements and correctness fixes to both moss and boulder.

With that all said and done, we are ready with our 2025.10 GNOME Live ISO refresh for you to enjoy along with this blog post.

September 2025 project update

As we reach the end of September, it’s time for another monthly update. For this month, we are deferring our technical blog post into October but wanted to provide wider project updates.

AerynOS is currently running on 6.16.8. While we are aware that Linux 6.17 was released on 28th September 2025, the AerynOS team takes the view that for larger software stacks, the wise thing to do is to hold back for a few point releases after a major version update to ensure there are no unexpected issues.

Even though AerynOS is a rolling release distribution, we don’t aim to be on the bleeding edge just for the sake of being on the bleeding edge.

The team updated to Gnome 49 within a couple of weeks following its release meaning it’s now available in our repository. Given the way Gnome updates and how extensions then follow suit, we are aware that many extensions have not yet been updated for compatibility with Gnome 49.

As such, we are holding back our next live installer ISO release until a larger number of extensions have had time to go through their own update cycles.

2025 has seen a massive lift in our infra code with the rewrite to Rust taking place in Q2, and its stabilization being pursued from the end of Q2 and up until now.

In this period, the team has been incrementally refining the code, and resolved bugs as they have arisen. This is work that average users will not have seen or been aware of as it only affects workflows behind the scenes. However, this has been extremely important work that has increasingly allowed us to become more confident in our new infra foundations.

We are now at the point where we are confident in calling our new infrastructure “stable”, with no bugs having presented themselves in several weeks and hundreds of packages built.

It cannot be overstated how much better the state of our infra is compared to the start of the year. In turn, this now allows us to focus our efforts in other areas of our technology stack to similarly work through bugs, implement features and generally refine our code base.

August 2025 project update

Closing out August 2025, we are proud to announce our third release of the year. This release follows an intense development period where the team has reevaluated its priorities / timelines and refocused efforts on “delivering core Linux distribution tooling that will simplify our ability to scale out over time”.

We have documented some of our progress in our last two blog posts and spent the last two months further progressing towards these goals. We have implemented a basic version of virtual packages (Package Sets), continued our hardware (and VM) enablement efforts and have selectively been growing our repository where we feel it’s beneficial to our users.

Whilst not an exhaustive list, some of the top line repository updates include:

  • GNOME 48.4
  • Plasma 6.4.4
  • Sway 1.11
  • Cosmic Alpha 7
  • Linux 6.15.11
  • Mesa 25.2.1
  • LLVM 20.1.8
  • uutils-coreutils 0.1.0
  • sudo-rs 0.2.8
  • ffmpeg 7.1.1
  • fastfetch 2.51.1 (adds AerynOS logo)
  • Waydroid: Add at 1.5.4
  • openvpn: Add at 2.6.14
  • protontricks: Add at 1.13.0
  • winetricks: Add at 20250102

In addition, we fixed a subtle issue with our PATH configuration that mostly affected our console logins. With this fix, we have made our login experience fully stateless. We have also enabled sulogin for a single-user root shell to diagnose and repair boot failures.

Development update: os-tools

In our recent mid-year blog post, we mentioned that it would be the first in a short series of posts providing updates on the various work streams we have been actively progressing during the last few months. Whilst that post focused primarily on our infrastructure, in this one, we will shift our focus towards the work we have been doing around our os-tools.

To recap, our os-tools consist of moss and boulder. Whilst also originally written in DLang, initial ports of these were built in Rust during the latter half of 2023. Though we made the odd improvement here and there during 2024, through Q2 2025 we set out to review the code, develop an improvement plan and then put that into action.

The TL;DR is:

  • Moss: Existing PRs reviewed, refined and merged (including a PR enabling faster package installation via parallel blitting). The code received various bug fixes and refactors for correctness and maintainability.
  • Boulder: Similarly, existing PRs reviewed, refined and merged including a few smaller features and packaging macros being added. Since Boulder uses the moss crate, it now builds packages slightly quicker due to faster buildroot creation. The odd bugfix was also made.
  • User Experience: Improved error reporting features were added to both Moss and Boulder to improve the troubleshooting experience for users and packagers. More to be done on this front.

We will be continuing our os-tools work over the coming months with a specific focus on tidying up the code, improving documentation, and ensuring better error handling and status reporting throughout the codebase. This will come in particularly handy when we do the feature work to make us able to produce JSON output as the final part of the alpha2 os-tools milestone.

Overall, this work will help users when they come across unexpected errors and also be beneficial when onboarding new developers.

The JSON output feature, in contrast, is largely targeted at convenient machine parsing of structured output for automation and integration purposes, which we are banking on will come in handy for future development work currently in the planning stages.

Mid Year Update

As we hit the middle of the year, it’s time for another update for those of you following along with AerynOS’s development.

Over the last few months, things may have seemed unusually quiet, however rest assured that there has been A LOT going on in the background. As such, we are preparing a short series of blog posts to go over the relevant topics in the coming weeks.

For this blog post, we are going to cover our infrastructure port, along with the process of rebuilding our entire package repository.

The TL;DR is that:

  • All core AerynOS tooling is now written in Rust
  • Every recipe in the repository has been rebuilt (twice!) with many packages then having been updated to newer versions after the rebuilds were completed
  • A CDN has been implemented for faster package installation and ISO downloads

When delivering a Linux distribution, its infrastructure and associated processes effectively act as the “spine” of the project. But spine surgery can be a delicate affair, particularly when it comes to rehabilitation after successful surgery.

For us, this cycle has been particularly demanding, as we have completed an MVP (Minimum Viable Product) port of our infrastructure tooling code to Rust, meaning that all core AerynOS tooling has now fully transitioned away from DLang.

We have covered the reasons for this transition previously, and it’s fair to say that we are already feeling the benefits of easy and native reuse of code in our tooling repositories and welcoming more Rust contributors into our community.

🏗️ AerynOS: The OS As Infrastructure

Taking a break from our usual release-oriented updates, I thought it was high time to dive into what AerynOS actually is and what sets it apart from other distros. Fair warning, this is a meaty, in depth post, and still only scratches the surface. If you’re interested in the future of Linux distributions, read on. It may also help to visit our work-in-progress documentation site for more information.

Firstly, AerynOS isn’t Yet Another Linux Distribution. It’s a platform, a foundation, and a set of tools engineered in accordance with a vision and design that just so happens to produce a Linux distribution. Were we to go back in time and build a design brief for AerynOS, the initial question might be:

What if the operating system itself behaved like modern infrastructure?

AerynOS is the answer to that question. What if, instead of doing things the way they’ve always been done, we started from the ground up and produced a well designed system, rather than the traditional model of in-place mutation internal to a distribution?

Leveraging years of experience across the industry, including some notable pioneering projects such as Solus, Clear Linux, and others, we set out to build things the hard way, in order to make them easy.. eventually.

At the very core, is the decision to not be another GNU/Linux distribution. We default to the LLVM toolchain, using libc++ and compiler-rt by default. This isn’t just a case of “we like LLVM”, but rather a strategic decision to leverage the superior diagnostics, and ensure correctness / portability of packages. Whilst in our very early stages a few years ago we did trial musl, we do default to glibc for compatibility reasons and its superior performance characteristics. The performance advantage of glibc over musl is well-documented, particularly for compute-intensive workloads and applications requiring optimal threading performance. We’re here to build a working, usable system, for a multitude of use cases and verticals, and glibc provides the best balance of compatibility and performance.

Note, we do still package gcc and it’s trivial to enforce its use in a package recipe by setting the toolchain field in stone.yaml to gnu.

Lastly, it’s worth noting we build all packages for x86_64-v2 as our minimum baseline, and perform targeted optimisation in our package recipes using the extensive tuning options configurable in boulder.

🚀 Hello AerynOS

Hello from AerynOS! ✨ As you recall from February, we set about rebranding Serpent OS into what you now see, AerynOS. It’s not been without challenges, and where possible we’ve ensured continuity. Additionally we silently dropped a few ISOs, and can now take our first formal steps under our new identity. TLDR: the transition has been entirely fluid and you only need to keep updating, no manual intervention is necessary! 🎉

Progress has been steady but quiet lately, as due to unfortunate circumstances I’ve been working in bursts from my wifes hospital bedside.

OK, lets get right to it. We’ve released AerynOS 2025.03, with the following shinies:

  • 🖥️ GNOME 48.0
  • 🐧 Linux 6.13.8
  • 🦊 Firefox 136.0.2
  • 🎮 Mesa 25.0.2
  • 🚀 Vulkan SDK 1.4.309.0
  • 🛠️ LLVM 19.1.7

We’ve also decoupled the internal tooling version from the ISO versions to make them a little bit more.. well.. readable, for humans.

Grab it now from the download page.

Evolve This OS

A long overdue, and highly requested update today. We’re finally rebranding the project!

The “Serpent OS” name was a quickly chosen name that stuck. Unfortunately “serpents” are often associated with negative connotations, and we’ve had a lot of feedback over the years that the name was off-putting. Let’s be completely honest, it’s not the most inviting name for a project. Who wants to trust a serpent? Generally speaking they’re considered dangerous at best.

It’s fair to say we’ve spent a long time in prototype and alpha phases. In order to move forward, our identity needs to be more befitting of the project we’re building. A move into the real world. This isn’t a hobby project, it’s a full blown Linux distribution with serious technical underpinnings, achievements and goals. Getting the tone right from day dot is critical.

Do note, there is no change to the internal core team, so we’re aiming for full continuity and a seamless transition. Had enough of these snakes on this plane? We have too.

Hah, it’s not “Evolve OS”, despite the clickbait title. Our new name is… drum roll please…

Hello 2025

TL;DR: Serpent OS is facing funding challenges but development continues. Alpha2 is coming soon with an improved installer. We’re seeking community support through donations and volunteers for key roles. Our technical roadmap includes versioned repositories, immutable OS features, and improved package management workflows.


We had a flurry of activity around the Christmas period, including our first alpha release as well as enabling offline rollbacks early in January. We’re actively working on alpha2, but we also need to talk about the elephant in the room.

Recently I posted a tweet that has been covered by a few news outlets. The long and short is quite simple: distinct lack of funds. A very long story short: a change in my contract led to a loss of income protection, combined with a herniated disc (C6-C7), culminated in having to resign. Naturally I’ve been working exclusively on Serpent OS since then, but I’ve also got to keep the lights on at home.

Broke

This doesn’t mean the project is closing or anything like that, but it does mean that I actively need to keep costs down whilst using daytime hours for financing. This is what’s led to the significant slowdown in the last few weeks, and I’m just trying to be as transparent as possible and manage expectations.

Offline Rollbacks Enabled

If a picture tells a thousand words… well then this video .. Yknow what, sod it. Here’s a video of the offline rollback support in action, as tested in a newly installed VM using xfs for the root filesystem. We jokingly refer to this as the “LTT” test due to LinusTechTips’ “interesting encounters” with package management systems in the past.

This has already landed in our volatile repository, so simply sudo moss sync -u and grab the latest updates. Any subsequent transactions will automatically generate the boot entries for you, no intervention required. Or mounting. Or anything. Just update and go. Srsly.

We actually saw this as a golden standard for the feature, as it’s a real-world scenario where you’d want to roll back to a known good state. The video shows the system booting and recovering in multiple ways, from complete system nuke (via glibc) and simpler scenarios like removal of the GNOME desktop environment.

The intent is to ensure a user can quickly revert to the last transaction that worked in the instance an update goes awry, without needing to boot into a rescue environment or live CD.

Investing in the Future

Here we are, at the end of 2024, and what a year it’s been! We’ve made significant progress on Serpent OS, and we’re excited to share some of the highlights with you.

Virtually 5 years in the making, we recently attained alpha status. Our tooling and concepts have aligned, allowing us to now rapidly iterate on the core deliverable itself: Serpent OS. We’ve seen an explosion in cadence, with our tooling enabling us to quickly and easily deliver updates, new packages and enabling new features.

Since the Alpha 1 ISO (0.24.5), we’ve fixed a number of issues in relation to hardware and booting. We’ve also added new features to moss (refined boot management) and boulder (our build system). Even now we have multiple PRs open to tidy up moss, add automatic generation of monitoring.yaml files, and more.

We’ve been working hard to keep the cadence up, and we’re now at a point where we can deliver new ISOs on a regular basis. Our main focus now is the tooling, in enabling a far more automatic delivery pipeline. The vision is having automated pull requests generated for package updates, automatically handling changes in dependencies and ABI, and ordered by build tier. This will significantly reduce the maintainer burden, in conjunction with planned security monitoring.

Alpha Refresh Available

In response to feedback from the community, we’re issuing a refresh of the Alpha 1 release, 0.24.6. This refresh includes a number of key updates and improvements, including:

  • Now works with etcher
  • Fractional scaling enabled by default
  • Prebuild icon theme caches to speed up trigger execution (except for hicolor)
  • Add new moss boot status command for introspection into blsforme functionality
  • Fix amdgpu initialisation for older devices, previously leading to black screen on boot

Fractional scaling, much loved

Serpent OS Enters Alpha

Following a successful prealpha phase, we’re excited to announce that Serpent OS has now entered the alpha stage of development. This milestone marks a significant step forward in the project’s journey. While the usual disclaimers apply, as a certain level of project fluidity is to be expected, we invite you to explore the latest alpha release and experience the cutting-edge features and improvements we’ve introduced.

Screenshot of a Gnome desktop showing files open

Serpent OS is a community-driven project, and we welcome contributions from developers, testers, and users alike. If you’re interested in helping shape the future of Linux distributions, we invite you to join our growing community and get involved in the project. Whether you’re interested in packaging, development, documentation, testing, or community support, there are many ways to contribute to Serpent OS.

Lastly, you can support the project by becoming a sponsor. Your support helps cover our hosting costs and allows us to dedicate more time to making Serpent OS exceptional. We’re grateful for the support we’ve received so far and look forward to continuing to build something meaningful in the Linux ecosystem.

Serpent OS Prealpha0 Released

Well, it didn’t take us that long, really … Our technical preview, prealpha0, is now available for testing!

A live GNOME environment running on prealpha0 of SerpentOS

Head to our download page to grab serpent-os-prealpha-0.iso now!

This is a super rough version of Serpent OS that is capable of being installed on baremetal hardware and VMs that support UEFI and OpenGL acceleration. It is however not recommended for use daily due to the early nature and a bunch of packages being wildly out of date. It features a minimal GNOME desktop with the Firefox web browser and a terminal.

Right now it contains a CLI installer that can be accessed from the terminal by typing:

Terminal window
sudo lichen

To detract from casual use it is necessary to manually partition the disk before running the (net) installer. You can use fdisk to create a GPT disk with a (mandatory) EFI System Partition and an optional XBOOTLDR partition. This currently also needs to be FAT32 until we integrate systemd-boot driver support. Lastly of course you will need a large enough root partition.

You will need a working network connection, so make sure you’re connected before starting the installer!

Calm Before The Storm

It’s that time of month again, and we have some details to share with you on boot management, as well as plans for real installs landing in May. Additionally, plans for “deferred updates” via mandatory reboots have been dropped!

wip boot code

To better understand the technical deep-dive on boot management further below, we should clarify a slight change in priorities for the project as a whole. For some time now we’ve served as a technical beacon for Solus to guide them towards better strategy and technical solutions whilst awaiting a situation where a rebase is feasible.

While we still look forwards to that future, the present day is the most important to us. Serpent OS must now transition from a collection of tools and promises into a daily driver with actual users. We still aim to cater to the more developer/technical oriented end of the spectrum rather than a “desktop distro”, and want to start this journey yesterday.

The boot management code is currently landing over the coming days, and will be immediately followed by an incredibly rudimentary system installer to facilitate an initial adoption of Serpent OS for the general public. Our intent is to ship some early test ISOs for supporters, and by the end of May have a feasible option for people to use and contribute to Serpent OS on real hardware.

We acknowledge this is an exchange of the pursuit of perfection for a pragmatic rapid iteration cycle and believe the best way forward is to open the doors wide, fixing issues on the job. The initial system will likely be rough and buggy, but it is an official starting point from which we can all build Serpent OS together.

The March Of Progress

Despite a brief excursion out of the country for a first-in-a-lifetime holiday, I’m happily back at the desk to bring you up to date with the latest goings in Serpent OS. TLDR: Loads of awesome, baremetal is enabled, ISO cycle in next couple of weeks.

GNOME on Serpent OS

OK, we all know you want to know about the baremetal stuff, but c’mon, check out this changeset!

End of February Update

This update came a little later in the month, as we’ve got a lot of exciting news to share. Everything from boulder in Rust, to the GNOME 45 Desktop complete with moss triggers built atop a rebootstrapped toolchain.

We’re pleased to announce that over the course of this weekend, once testing has completed we’ll deploy the latest version of boulder, our packaging build tool. This has been given the Rust treatment, directly sharing the codebase with moss.

Boulder in action

Unlike our PoC implementation, the new boulder makes use of the clone syscall to execute portions of itself under user namespaces, eliminating a longstanding deployment irritation of having the main build binary inside packaging.

Just like moss it makes use of tokio for async right where we need it, and is rootless to make life easier for our contributors. Far from being just a direct port of the proof of concept, it adds new tools to empower developers, such as breakpoints in packaging recipes.

Breakpoints, in recipes

January Updates

Precisely one month since our end of year summary, so, what have we been up to? With a new year, a new start was long overdue. We’re pleased to announce that we’ve finally shifted our butts over to our own server (again).

While this isn’t quite as chunky a news update as the last post, we rather hope you appreciate the regularity of the news.

moss in action

‘tarkah’ has been tirelessly porting moss-service, the core foundation of our automated build system, to a shiny new Rust codebase. While we don’t have an immediately pressing need for the port to come online, it will certainly address some “v1 issues” and ensure the best-in-class development experience for our contributors.

In our pursuit for a sustainable, community oriented project, we’ve been doing a lot of in house organisation. Very importantly, we’ve now relicensed our packaging recipes to MPL-2.0, bringing them in line with the license chosen for our current engineering efforts (moss, etc).

To reiterate a critical point, we require that our contributions are attributed to the virtual collective “Serpent OS Developers” to ensure we’re unable to relicense any contributions without express consent of the contributors involved.

Additionally we picked MPL-2.0 over Zlib due to some distinct advantages in relation to patent trolls. In combination we believe both the project and our users will have the greatest protection from patent trolls and fiscally-motivated relicensing.

End Of Year Summary

Burning question - how long before we can use Serpent OS on our development systems? It’s a fair question - and one that requires context to answer. Let’s start our new monthly update cycle with a recap of progress so far.

screenshot of moss

We’d like to extend a huge official welcome, and thank you, to newest team member Cory Forsstrom (aka tarkah)!

Firstly, a quote from Cory:

I’ve been a long time Solus user and was very excited about what serpent os was doing. Really got invested once I started diving into the D code and seeing how powerful the tools and ideas were. The Rust rewrite was just a perfect storm for me with my experience there and my desire to make contributions. Getting to know you and ermo has just been icing on the cake, you’ve both been so welcoming and friendly. So lots of fun times.

On the personal side, I’m on the west coast in the States, have a lovely wife, just had a baby girl and am enjoying my time with fatherhood and coding

Chances are you know tarkah for his contributions to iced-rs, and to Solus. His contributions to the moss-rs project have been absolutely critical, and his patience essential as we got ourselves quickly up to speed with Rust. It is entirely fair to say that our Rust efforts would not have been possible without him!

Oxidised Moss

Allow me to start this post by stating something very important to me: I absolutely love the D programming language, along with the expressivity and creative freedom it brings. Therefore please do not interpret this post as an assassination piece.

For a number of months now the Serpent OS project has stood rather still. While this could naively be attributed to our shared plans with Solus - a deeper, technical issue is to be acredited.

img

D isn’t quite there yet. But it will be, some day.

Section titled “D isn’t quite there yet. But it will be, some day.”

Again, allow me to say I have thoroughly enjoyed my experience with D over the last 3 or so years, it has been truly illuminating for me as an engineer. With that said, we have also become responsible for an awful lot of code. As an engineering-led distribution + tooling project, our focus is that of secure and auditable code paths. To that extent we pursued DIP1000 as far as practical and admit it has a way to go before addressing our immediate needs of memory safety.

While we’re quite happy to be an upstream for Linux distributions by way of release channels and tooling releases, we don’t quite have the resources to also be an upstream for the numerous D packages we’d need to create and maintain to get our works over the finish line.

With that said, I will still continue to use D in my own personal projects, and firmly believe that one day D will realise its true potential.

Snakes On A Boat

We had intended to get a blog post out a little bit quicker, but the last month has been extremely action packed. However, it has paid off immensely. As our friends at Solus recently announced it is time to embark on a new voyage.

dashboard preview

There is very little point in rehashing the post, so I’ll cover the essentials

  • We assisted the Solus team in deploying a complete new infrastructure. Some of this work is ongoing
  • The package delivery pipeline is in place, meaning updates will soon reach users.
  • We’re working towards a mutually beneficial future whereby Solus adopts our tooling.

Oh no, no. The most practical angle to view this from is that Serpent OS is free to build and innovate to create the basis of Solus 5. Outside of the distribution we’re still keen to continue development on our tooling and strategies.

It is therefore critical that we continue development, and find the best approach for both projects, to make the transition to Solus 5 as seamless as possible. To that end we will still need to produce ISOs and have an active community of users and testers. During this transition period, despite being two separate projects, we’re both heading to a common goal and interest.

Infrastructure Launched!

After many months and much work, our infrastructure is finally online1. We’ve had a few restarts, but it’s now running fully online with 2 builders, ready to serve builds around the clock.

Firstly, I’d like to apologise for the delay since our last blog post. We made the decision to move this website to static content, which took longer than expected. We’re still using our own D codebase, but prebuilding the site (and fake “API”) so we can lower the load on our web server.

Our infrastructure is composed of 3 main components.

First look at summit

This is the page you can see over at dash.serpentos.com1. It contains the build scheduler. It monitors our git repositories, and as soon as it discovers any missing builds it creates build tasks for them. It uses a graph to ensure parallel builds happen as much as possible, and correctly orders (and blocks) builds based on build dependencies.

We have 2 instances of Avalanche, our builder tool, running. This accepts configuration + build requests from Summit, reporting status and available builds.

This is an internal repository manager, which accepts completed packages from an Avalanche instance. Published packages land at build.aerynos.dev2

Super early days with the infra but we now have builds flowing as part of our continuous delivery solution. Keeping this blog post short and sweet… we’re about to package GNOME and start racing towards our first real ISOs!

If you like what the project is doing, don’t forget to sponsor!


  1. The infrastructure used to be hosted at dash.serpentos.com but has since moved to our aerynOS domain. ↩ ↩2

  2. Vessel used to be hosted at dev.serpentos.com but has since moved to our aerynOS domain. ↩

Lift Off

Enough of this “2 years” nonsense. We’re finally ready for lift off. It is with immense pleasure we can finally announce that Serpent OS has transitioned from a promise to a deliverable. Bye bye, phantomware!

A booted Serpent OS

As mentioned, we spent 2 years working on tooling and process. That’s .. well. Kinda dull, honestly. You’re not here for the tooling, you’re here for the OS. To that end I made a decision to accelerate development of the actual Linux distro - and shift development of tooling into a parallel effort.

I deferred final enabling of the infrastructure until January to rectify the chicken/egg scenario whilst allowing us to grow a base of contributors and an actual distro to work with. We’re in a good position with minimal blockers so no concern there.

The Big Update

Well - we’ve got some big news! The past few weeks have been an incredibly busy time for us, and we’ve hit some major milestones.

After much deliberation - we’ve decided to pull out of Open Collective1. Among other reasons, the fees are simply too high and severely impact the funds available to us. In our early stages, the team consensus is that funds generated are used to compensate my time working on Serpent OS.

As such I’m now moving funding to my own GitHub Sponsors page1 - please do migrate! It ensures your entire donation makes it and keeps the lights on for longer =) Please remember I’m working full time on Serpent OS exclusively - I need your help to keep working.

We’ve pretty much completed our transition to GitHub. We’ve now got the following organisations:

Don’t forget - our forums are live over at forums.serpentos.com2 - please feel free to drop in and join in with the community 😀

OK so what exactly are moss and boulder? In short - they’re the absolute core pieces of our distribution model.

On the surface, moss looks and feels roughly the same as just about any other traditional package manager out there. Internally, however, its far more modern and has a few tricks up its sleeve. For instance, every time you initiate an operation in moss, be it installation, removal, upgrade, etc, a new filesystem transaction is generated. In short, if something is wrong with the new transaction - you can just boot to an older transaction when things worked fine.

Now, it’s not implemented using bundles or filesystem specific functionality, internally its just intelligent use of hardlinks and deduplication policies, and we have our own container format with zstd based payload compression. Our strongly typed, deduplicating binary format is what powers moss.

Behind the scenes we also use some other cool technology, such as LMDB for super reliable and speedy database access. The net result is a next generation package management solution that offers all the benefits of traditional package managers (i.e. granularity and composition) with new world features, like atomic updates, deduplication, and repository snapshots.

It’s one thing to manage and install packages, it’s another entirely to build them. boulder builds conceptually on prior art such as pisi and the package.yml format used in ypkg. It is designed with automation and ease of integration in mind, i.e. less time spent focusing on packaging and more time on actually getting the thing building and installing correctly.

Boulder supports “macros” as seen in the RPM and ypkg world, to support consistent builds and integration. Additionally it automatically splits up packages into the appropriate subpackages, and automatically scans for binary, pkgconfig, perl and other dependencies during source analysis and build time. The end result is some .stone binary packages and a build manifest, which we use to flesh out our source package index.

We’ve spent considerable time reworking moss, our package manager. It now features a fresher (terminal) user interface, progress bars, and is rewritten to use the moss-db module encapsulating LMDB.

nspawn

It’s also possible to manipulate the binary collections (software repositories) used by moss now. Note we’re going to rename “remote” to “collection” for consistency.

At the time of writing:

Terminal window
$ mkdir destdir
$ sudo moss remote add -D destdir protosnek https://dev.serpentos.com/protosnek/x86_64/stone.index
$ sudo moss install -D destdir bash dash dbus dbus-broker systemd coreutils util-linux which moss nano
$ sudo systemd-nspawn -b -D destdir

This will be simplified once we introduce virtual packages (Coming Soon ™)

Boulder can now be instructed to utilise a local collection of stone packages, simplifying the development of large stack items.

Terminal window
sudo boulder bi stone.yml -p local-x86_64

Packages should be moved to /var/cache/boulder/collections/local-x86_64 and the index can be updated by running:

Terminal window
sudo moss idx /var/cache/boulder/collections/local-x86_64

Serpent OS is now officially self hosting. Using our own packages, we’re able to construct a root filesystem, then within that rootfs container we can use our own build tooling (boulder) to construct new builds of our packages in a nested container.

The protosnek collection has been updated to include the newest versions of moss and boulder.

self hosting

As a fun experiment, we wanted to see how far along things are. Using a throwaway kernel + initrd build, we were able to get Serpent OS booting using virtualisation (qemu)

booting

Right now everyone is working in the snekpit organisation to get our base packaging in order. I’m looking to freeze protosnek, our bootstrap collection, at the latest of tomorrow evening.

We now support layered, priority based collections (repositories) and dependency solving across collection boundaries, allowing us to build our new main collection with protosnek acting as a bootstrap seed.

Throughout this week, I’ll focus on getting Avalanche, Summit and Vessel into shape for PoC so that we can enjoy automated builds of packages landing in the yet-to-be-launched volatile collection.

From there, we’re going to iterate + improve packaging, fixing bugs and landing features as we discover the issues. Initially we’ll look to integrate triggers in a stateless-friendly fashion (our packages can only ship /usr by design) - after that will come boot management.

An early target will be Qemu support via a stripped linux-kvm package to accelerate the bring up, and we encourage everyone to join in the testing. We’re self hosting, we know how to boot, and now we’re able to bring the awesome.

I cannot stress how important the support to the project is. Without it - I’m unable to work full time on the project. Please consider supporting my development work via GitHub Sponsors1.

I’m so broke they’ve started naming black holes after my IBAN.

Thank you!

You can discuss this blog post over on forums2.


  1. Since this blog post, we no longer use OpenCollective or GitHub Sponsors. For funding, please see our Funding Page and for ISOs, please see our Download Page. ↩ ↩2 ↩3

  2. Since this blog post, we now use Discourse as our forum. ↩ ↩2

Infrastructure Update

Since the last post, I’ve pivoted to full time work on Serpent OS, which is made all the more possible thanks to everyone supporting us via OpenCollective1 ❤️.

We’ve been working towards establishing an online infrastructure to support the automation of package builds, while revisiting some core components.

During the development of the Serpent OS tooling we’ve been exploring the possibilities of D Lang, picking up new practices and refining our approach as we go. Naturally, some of our older modules are somewhat … smelly. Most noticeable is our moss-db module, which was initially intended as a lightweight wrapper around RocksDB.

In practice that required an encapsulation API written in D around the C API, and our own wrapping on top of that. Naturally, it resulted in a very allocation-heavy implementation that just didn’t sit right with us, and due to the absurd complexity of RocksDB was still missing quite a few features.

Packaging Automation, Next Steps

Hot damn we’ve been busy lately. No, really. The latest development cycle saw us focus exclusively on boulder, our build tooling. As of today it features a proof of concept boulder new subcommand for the automatic generation of packaging templates from an upstream source (i.e. tarball).

Before we really start this blog post off, I’d like to thank everyone who is supporting the project! All of the OpenCollective1 contributions will make it easier for me to work full time on Serpent OS 😀 Much love ❤️

Look at all the buildiness

Alright you got me there, certain projects prefer to abstract the configuration, build and installation of packages and be provided with some kind of hint to the build system, i.e. manually setting autotools, etc.

Serpent OS packaging is declarative and well structured, and relies on the use of RPM-style “macros” for distribution integration and common tasks to ensure consistent packaging.

We prefer a self documenting approach that can be machine validated rather than depending on introspection at the time of build. Our stone.yml format is very flexible and powerful, with automatic runtime dependencies and package splitting as standard.

..Doesn’t mean we can’t make packaging even easier though.

A Word From The Founder

Well well, it’s been a long time since I personally wrote a post.. 😀 So let’s keep this short and sweet, shall we? I’m returning to full time work on Serpent OS.

The 6th of July will be my last day at my current employment having tendered my 30 day notice today. Despite having enjoyment at my current position, the reality is that my passion and focus is Serpent OS.

I’m now in a transition process and will ramp up my efforts with Serpent OS. Realistically I need to reduce the outgoing costs of the project and with your help I can gain some level of financial support as we move through the next stages of development. Worst case, I will only take on any part-time or contractual gigs, allowing my primary focus to be Serpent OS.

I’ll begin accelerating works and enabling community contribution so we can get the derailed-alpha train back on the tracks.

I have absolute faith in this project, the community and our shared ability to deliver the OS and tooling. To achieve it will require far more of my time and I’m perfectly willing to give it.

Thank you all to everyone who has been supporting the project, it is now time to deliver. Not just another run of the mill distribution but a technically competent and usable distribution that is not only different but better.

Let’s do this in the most grassroots and enjoyable way possible 😊

RELR Brings Smaller Files, More Performance?

RELR is an efficient method of storing relative relocations (but is not yet available in glibc upstream). This has a significant reduction on file size often in the vicinity of 5% for libraries and even higher for PIE binaries. We also take a look at the performance impact on enabling RELR and that looks really good too! Smaller files with more performance - you can have your cake and eat it too!

Everyone enjoys smaller files, especially when it’s for free! RELR provides a very efficient method of storing relative relocations where it only requires a few percent compared to storing them in the .rela.dyn section which is what currently happens. However, it can’t store everything so the .rela.dyn section remains (though much smaller).

Here’s an example of the sections of libLLVM-13.so with and without RELR. Here we see the .rela.dyn section taking up a whole 2MB! When enabling RELR, .rela.dyn shrinks down to just under 100KB while adding a new section .relr.dyn which is just over 20KB! That’s nearly a 1.9MB file size reduction, so you’ll get smaller packages, smaller updates and it will be even faster to create delta packages from our servers. For reference, some of the biggest files have a .rela.dyn section over 10MB!

Section Without RELR With RELR
.dynstr 2,285,738 2,285,723
.rela.dyn 2,006,472 97,464
.relr.dyn - 21,688
.text 28,708,290 28,708,386
.dynamic 592 608
Total 44,853,987 42,966,764

Making Deltas Great Again! (Part 1)

In Optimising Package Distribution we discussed some early findings for implementing binary deltas in Serpent OS. While discussing the implementation we have found the requirements to be suboptimal for what we were after. We provide a fresh look at the issue and what we can do to make it useful in almost all situations without the drawbacks.

I remember back in the early 2000s on Gentoo when someone set up a server to produce delta patches from source tarballs to distribute to users with small data caps such as myself. When requesting the delta, the job would be added to the queue (which occasionally could be minutes) and then created a small patch to download. This was so important at the time to reduce the download size that the extra time was well worth it!

Today things are quite different. The case for deltas has reduced for users as internet speeds have increased. Shrinking 20MB off a package size may be a second reduction for some, but 10 seconds for others. The largest issue is that deltas have typically pushed extra computational effort onto the users computer in compensation for the smaller size. With a fast internet connection that cost is a real burden where deltas take longer to install than simply downloading the full package.

Can Hardly Contain Myself, Plus a Bonus

One of the core steps for building a package is setting up a minimal environment with only the required (and stated) dependencies. Currently we have been building our stones in an systemd-nspawn container, where the root contains every package that’s been built so far. This makes the environment extremely difficult to reproduce!

Today we announce moss-container, a simple but flexible container creator that we can integrate for proper containerized builds.

Containers have a multitude of uses for a Linux distro, but our immediate use case is for reproducible container builds for boulder. However, we have plans to use moss-container for testing, validation and benchmarking purposes as well. Therefore it’s important to consider all workloads, so features like fakeroot and networking can be toggled on or off depending on what features are needed.

moss-container takes care of everything, the device nodes in the /dev tree, mounting directories as tmpfs so the environment is left in a clean state, and mounting the /sys and /proc special file-systems. These are essential for a fully functioning container where programs like python and even clang won’t work without them. And best of all, it’s very fast so fits in well with the rest of our tooling!

The next step is integrating moss-container into boulder, so that builds become reproducible across machines, and makes it much easier for users to run builds on their host machines.

Performance Corner: Small Changes Pack a Punch

Here we have another round of changes to make packages smaller and show just how much we care about performance and efficiency! Today we are focusing mainly on moss-format changes to reduce the size of its payloads. The purpose of these changes is to reduce the size of packages, DB storage for transactions and memory usage for moss. These changes were made just before we moved out of bootstrap, so we wouldn’t have to rebuild the world after changing the format. Lets get started!

Squeezing Out the Last Blit of Performance

Section titled “Squeezing Out the Last Blit of Performance”

Blitting in Serpent OS is the process of setting up a root, using hardlinks of files installed in the moss store to construct the system /usr directory. Some initial testing showed buildPath using about 3% of the total events when installing a package with many files. As a system is made up of over 100,000 files, that’s a lot of paths that need to be calculated! Performance is therefore important to minimize time taken and power consumed.

After running a few benchmarks of buildPath, there were a few tweaks which could improve the performance. Rather than calculating the full path of the file, we can reuse the constant root path for each file instead, reducing buildPath events by 15%. Here we see some callgrind data from this small change (as it was difficult to pick up in the noise).

Before:
48,766,347 ( 1.50%) /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath
34,241,805 ( 1.05%) /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath
14,363,684 ( 0.44%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath
After:
40,357,800 ( 1.25%) /usr/include/dlang/ldc/std/path.d:pure nothrow @safe immutable(char)[] std.path.buildPath
29,523,966 ( 0.91%) /usr/include/dlang/ldc/std/utf.d:pure nothrow @safe immutable(char)[] std.path.buildPath
12,814,283 ( 0.40%) /usr/include/dlang/ldc/std/range/package.d:pure nothrow @safe immutable(char)[] std.path.buildPath

Out of the Bootstrap - Towards Serpent OS

The initial stone packages that will seed the first Serpent OS repo have now been finalized! This means that work towards setting up the infrastructure for live package updates begins now. We plan on taking time to streamline the processes with a focus of fixing issues at the source. In this way we can make packaging fast and efficient so we can spend time working on features rather than package updates.

Bootstrapping a distribution involves building a new toolchain and many packages needed to support it. For us bootstrap was getting us to the point where we have built stone packages that we can use to start an initial repository with full working dependencies. This has been enabled by integrating dependencies into moss, creating the first repo index. Of note is that it is already enabled for 32bit support, so we have you covered there. While this is the end of bootstrap, the fun has only just begun!

The first install from the bootstrap

The next goal is to make Serpent OS self-hosting, where we can build packages in a container and update the repo index with the newly built packages. It is essentially a live repository accessible from the internet. There’s still plenty of improvements to be made with the tooling, but will soon enable more users to participate in bringing Serpent OS into fruition.

While there’s a strong focus in Serpent OS on performance, the decision has been made to lower the initial requirements for users. Despite AVX2 being an older technology, there are still computers sold today that don’t support it. Because of this (and already having interested users who don’t have newer machines), the baseline requirement for Serpent OS will be x86_64-v2, which only requires SSE4.2.

It was always the plan to add support for these machines, just later down the track. In reality, this makes a lot more sense, as there will be many cases where building 2 versions of a package provides little value. This is where a package takes a long time to build and doesn’t result in a notable performance improvement. We will always need the x86_64-v2 version of a package to be compatible with the older machines. With this approach we can reduce the build server requirements without a noticeable impact to users as only a few packages you use will be without extra optimizations (and probably don’t benefit from them anyway).

I want to make it clear that this will be temporary, with impactful x86_64-v3+ packages rolling out as soon as possible. This change paves the way to integrate our technology stack taking care of your system for you and increases its priority. Users meeting the requirements of the x86_64-v3+ instruction set (this includes additional instructions beyond x86_64-v3) will automatically start installing these faster packages as they become available. Our subscriptions model will seamlessly take care of everything for you behind the scenes so you don’t need to read a wiki or forum to learn how to get these faster packages. We can utilize the same approach in future for our ARM stack, offering more optimized packages where it provides the most benefit.

Note that from the bootstrap, most packages built in under 15s and only three took longer than 2 minutes.

While the project is young is a great time to test out new technologies. The use of clang and lld open up new possibilities to reduce file sizes, shorten load times and decrease memory usage. Some of these choices may have impacts on compatibility, so testing them out will be the best way to grasp that. Making sure that you can run apps like Steam is vital to the experience, so whatever happens we will make sure it works. The good news is that due to the unique architecture of Serpent OS, we can push updates that break compatibility with just a reboot, so if we ever feel the need to change the libc, we can make the change without you having to reinstall! More importantly, we can test major stack updates by rebooting into a staged update and go straight back to the old system, regardless of your file system.

In the early days of the repository, tooling to make creating new packages as simple as possible is vital for efficiency. Spending some time automating as much of the process as possible will take weeks off bringing up Serpent OS. By making packaging as easy as possible will also help users when creating their own packages. While it would be faster to work around issues, the build tooling upgrades will benefit everyone.

The other way we’ll be speeding up the process is by holding back some of the tuning options by default. LTO for instance can result in much longer build times so will not initially be the default. The same is true for debug packages, where it slows down progress without any tangible benefit.

We hope you are as excited as we are!

It All Depends

It all depends.. it really does. On shared libraries.. interpreters.. pkg-config providers and packages. It’s the same story for all “package managers”, how do we ensure that the installed software has everything it needs at runtime?

Our merry crew has been involved in designing and building Linux distributions for a very, very long time, so we didn’t want to simply repeat history.

Using updated moss

Thanks to many improvements across our codebase, including moss-deps, we automatically analyse the assets in a package (rapidly too) to determine any dependencies we can add without requiring the maintainer to list them. This is similar in nature to RPM’s behaviour.

As such we encode dependencies into our (endian-aware, binary) format which is then stored in the local installation database. Global providers are keyed for quick access, and the vast majority of packages will not explicitly depend on another package’s name, rather, they’ll depend on a capability or provider. For subpackage dependencies that usually depend on “NEVRA” equality (i.e. matching against a name with a minimum version, release, matching epoch and architecture), we’ll introduce a lockstep dependency that can only be resolved from its origin source (repo).

Lastly, we’ll always ensure there is no possibility for “partial update” woes. With these considerations, we have no need to support >= style dependencies, and instead rely on strict design goals and maintainer responsibility.

Performance Corner: Faster Builds, Smaller Packages

Performance Corner is a new series where we highlight to you some changes in Serpent OS that may not be obvious, but show a real improvement. Performance is a broad term that also covers efficiency, so things like making files smaller, making things faster or reducing power consumption. In general things that are unquestionably improvements with little or no downside. While the technical details may be of interest to some, the main purpose is to highlight the real benefit to users and/or developers that will make using Serpent OS a more enjoyable experience. Show me the numbers!

Here we focus on a few performance changes Ikey has been working on to the build process that are showing some pretty awesome results! If you end up doing any source builds, you’ll be thankful for these improvements. Special thanks to ermo for the research into hash algorithms and enabling our ELF processing.

When measuring changes, it’s always important to know where you’re starting from. Here are some results from a recent glibc build, but before these latest changes were incorporated.

Payload: Layout [Records: 5441 Compression: Zstd, Savings: 83.13%, Size: 673.46 KB]
Payload: Index [Records: 2550 Compression: Zstd, Savings: 55.08%, Size: 247.35 KB]
Payload: Content [Records: 2550 Compression: Zstd, Savings: 81.46%, Size: 236.72 MB]
==> 'BuildState.Build' finished [4 minutes, 6 secs, 136 ms, 464 μs, and 7 hnsecs]
==> 'BuildState.Analyse' finished [21 secs, 235 ms, 300 μs, and 2 hnsecs]
==> 'BuildState.ProducePackages' finished [25 secs, 624 ms, 996 μs, and 8 hnsecs]

The build time is a little high, but a lot of that is due to a slow compiler on the host machine. But analysing and producing packages was also taking a lot longer than it needed to.

Optimal File Locality

File locality in this post refers to the order of files in our content payload. Yes that’s right, we’re focused on the small details and incremental improvements that combined add up to significant benefits! All of this came about from testing the efficiency of content payload in moss-format and how well it compared against a plain tarball. One day boulder was looking extremely inefficient and then retesting the following day was proving to be extremely efficient without any changes made to boulder or moss-format. What on Earth was going on?

To test the efficiency our content payload, the natural choice was to compare it to a tarball containing the same files. When first running the test the results were quite frankly awful! Our payload was 10% larger than the equivalent tarball! It was almost unbelievable in a way, so the following day I repeated the test again only this time the content payload was smaller than the tarball. This didn’t actually make sense, I made the tarball with the same files, but only changed the directory it was created from. Does it really matter?

Of course it does (otherwise it would be a pretty crappy blog post!). When extracting a .stone package it creates two directories, mossExtract where the sha256sum named files are stored and mossInstall where those files are hardlinked to their full path name. The first day I created the tarball from mossInstall and the second day I realised that creating the tarball from mossExtract would provide the closest match to the content payload since it was a direct comparison. When compressing the tarballs to match the .stone compression level, the tarball compressed from mossInstall was 10% smaller, despite the uncompressed tarball being slightly larger.

Unpacking the Build Process: Part 2

Part 2 looks at the core of the build process, turning the source into compiled code. In Serpent OS this is handled by our build tool boulder. It is usually the part of the build that takes the longest, so where speed ups have the most impact. How long it takes is largely down to the performance of your compiler and what compile flags you are building with.

This post follows on from Unpacking the Build Process: Part 1.

The steps for compiling code are generally quite straight-forward:

  • Setting up the build (cmake, configure, meson)
  • Compiling the source (in parallel threads)
  • Installing the build into a package directory

This will build software compiled against packages installed on your system. It’s a bit more complicated when packaging as we first set up an environment to compile in (Part 1). But even then you have many choices to make and each can have an impact on how long it takes to compile the code. Do you build with Link Time Optimizations (LTO) or Profile Guided Optimizations (PGO), do you build the package for performance or for the smallest size? Then there’s packages that benefit considerably from individual tuning flags (like -fno-semantic-interposition with python). With so many possibilities, boulder helps us utilize them through convenient configuration options.

Unpacking the Build Process: Part 1

While the build process (or packaging as it’s commonly referred to) is largely hidden to most users, it forms a fundamental and important aspect to the efficiency of development. In Serpent OS this efficiency also extends to users via source based builds for packages you may want to try/use that aren’t available as binaries upstream.

The build process can be thought of in three distinct parts, setting up the build environment, compiling the source and post build analysis plus package creation. Please note that this process hasn’t been finalized in Serpent OS so we will be making further changes to the process where possible.

Some key parts to setting up the build environment:

  • Downloading packages needed as dependencies for the build
  • Downloading upstream source files used in the build
  • Fetching and analyzing the latest repository index
  • Creating a reproducible environment for the build (chroot, container or VM for example)
  • Extracting and installing packages into the environment
  • Extracting tarballs for the build (this is frequently incorporated as part of the build process instead)

While the focus of early optimization work has been on build time performance, there’s more overhead to creating packages time than simply compiling code. Now the compiler is in a good place, we can explore the rest of the build process.

A Rolling Boulder Gathers No Moss

We actually did it. Super pleased to announce that moss is now capable of installing and removing packages. Granted, super rough, but gotta start somewhere right?

Transactional system roots + installation in moss

OK let’s recap. A moss archive is super weird, and consists of multiple containers, or payloads. We use a strongly typed binary format, per-payload compression (Currently zstd), and don’t store files in a typical archive fashion.

Instead a .stone file (moss archive) has a Content Payload, which is a compressed “megablob” of all the unique files in a given package. The various files contained within that “megablob” are described in an IndexPayload, which simply contains some IDs and offsets, acting much like a lookup table.

That data alone doesn’t actually tell us where files go on the filesystem when installed. For that, we have a specialist Layout Payload, encoding the final layout of the package on disk.

As can be imagined, the weirdness made it quite difficult to install in a trivial fashion.

Well, persistence really. Thanks to RocksDB and our new moss-db project, we can trivially store information we need from each package we “precache”. Primarily, we store full system states within our new StateDB, which at present is simply a series of package ID selections grouped by a unique 64-bit integer.

Additionally, we remember the layouts within the LayoutDB so that we can eventually recreate said layout on disk.

Moss DB Progress

I’ll try to make this update as brief as I can but it’s certainly an important one, so let’s dive right into it. The last few weeks have been rough but work on our package manager has still been happening. Today we’re happy to reveal another element of the equation: moss-db.

Putting moss-db to the test

moss-db is an abstract API providing access to simplistic “Key Value” stores. We had initially used some payload based files as databases but that introduced various hurdles, so we decided to take a more abstract approach to not tie ourselves to any specific implementation of a database.

Our main goal with moss-db is to encapsulate the RocksDB library, providing sane, idiomatic access to a key value store.

At the highest level, we needed something that could store arbitrary keys and values, grouped by some kind of common key (commonly known as “buckets”). We’ve succeeded in that abstraction, which also required us to fork a rocksdb-binding to add the Transform APIs required.

Additionally, we required idiomatic range behaviours for iteration, as well as generic access patterns. To that affect we can now foreach a bucket, pipe it through the awesomely powerful std.algorithm APIs, and automatically encode/decode keys and values through our generic APIs when implementing the mossdbEncode() and mossdbDecode() functions for a specific type.

In a nutshell, this was the old, ugly, hard way:

/* old, hard way */
auto nameZ = name.toStringz();
int age = 100;
ubyte[int.sizeof] ageEncoded = nativeToBigEndian(ageEncoded);
db.setDatum(cast(ubyte[]) (nameZ[0 .. strlen(nameZ)]), ageEncoded);

And this is the new, shmexy way:

db.set("john", 100);
db.set("user 100", "bobby is my name");
auto result = db.get!int("john");
if (result.found)
{
writeln(result.value);
}
auto result2 = db.get!string("user 100");
if (result2.found)
{
writeln(result2.value);
}

It’s quite easy to see the new API lends itself robustly to our needs, so that we may implement stateful, strongly typed databases for moss.

Initial Performance Testing

With further progress on boulder, we can now build native stone packages with some easy tweaks such as profile guided optimizations (PGO) and link time optimizations (LTO). That means we can take a first look at what the performance of the first cut of Serpent OS shows for the future. The tests have been conducted using benchmarking-tools with Serpent OS measured in a chroot on the same host with the same kernel and config.

One of the key focuses for early in the project is on reducing build time. Every feature can either add or subtract from the time it takes to produce a package. With a source/binary hybrid model, users will greatly benefit from the faster builds as well. In terms of what I’ve targeted in these tests is the performance of clang and testing some compiler flag options on cmake.

clang has always been a compiler with a big future. The performance credentials have also been improving each release and are now starting to see it perform strongly against its GNU counterpart. It is common to hear that clang is slow and produces less optimized code. I will admit that most distros provide a slow build of clang, but that will not be the case in Serpent OS.

It is important to note that in this comparison the Host distro has pulled in some patches from LLVM-13 that greatly improve the performance of clang. Prior to this, their tests actually took 50% longer for cmake and configure but only 10% longer for compiling. boulder does not yet support patching in builds so the packages are completely vanilla.

Test using clang Serpent Host Difference
cmake LLVM 5.89s 10.58s 79.7%
Compile -j4 llvm-ar 126.16s 155.32s 23.1%
configure gettext 36.64s 63.55s 73.4%

Based on the results during testing, the performance of clang in Serpent OS still has room to improve and was just a quick tuning pass. At stages where I would have expected to be ahead already, the compile performance was only equal (but cmake and configure were still well ahead).

Boulder Keeps On Rolling

We have been squirreling away in the background to bring some great changes to bring boulder closer to its full potential. Here’s a quick recap of some of the more important ones.

Boulder hard at work

  • Fixed a path issue that prevented manifests from being written for 32bit builds
  • Added keys to control where the tarballs are extracted to
    • This results in a greatly simplified setup stage when using multiple upstreams
  • More customizations to control the final c{,xx}flags exported to the build
  • Added a key to run at the start of every stage so definitions can be exported easily in the stone.yml file
  • Fixed an issue where duplicate hash files were being included in the Content Payload
    • This resulted in reducing the Content Payload size by 750MB of a glibc build with duplicate locale files
  • Finishing touches on profile guided optimization (PGO) builds - including clang’s context-sensitive PGO
    • Fixed a few typos in the macros to make it all work correctly
    • Profile flags are now added to the build stages
    • Added the llvm profile merge steps after running the workload
    • Recreate a clean working directory at the start of each PGO phase

With all this now in place, the build stages of boulder are close to completion. But don’t worry, there’s plenty more great features to come to make building packages for Serpent OS simple, flexible and performant. Next steps will be testing out these new features to see how much they can add to the overall stage4 performance.

Let There Be Databases

We haven’t been too great on sharing progress lately, so welcome to an overdue update on timelines, progress, and database related shmexiness.

Emerging DB design

OK, so you may remember moss-format, our module for reading and writing moss binary archives. It naturally contains much in the way of binary serialisation support, so we’ve extended the format to support “database” files. In reality, they are more like tables binary encoded into a single file, identified by a filepath.

The DB archives are currently stored without compression to ensure 0-copy mmap() access when loading from disk, as a premature optimisation. This may change in future if we find the DB files taking up too much disk space.

So far we’ve implemented a “StateMetaDB”, which stores metadata on every recorded State on the system, and right now I’m in the progress of implementing the “StateEntriesDB”, which is something akin to a binary encoded dpkg selections file with candidate specification reasons.

Next on the list is the LayoutsDB (file manifests) and the CacheDB, for recording refcounts of every cached file in the OS pool.

Moss Unlocked

Well, it’s not all doom and gloom these days. We’ve actually made some significant progress in the last few days, so it seems a good time to share a progress update.

Extracting content from moss archives

Oh yeah, that totally happened. So, we can now successfully build moss packages from boulder and then extract them to disk once again with moss. This might sound totally uninteresting, but it demonstrates that our format is actually working as intended.

Admittedly the code is super rough within moss and somewhat proof of concept, however we’re able to extract the contents of the moss archive and rebuild the layout on disk.

Optimising Package Distribution

Getting updates as fast as possible to users has made deltas a popular and sought after feature for distributing packages. Over the last couple of days, I’ve been investigating various techniques we can look at to support deltas in moss.

Minimising the size of updates is particularly valuable where files are downloaded many times and even better if they’re updated infrequently. With a rolling release, packages will be updated frequently, so creating deltas can become resource intensive, especially if supporting updates over a longer period of time. Therefore, it’s important to get the right balance between compression speed, decompression memory and minimising file size.

Package priorities How best to meet these needs
Developers Creation speed Quickly created packages
Users File size and update speed Size minimised deltas

From the users point of view, minimising file size and upgrade time are important priorities, but for a developer, the speed at which packages are created and indexed is vital to progression. Deltas are different to packages in that they aren’t required immediately, so there’s minimal impact in taking longer to minimise their size. By getting deltas right, we can trade-off the size of packages to speed up development, while users will not be affected and fetch only size optimised deltas.

Moss Format: Read Write Support

It’s been 8 days since our last blogpost and a lot of development work has happened in that time. Specifically, we’ve completely reworked the internals of the moss-format module to support read/write operations… Which means installation is coming soon (TM)

Development work on moss-format

So, many commits have been made to the core repositories, however the most important project to focus on right now is moss-format, which we used to define and implement our binary and source formats. This module is shared between boulder, our build tool, and moss, our package manager.

We’ve removed the old enumeration approach from the Reader class, instead requiring that it processes data payloads in-place, deferring reading the content payload streams. We’ve also enforced strong typing to allow safe and powerful APIs:

import moss.format.binary.payload.meta : MetaPayload;
auto reader = new Reader(File(argv[0], "rb"));
auto metadata = reader.payload!MetaPayload();

Right now we can read and write our MetaPayload from and to the stream, allowing us to encode & decode the metadata associated with the package, with strong type information (i.e. Uint64, String, etc.)

Unlocking Moss

Wait, what? Another blog post? In the same WEEK? Yeah totally doing that now. So, this is just another devlog update but there have been some interesting updates that we’d like to share.

Thanks to some awesome work from Peter, we now have LDC (The LLVM D Lang Compiler) present in the stage3 bootstrap. To simplify the process we use the official binary release of LDC to bootstrap LDC for Serpent OS.

In turn, this has allowed us to get to a point where we can now build moss and boulder within stage3. This is super important, as we’ll use the stage3 chroot and boulder to produce the binary packages that create stage4.

Some patching has taken place to prevent using ld.gold and instead use lld to integrate with our toolchain decisions.

A wild LDC appears

Cleanups Complete

Well, we’re officially back to working around the clock. After spending some time optimising my workflow, I’ve been busy getting the entire codebase cleaned up, so we can begin working towards an MVP.

Lots and lots of work

Since Friday, I’ve been working extensively on cleaning up the codebase for the following projects:

  • boulder
  • moss
  • moss-core
  • moss-format

As it now stands, 1 lint issue (a non-issue, really) exists across all 4 projects. A plethora of issues have been resolved, ranging from endian correctness in the format to correct idiomatic D-lang integration and module naming.

Granted, cleanups aren’t all that sexy. Peter has been updating many parts of the bootstrap project, introducing systemd 247.3, LLVM 11.0.1, etc. We now have all 3 stages building correctly again for the x86_64 target.

Build System Functional

Wow, has it been a hectic few weeks, and it definitely shows: last time we blogged it was about re-bootstrapping with glibc. Feels like ancient news already! So, what’s new in the world of Serpent OS? Apart from yours truly now being proud parent to a beautiful baby girl, work has resumed on the development of Moss, our package manager. And it builds stuff. Awesomely. Let’s quickly catch up with those updates and see where we’re headed next.

Look at all the buildiness

We’re now able to build the majority of a moss package. Notice I’ve made a distinction there. So, we’re able to run the build instructions, and use all the metadata, configuration and macros available. The only thing we’re not actually doing is dumping the built files into a binary package.

We’re able to do the build-system part rather well, now. Right now we support the following features in the build system:

  • Multiple, profile-based architecture support
  • Automatic -m32 profile based cross compilation support for 32-bit libraries on 64-bit OS
  • Basic Profile Guided Optimisation via the workload key. Set a workload, optimise accordingly.
  • LLVM Context Sensitive Profile Guided Optimisation. This is default for LLVM with the workload key, and comes for free, with multiple build stages.
  • Profile based tuning options, such as optimize for speed, size, disabling hardening, etc.
  • Trivially switch between gnu and llvm toolchain in stone.yml, with profiles knowing the right flags to use with tuning options.
  • Recursive macro support in build scripts, defined on per-profile level
  • Architecture specific profiles support in stone.yml, i.e. profiles -> aarch64 -> setup: to avoid if/else spaghetti.

Results Of The Experiment

It seems like only yesterday we announced to the world a Great Experiment. It was in fact 2 months ago, and a whole lot of work has happened since that point. A few take-homes are immediately clear, the primary one being the need to be a community-oriented Linux distribution.

To quote ourselves 2 months ago:

If the experiment is a success, which of course means having tight controls on scope and timescale, then one would assume the primary way to use Serpent OS would be through some downstream repackaging or reconfiguration, i.e. basing upon the project, to meet specific user demand.

It turns out that so far the experiment has been successful, and being forkable is still at the very heart of our goals. Others have joined us on our journey, and expressed the same passion in our goals as we have. A community has formed around the project, with individuals sharing the same ambitions for a reliable, rolling operating system with a powerful package manager.

Toolchain bootstrap

Source Format Defined

Following quickly on the heels of yesterday’s announcement that the binary format has been defined, we’ve now implemented the initial version of our source format. The source format provides metadata on a package along with instructions on how to build the package.

The next step of course, is to implement the build tool, converting the source specification into a binary package that the end user can install. With our 2 formats defined, we can now go ahead and implement the build routines.

Very trivial package recipe

The eagle-eyed among you will already see this is a derivation of the package.yml format I originally created while at Solus. Minor adaptations to the format have been made to support multiple architectures via the profiles key, and package splitting behaviour has now been grouped under a packages key to make the structure more readable.

In package.yml, one would have to redefine subpackage summaries as a key in a list of the primary summary key, such as:

rundeps:
- primary-run-dep
- dev: secondary-run-dep
summary:
- Some Summary
- dev: Some different summary

We’ve opted to group “Package Definition” behaviour into core structs, which are allowed to appear in the root-level package and subpackages:

summary: Top-level summary
packages:
- dev:
summary: Different summary
rundeps: Different rundeps

Moss Format Defined

The core team have been hard at work lately implementing the Moss package manager. We now have an initial version of the binary format that we’re happy with, so we thought we’d share a progress update with you.

Development work on moss

Briefly, the binary container format consists of 4 payloads:

  • Meta (Information on the package)
  • Content (a binary blob containing all files)
  • Index (indices to files within the binary blob)
  • Layout (How to apply the files to disk)

Each payload is verified internally using a CRC64-ISO, and contains basic information such as the length of the payload both compressed and uncompressed, the compression algorithm used (zstd and zlib supported) as well as the type and version of the payload. All multiple-byte values are stored in Big Endian order (i.e. Network Byte Order).

Payloads

Internally the representation of a Payload is defined as a 32-byte struct:

@autoEndian uint64_t length = 0; /* 8 bytes */
@autoEndian uint64_t size = 0; /* 8 bytes */
ubyte[8] crc64 = 0; /* CRC64-ISO */
@autoEndian uint32_t numRecords = 0; /* 4 bytes */
@autoEndian uint16_t payloadVersion = 0; /* 2 bytes */
PayloadType type = PayloadType.Unknown; /* 1 byte */
PayloadCompression compression = PayloadCompression.Unknown; /* 1 byte */

We merge all unique files in a package rootfs into the Content payload, and compress that using zstd. The offsets to each unique file (i.e. the sha256sum) are stored within the Index payload, allowing us to extract relevant portions from the “megablob” using copy_file_range().

These files will become part of the system hash store, allowing another level of deduplication between all system packages. Finally, we use the Layout payload to apply the layout of the package into a transactional rootfs. This will define paths, such as /usr/bin/nano, along with permissions, types, etc. All regular files will actually be created as hard links from the hash store, allowing deduplication and snapshots.

The Meta payload consists of a number of records, each with strongly defined types (such as String or Int64) along with the tag, i.e. Name or Summary. The entire format is binary to ensure greater resilience and a more compact representation. For example, each metadata key is only 8 bytes.

@autoEndian uint32_t length; /** 4 bytes per record length*/
@autoEndian RecordTag tag; /** 2 bytes for the tag */
RecordType type; /** 1 byte for the type */
ubyte[1] padding = 0;

Defining Moss

Over the past few weeks, throughout the entire bootstrap process, we’ve been deliberating on what our package manager is going to look like. We now have a very solid idea on what that’ll be, so it’s time for a blog post to share with the community.

Initial moss prototype CLI

The team has been very clear in wanting a traditional package management solution, whereby repositories and packages compose the OS as a whole. However, we also want modern features, such as being stateless by default. A common theme is wanting to reduce the complex iterations of an OS into something that is sanely versioned, but also flexible, to ensure a consistent, tested experience with multiple end targets.

Additionally, the OS must be incredibly easy for contributors and team members to maintain, with intelligent tooling and simple, but powerful formats.

One of the most recent software update trends of recent years is atomic updates. In essence this allows applying an update in a single operation, and importantly, reversing an update in a single operation, without impacting the running system.

This is typically achieved using a so-called A/B switch, which is what we will also do with moss. We won’t rely on any specific filesystem for this implementation, instead relying on a smart layout, pivot_root and a few other tricks.

Primarily we’ll update a single /usr symlink to point to the current OS image, with / being a read-only faux rootfs, populated with established mountpoints and symlinks. Mutation will be possible only via moss transactions, or in world-writable locations (/opt, /usr/local, /etc …)

Stage3 Complete

Another week, another milestone completed. We’re pleased to announce that we’ve completed the initial stage3 bootstrap. While we have some parallel works to complete for stage4, we can now begin to work on the exciting pieces! We’re now bootstrapped on ARMv8(-a) and x86_64

Build

Our immediate focus is now to implement our package manager: moss. Currently it only has a trivial CLI and does absolutely nothing. We will now shift our attention to implement this as a core part of the stage4 bootstrap. Moss is so-called as we’re a rolling release, a rolling stone gathers no moss. The package manager is being implemented in the D Programming Language1:.

Moss does not aim to be a next generation package management solution, it instead inspired by eopkg, RPM and swupd, aiming to provide a reliable modern package management and update solution with stateless design at the core. Core features and automatic dependencies will be managed through capability subscriptions. The binary format will be versioned and deduplicated, with multiple internal lookup tables and checksums. Our chief focus with moss is a reliable system management tool that is accessible even in situations where internet access is limited.

In order to complete stage3, we provided linker stubs in libwildebeest to ensure we can build. Obviously this introduces runtime errors in systemd and our stage3 isn’t bootable, just chrootable. This will be resolved when systemd is properly packaged by moss in stage4.

Looking Stage4 In The Eye

Well, we’ve made an awful lot of progress in these last few days. It wasn’t that long ago that we introduced some of the new projects required to get stage3 off the ground.

Building systemd the easy way

Our libc-support project has been growing, thanks primarily to the contributions of Jouni Roivas. We now have initial working versions of getent and getconf. The getconf program is considered feature-complete for our current requirements, and the focus is now on cleaning up getent making it more modular and easy to maintain in the long run.

Stage 3 Progress

Well, it’s been a few days since we last spoke, so now it’s time for a quick roundup. Long story short, we’re approaching the end of the stage3 bootstrap.

Fully functional chroot

In an effort to simplify our bootstrap process, we dropped the newly-introduced stage2.5 and came up with a new strategy for stage3. In order to make it all work nicely, we bind-mount the stage2 resulting runtime at /serpent within the stage3 chroot environment, executing the /serpent/usr/bin/bash shell.

In order to make this work, we build an intermediate musl package for libc.so in the very start of stage3, with all subsequent builds being performed in chroots. Part of the build is done on the host, i.e. extraction and patching, minimising the tool requirements for the chroot environment. The configuration, build and install is performed from within the initially empty chroot environment, replacing all the /serpent/usr/bin tools and /serpent/usr/lib libraries.

As we move further through stage3, towards a fully usable chroot environment, we’ve encountered a small number of blockers. Now, we could solve them by using existing patchwork and workarounds, but most have not and will not be accepted upstream. Additionally it is incredibly hard to track the origin and history of most of these, making security rather more painful.

We’re going to start working on a project to flesh out the musl runtime with some missing utilities, written with a clean-room approach. These will initially include the getconf and getent tools, which will be written only with Linux in mind, and no legacy/BSD support.

These will be maintained over at our GitHub

Stage2 Complete

Just in case you thought we were sleeping behind the wheel, we’ve got another blogpost for your viewing pleasure. In a nutshell, we completed stage2 bootstrap.

Complete build-target for ARMv8

In order to simplify life, we greatly reduced the size of the stage2 build component. This decision was taken to better support cross-compilation in the face of software that is distinctly cross-compilation unfriendly.

A support stage, stage2.5 will be added which will chroot into a copy of stage2, and natively compile a small handful of packages required to complete stage3, also within the chroot environment.

For cross-compilation, we’ll be relying on qemu-static to complete 2.5 and 3. However, at this point in time, we have the following:

  • Working cross-compilation of the entire bootstrap
  • Complete LLVM based toolchain: clang, llvm, libc++, lib++abi, libunwind
  • Entirety of stage2 built with musl libc.
  • Working, minimal, chroot environment as product of stage2, with working compiler (C & C++)

x86_64

This is a major milestone for the project, as it is an early indication that we’re self hosting.

Stage1 Complete

Short and sweet, stage1 of the bootstrap is complete. As I indicated on the Lispy Snake1 blog, I’m still in the process of settling into new accommodation. This is going well, but still awaiting proper broadband connectivity. Work has begun, however, and we’re now moving onto stage2 of the bootstrap.

This is handled via our bootstrap-scripts2 project and can be run by anyone on a relatively modern Linux distribution.

Validating stage1 cross-compiler

Next on the list is completing stage2, which we’ve already started on. This is simply a cross-compiled chroot environment with all the basic bits in place to build stage3, sanitizing and cleansing the toolchain.

Even though it is early days, we can already, automatically produce a working cross-compiler that targets LLVM’s libc++, the musl libc.so, and x86_64-serpent-linux-musl host triplet.

Probably a dull update for most, but an update it is.


  1. Since this blog post, the Lipsy Snake website no longer exists. ↩

  2. Since this blog post, the bootstrap-scripts project no longer exists. ↩

Welcome: Project Manager

The Serpent OS team wishes to formally welcome Aydemir Ulaş Şahin to the core team in the primary capacity of Project Manager. Aydemir will help us to coordinate and manage the project goals and infrastructure, being responsible for certain “keys to the kingdom”.

Aydemir has extensive experience with Linux, both as a user and developer, having worked in the media industry for many years. Additionally he will be contributing to the core system, code and packaging.

Our initial trio has been announced, and you can always access information on the current team at the new About page. Now that much of our groundwork has been laid, we can get on with building the project. Check progress at the new Roadmap1 page.


  1. Since this blog post, we have removed our Roadmap page. ↩

Welcome Peter

Recently we did reveal on Twitter that long time friend and colleague-of-many-projects, Peter O’Connor, has formally joined the core team for Serpent OS. Many of you know him as sunnyflunk. He will be working on core toolchain optimisations, benchmarking, and eventually Plasma inclusion.

Peter has long wanted to get the most out of his computing experience, starting from Gentoo back in the 2000’s to ensure there was no wasted resources on features he didn’t use. The desire to over-tinker and use unstable software often led to poor outcomes. While the itch laid dormant for over a decade, it has always been there.

Peter O’Connor (sunnyflunk)

There’s not much about Linux that currently excites me, but I’ve always wanted to see what’s possible with a full LLVM/Clang stack. There’s never been a better time to find out, but it can be difficult to push the envelope when limited with backwards compatibility and supporting older CPU architectures. After experimenting with some tool-chain customisations the chance to test them out more broadly and with a new tool-chain was too hard to pass up. This is the research project I’ve always wanted to undertake and looking forward to using it as my daily driver

We’d like to officially extend our welcome to Peter, as the ramp up gets in place. It should be noted, he is partly to blame for Serpent OS becoming manifest, as the core team got together after discussions on IRC1. He has some fantastic ideas, and is a brilliant distribution engineer with proven success.

As an aside, we’re working on improving the website to incorporate a team page, ready for the third team member reveal.


  1. aerynOS now uses a Zulip server for community discussion. ↩

The Great Experiment

So as many have come to realise, we had to rush out a website super quick yesterday as the cat was already out of the bag. One thing that should also be clarified, is our approach to development.

Primarily (but not exclusively) Serpent OS (future: Serpent Linux) is an endeavour worked on by the Lispy Snake, Ltd1 crew. So we’re all about creating awesome technology and trying to further the playing field.

Serpent OS is not, however, owned by Lispy Snake. It is a contributor-based open source project, with specific roles. Those will be clarified over the coming week, but there is a dedicated Project Manager. Hint: It’s not me.

So, how does an experimental project… measure success? Quite simply - if others adopt our methodologies or technology, we’ve succeeded. We’re really not looking at “number of installs” or “size of userbase” as a metric, as our chief aim is to build technology.

If the experiment is a success, which of course means having tight controls on scope and timescale, then one would assume the primary way to use Serpent OS would be through some downstream repackaging or reconfiguration, i.e. basing upon the project, to meet specific user demand.

This is the heart and soul of all Linux development. Developers should enjoy the involvement and seeing the end result. If we build something more in the end, then that’s a journey we can create together.


  1. Since this blog post, the Lipsy Snake website no longer exists. ↩

First Post

In a nutshell, this post wasn’t meant to be coming out for a few weeks. As it turns out, we had to get ahead of the cat coming out of the bag, and ensure clarity of goals and ambition.

We’ve got an awful lot of work ahead of us across the next few months and weeks, and we strongly suggest checking out the About page which should answer most questions.

TL;DR This is not your typical user-facing Linux distribution.

So go ahead, check out the About page.

Posts will follow in the coming weeks, introducing the core team. It should be pointed out, internally, we’re in a stage1 bootstrap phase, with stage2 ready. Tooling is next on the agenda.

If you’re looking for a modern, lightweight, user-friendly, privacy-focused Linux desktop distribution, then you’re in the wrong place.

Otherwise, stay tuned for the updates. We know our website is insanely wbasic, but it will be improved when it becomes a priority. You’re more than welcome to join us on #serpentOS on freenode1 to become part of the community discussion in the meantime.

Til next time.


  1. aerynOS now uses a Zulip server for community discussion. ↩