Skip to content

Ikey

44 posts by Ikey

🏗️ AerynOS: The OS As Infrastructure

Taking a break from our usual release-oriented updates, I thought it was high time to dive into what AerynOS actually is and what sets it apart from other distros. Fair warning, this is a meaty, in depth post, and still only scratches the surface. If you’re interested in the future of Linux distributions, read on. It may also help to visit our work-in-progress documentation site for more information.

Firstly, AerynOS isn’t Yet Another Linux Distribution. It’s a platform, a foundation, and a set of tools engineered in accordance with a vision and design that just so happens to produce a Linux distribution. Were we to go back in time and build a design brief for AerynOS, the initial question might be:

What if the operating system itself behaved like modern infrastructure?

AerynOS is the answer to that question. What if, instead of doing things the way they’ve always been done, we started from the ground up and produced a well designed system, rather than the traditional model of in-place mutation internal to a distribution?

Leveraging years of experience across the industry, including some notable pioneering projects such as Solus, Clear Linux, and others, we set out to build things the hard way, in order to make them easy.. eventually.

At the very core, is the decision to not be another GNU/Linux distribution. We default to the LLVM toolchain, using libc++ and compiler-rt by default. This isn’t just a case of “we like LLVM”, but rather a strategic decision to leverage the superior diagnostics, and ensure correctness / portability of packages. Whilst in our very early stages a few years ago we did trial musl, we do default to glibc for compatibility reasons and its superior performance characteristics. The performance advantage of glibc over musl is well-documented, particularly for compute-intensive workloads and applications requiring optimal threading performance. We’re here to build a working, usable system, for a multitude of use cases and verticals, and glibc provides the best balance of compatibility and performance.

Note, we do still package gcc and it’s trivial to enforce its use in a package recipe by setting the toolchain field in stone.yaml to gnu.

Lastly, it’s worth noting we build all packages for x86_64-v2 as our minimum baseline, and perform targeted optimisation in our package recipes using the extensive tuning options configurable in boulder.

🚀 Hello AerynOS

Hello from AerynOS! ✨ As you recall from February, we set about rebranding Serpent OS into what you now see, AerynOS. It’s not been without challenges, and where possible we’ve ensured continuity. Additionally we silently dropped a few ISOs, and can now take our first formal steps under our new identity. TLDR: the transition has been entirely fluid and you only need to keep updating, no manual intervention is necessary! 🎉

Progress has been steady but quiet lately, as due to unfortunate circumstances I’ve been working in bursts from my wifes hospital bedside.

OK, lets get right to it. We’ve released AerynOS 2025.03, with the following shinies:

  • 🖥️ GNOME 48.0
  • 🐧 Linux 6.13.8
  • 🦊 Firefox 136.0.2
  • 🎮 Mesa 25.0.2
  • 🚀 Vulkan SDK 1.4.309.0
  • 🛠️ LLVM 19.1.7

We’ve also decoupled the internal tooling version from the ISO versions to make them a little bit more.. well.. readable, for humans.

Grab it now from the download page.

Evolve This OS

A long overdue, and highly requested update today. We’re finally rebranding the project!

The “Serpent OS” name was a quickly chosen name that stuck. Unfortunately “serpents” are often associated with negative connotations, and we’ve had a lot of feedback over the years that the name was off-putting. Let’s be completely honest, it’s not the most inviting name for a project. Who wants to trust a serpent? Generally speaking they’re considered dangerous at best.

It’s fair to say we’ve spent a long time in prototype and alpha phases. In order to move forward, our identity needs to be more befitting of the project we’re building. A move into the real world. This isn’t a hobby project, it’s a full blown Linux distribution with serious technical underpinnings, achievements and goals. Getting the tone right from day dot is critical.

Do note, there is no change to the internal core team, so we’re aiming for full continuity and a seamless transition. Had enough of these snakes on this plane? We have too.

Hah, it’s not “Evolve OS”, despite the clickbait title. Our new name is… drum roll please…

Hello 2025

TL;DR: Serpent OS is facing funding challenges but development continues. Alpha2 is coming soon with an improved installer. We’re seeking community support through donations and volunteers for key roles. Our technical roadmap includes versioned repositories, immutable OS features, and improved package management workflows.


We had a flurry of activity around the Christmas period, including our first alpha release as well as enabling offline rollbacks early in January. We’re actively working on alpha2, but we also need to talk about the elephant in the room.

Recently I posted a tweet that has been covered by a few news outlets. The long and short is quite simple: distinct lack of funds. A very long story short: a change in my contract led to a loss of income protection, combined with a herniated disc (C6-C7), culminated in having to resign. Naturally I’ve been working exclusively on Serpent OS since then, but I’ve also got to keep the lights on at home.

Broke

This doesn’t mean the project is closing or anything like that, but it does mean that I actively need to keep costs down whilst using daytime hours for financing. This is what’s led to the significant slowdown in the last few weeks, and I’m just trying to be as transparent as possible and manage expectations.

Offline Rollbacks Enabled

If a picture tells a thousand words… well then this video .. Yknow what, sod it. Here’s a video of the offline rollback support in action, as tested in a newly installed VM using xfs for the root filesystem. We jokingly refer to this as the “LTT” test due to LinusTechTips’ “interesting encounters” with package management systems in the past.

This has already landed in our volatile repository, so simply sudo moss sync -u and grab the latest updates. Any subsequent transactions will automatically generate the boot entries for you, no intervention required. Or mounting. Or anything. Just update and go. Srsly.

We actually saw this as a golden standard for the feature, as it’s a real-world scenario where you’d want to roll back to a known good state. The video shows the system booting and recovering in multiple ways, from complete system nuke (via glibc) and simpler scenarios like removal of the GNOME desktop environment.

The intent is to ensure a user can quickly revert to the last transaction that worked in the instance an update goes awry, without needing to boot into a rescue environment or live CD.

Investing in the Future

Here we are, at the end of 2024, and what a year it’s been! We’ve made significant progress on Serpent OS, and we’re excited to share some of the highlights with you.

Virtually 5 years in the making, we recently attained alpha status. Our tooling and concepts have aligned, allowing us to now rapidly iterate on the core deliverable itself: Serpent OS. We’ve seen an explosion in cadence, with our tooling enabling us to quickly and easily deliver updates, new packages and enabling new features.

Since the Alpha 1 ISO (0.24.5), we’ve fixed a number of issues in relation to hardware and booting. We’ve also added new features to moss (refined boot management) and boulder (our build system). Even now we have multiple PRs open to tidy up moss, add automatic generation of monitoring.yaml files, and more.

We’ve been working hard to keep the cadence up, and we’re now at a point where we can deliver new ISOs on a regular basis. Our main focus now is the tooling, in enabling a far more automatic delivery pipeline. The vision is having automated pull requests generated for package updates, automatically handling changes in dependencies and ABI, and ordered by build tier. This will significantly reduce the maintainer burden, in conjunction with planned security monitoring.

Alpha Refresh Available

In response to feedback from the community, we’re issuing a refresh of the Alpha 1 release, 0.24.6. This refresh includes a number of key updates and improvements, including:

  • Now works with etcher
  • Fractional scaling enabled by default
  • Prebuild icon theme caches to speed up trigger execution (except for hicolor)
  • Add new moss boot status command for introspection into blsforme functionality
  • Fix amdgpu initialisation for older devices, previously leading to black screen on boot

Fractional scaling, much loved

Serpent OS Enters Alpha

Following a successful prealpha phase, we’re excited to announce that Serpent OS has now entered the alpha stage of development. This milestone marks a significant step forward in the project’s journey. While the usual disclaimers apply, as a certain level of project fluidity is to be expected, we invite you to explore the latest alpha release and experience the cutting-edge features and improvements we’ve introduced.

Screenshot of a Gnome desktop showing files open

Serpent OS is a community-driven project, and we welcome contributions from developers, testers, and users alike. If you’re interested in helping shape the future of Linux distributions, we invite you to join our growing community and get involved in the project. Whether you’re interested in packaging, development, documentation, testing, or community support, there are many ways to contribute to Serpent OS.

Lastly, you can support the project by becoming a sponsor. Your support helps cover our hosting costs and allows us to dedicate more time to making Serpent OS exceptional. We’re grateful for the support we’ve received so far and look forward to continuing to build something meaningful in the Linux ecosystem.

Serpent OS Prealpha0 Released

Well, it didn’t take us that long, really … Our technical preview, prealpha0, is now available for testing!

A live GNOME environment running on prealpha0 of SerpentOS

Head to our download page to grab serpent-os-prealpha-0.iso now!

This is a super rough version of Serpent OS that is capable of being installed on baremetal hardware and VMs that support UEFI and OpenGL acceleration. It is however not recommended for use daily due to the early nature and a bunch of packages being wildly out of date. It features a minimal GNOME desktop with the Firefox web browser and a terminal.

Right now it contains a CLI installer that can be accessed from the terminal by typing:

Terminal window
sudo lichen

To detract from casual use it is necessary to manually partition the disk before running the (net) installer. You can use fdisk to create a GPT disk with a (mandatory) EFI System Partition and an optional XBOOTLDR partition. This currently also needs to be FAT32 until we integrate systemd-boot driver support. Lastly of course you will need a large enough root partition.

You will need a working network connection, so make sure you’re connected before starting the installer!

Calm Before The Storm

It’s that time of month again, and we have some details to share with you on boot management, as well as plans for real installs landing in May. Additionally, plans for “deferred updates” via mandatory reboots have been dropped!

wip boot code

To better understand the technical deep-dive on boot management further below, we should clarify a slight change in priorities for the project as a whole. For some time now we’ve served as a technical beacon for Solus to guide them towards better strategy and technical solutions whilst awaiting a situation where a rebase is feasible.

While we still look forwards to that future, the present day is the most important to us. Serpent OS must now transition from a collection of tools and promises into a daily driver with actual users. We still aim to cater to the more developer/technical oriented end of the spectrum rather than a “desktop distro”, and want to start this journey yesterday.

The boot management code is currently landing over the coming days, and will be immediately followed by an incredibly rudimentary system installer to facilitate an initial adoption of Serpent OS for the general public. Our intent is to ship some early test ISOs for supporters, and by the end of May have a feasible option for people to use and contribute to Serpent OS on real hardware.

We acknowledge this is an exchange of the pursuit of perfection for a pragmatic rapid iteration cycle and believe the best way forward is to open the doors wide, fixing issues on the job. The initial system will likely be rough and buggy, but it is an official starting point from which we can all build Serpent OS together.

The March Of Progress

Despite a brief excursion out of the country for a first-in-a-lifetime holiday, I’m happily back at the desk to bring you up to date with the latest goings in Serpent OS. TLDR: Loads of awesome, baremetal is enabled, ISO cycle in next couple of weeks.

GNOME on Serpent OS

OK, we all know you want to know about the baremetal stuff, but c’mon, check out this changeset!

End of February Update

This update came a little later in the month, as we’ve got a lot of exciting news to share. Everything from boulder in Rust, to the GNOME 45 Desktop complete with moss triggers built atop a rebootstrapped toolchain.

We’re pleased to announce that over the course of this weekend, once testing has completed we’ll deploy the latest version of boulder, our packaging build tool. This has been given the Rust treatment, directly sharing the codebase with moss.

Boulder in action

Unlike our PoC implementation, the new boulder makes use of the clone syscall to execute portions of itself under user namespaces, eliminating a longstanding deployment irritation of having the main build binary inside packaging.

Just like moss it makes use of tokio for async right where we need it, and is rootless to make life easier for our contributors. Far from being just a direct port of the proof of concept, it adds new tools to empower developers, such as breakpoints in packaging recipes.

Breakpoints, in recipes

January Updates

Precisely one month since our end of year summary, so, what have we been up to? With a new year, a new start was long overdue. We’re pleased to announce that we’ve finally shifted our butts over to our own server (again).

While this isn’t quite as chunky a news update as the last post, we rather hope you appreciate the regularity of the news.

moss in action

‘tarkah’ has been tirelessly porting moss-service, the core foundation of our automated build system, to a shiny new Rust codebase. While we don’t have an immediately pressing need for the port to come online, it will certainly address some “v1 issues” and ensure the best-in-class development experience for our contributors.

In our pursuit for a sustainable, community oriented project, we’ve been doing a lot of in house organisation. Very importantly, we’ve now relicensed our packaging recipes to MPL-2.0, bringing them in line with the license chosen for our current engineering efforts (moss, etc).

To reiterate a critical point, we require that our contributions are attributed to the virtual collective “Serpent OS Developers” to ensure we’re unable to relicense any contributions without express consent of the contributors involved.

Additionally we picked MPL-2.0 over Zlib due to some distinct advantages in relation to patent trolls. In combination we believe both the project and our users will have the greatest protection from patent trolls and fiscally-motivated relicensing.

End Of Year Summary

Burning question - how long before we can use Serpent OS on our development systems? It’s a fair question - and one that requires context to answer. Let’s start our new monthly update cycle with a recap of progress so far.

screenshot of moss

We’d like to extend a huge official welcome, and thank you, to newest team member Cory Forsstrom (aka tarkah)!

Firstly, a quote from Cory:

I’ve been a long time Solus user and was very excited about what serpent os was doing. Really got invested once I started diving into the D code and seeing how powerful the tools and ideas were. The Rust rewrite was just a perfect storm for me with my experience there and my desire to make contributions. Getting to know you and ermo has just been icing on the cake, you’ve both been so welcoming and friendly. So lots of fun times.

On the personal side, I’m on the west coast in the States, have a lovely wife, just had a baby girl and am enjoying my time with fatherhood and coding

Chances are you know tarkah for his contributions to iced-rs, and to Solus. His contributions to the moss-rs project have been absolutely critical, and his patience essential as we got ourselves quickly up to speed with Rust. It is entirely fair to say that our Rust efforts would not have been possible without him!

Oxidised Moss

Allow me to start this post by stating something very important to me: I absolutely love the D programming language, along with the expressivity and creative freedom it brings. Therefore please do not interpret this post as an assassination piece.

For a number of months now the Serpent OS project has stood rather still. While this could naively be attributed to our shared plans with Solus - a deeper, technical issue is to be acredited.

img

D isn’t quite there yet. But it will be, some day.

Section titled “D isn’t quite there yet. But it will be, some day.”

Again, allow me to say I have thoroughly enjoyed my experience with D over the last 3 or so years, it has been truly illuminating for me as an engineer. With that said, we have also become responsible for an awful lot of code. As an engineering-led distribution + tooling project, our focus is that of secure and auditable code paths. To that extent we pursued DIP1000 as far as practical and admit it has a way to go before addressing our immediate needs of memory safety.

While we’re quite happy to be an upstream for Linux distributions by way of release channels and tooling releases, we don’t quite have the resources to also be an upstream for the numerous D packages we’d need to create and maintain to get our works over the finish line.

With that said, I will still continue to use D in my own personal projects, and firmly believe that one day D will realise its true potential.

Snakes On A Boat

We had intended to get a blog post out a little bit quicker, but the last month has been extremely action packed. However, it has paid off immensely. As our friends at Solus recently announced it is time to embark on a new voyage.

dashboard preview

There is very little point in rehashing the post, so I’ll cover the essentials

  • We assisted the Solus team in deploying a complete new infrastructure. Some of this work is ongoing
  • The package delivery pipeline is in place, meaning updates will soon reach users.
  • We’re working towards a mutually beneficial future whereby Solus adopts our tooling.

Oh no, no. The most practical angle to view this from is that Serpent OS is free to build and innovate to create the basis of Solus 5. Outside of the distribution we’re still keen to continue development on our tooling and strategies.

It is therefore critical that we continue development, and find the best approach for both projects, to make the transition to Solus 5 as seamless as possible. To that end we will still need to produce ISOs and have an active community of users and testers. During this transition period, despite being two separate projects, we’re both heading to a common goal and interest.

Infrastructure Launched!

After many months and much work, our infrastructure is finally online1. We’ve had a few restarts, but it’s now running fully online with 2 builders, ready to serve builds around the clock.

Firstly, I’d like to apologise for the delay since our last blog post. We made the decision to move this website to static content, which took longer than expected. We’re still using our own D codebase, but prebuilding the site (and fake “API”) so we can lower the load on our web server.

Our infrastructure is composed of 3 main components.

First look at summit

This is the page you can see over at dash.serpentos.com1. It contains the build scheduler. It monitors our git repositories, and as soon as it discovers any missing builds it creates build tasks for them. It uses a graph to ensure parallel builds happen as much as possible, and correctly orders (and blocks) builds based on build dependencies.

We have 2 instances of Avalanche, our builder tool, running. This accepts configuration + build requests from Summit, reporting status and available builds.

This is an internal repository manager, which accepts completed packages from an Avalanche instance. Published packages land at build.aerynos.dev2

Super early days with the infra but we now have builds flowing as part of our continuous delivery solution. Keeping this blog post short and sweet… we’re about to package GNOME and start racing towards our first real ISOs!

If you like what the project is doing, don’t forget to sponsor!


  1. The infrastructure used to be hosted at dash.serpentos.com but has since moved to our aerynOS domain. ↩ ↩2

  2. Vessel used to be hosted at dev.serpentos.com but has since moved to our aerynOS domain. ↩

Lift Off

Enough of this “2 years” nonsense. We’re finally ready for lift off. It is with immense pleasure we can finally announce that Serpent OS has transitioned from a promise to a deliverable. Bye bye, phantomware!

A booted Serpent OS

As mentioned, we spent 2 years working on tooling and process. That’s .. well. Kinda dull, honestly. You’re not here for the tooling, you’re here for the OS. To that end I made a decision to accelerate development of the actual Linux distro - and shift development of tooling into a parallel effort.

I deferred final enabling of the infrastructure until January to rectify the chicken/egg scenario whilst allowing us to grow a base of contributors and an actual distro to work with. We’re in a good position with minimal blockers so no concern there.

The Big Update

Well - we’ve got some big news! The past few weeks have been an incredibly busy time for us, and we’ve hit some major milestones.

After much deliberation - we’ve decided to pull out of Open Collective1. Among other reasons, the fees are simply too high and severely impact the funds available to us. In our early stages, the team consensus is that funds generated are used to compensate my time working on Serpent OS.

As such I’m now moving funding to my own GitHub Sponsors page1 - please do migrate! It ensures your entire donation makes it and keeps the lights on for longer =) Please remember I’m working full time on Serpent OS exclusively - I need your help to keep working.

We’ve pretty much completed our transition to GitHub. We’ve now got the following organisations:

Don’t forget - our forums are live over at forums.serpentos.com2 - please feel free to drop in and join in with the community 😀

OK so what exactly are moss and boulder? In short - they’re the absolute core pieces of our distribution model.

On the surface, moss looks and feels roughly the same as just about any other traditional package manager out there. Internally, however, its far more modern and has a few tricks up its sleeve. For instance, every time you initiate an operation in moss, be it installation, removal, upgrade, etc, a new filesystem transaction is generated. In short, if something is wrong with the new transaction - you can just boot to an older transaction when things worked fine.

Now, it’s not implemented using bundles or filesystem specific functionality, internally its just intelligent use of hardlinks and deduplication policies, and we have our own container format with zstd based payload compression. Our strongly typed, deduplicating binary format is what powers moss.

Behind the scenes we also use some other cool technology, such as LMDB for super reliable and speedy database access. The net result is a next generation package management solution that offers all the benefits of traditional package managers (i.e. granularity and composition) with new world features, like atomic updates, deduplication, and repository snapshots.

It’s one thing to manage and install packages, it’s another entirely to build them. boulder builds conceptually on prior art such as pisi and the package.yml format used in ypkg. It is designed with automation and ease of integration in mind, i.e. less time spent focusing on packaging and more time on actually getting the thing building and installing correctly.

Boulder supports “macros” as seen in the RPM and ypkg world, to support consistent builds and integration. Additionally it automatically splits up packages into the appropriate subpackages, and automatically scans for binary, pkgconfig, perl and other dependencies during source analysis and build time. The end result is some .stone binary packages and a build manifest, which we use to flesh out our source package index.

We’ve spent considerable time reworking moss, our package manager. It now features a fresher (terminal) user interface, progress bars, and is rewritten to use the moss-db module encapsulating LMDB.

nspawn

It’s also possible to manipulate the binary collections (software repositories) used by moss now. Note we’re going to rename “remote” to “collection” for consistency.

At the time of writing:

Terminal window
$ mkdir destdir
$ sudo moss remote add -D destdir protosnek https://dev.serpentos.com/protosnek/x86_64/stone.index
$ sudo moss install -D destdir bash dash dbus dbus-broker systemd coreutils util-linux which moss nano
$ sudo systemd-nspawn -b -D destdir

This will be simplified once we introduce virtual packages (Coming Soon ™)

Boulder can now be instructed to utilise a local collection of stone packages, simplifying the development of large stack items.

Terminal window
sudo boulder bi stone.yml -p local-x86_64

Packages should be moved to /var/cache/boulder/collections/local-x86_64 and the index can be updated by running:

Terminal window
sudo moss idx /var/cache/boulder/collections/local-x86_64

Serpent OS is now officially self hosting. Using our own packages, we’re able to construct a root filesystem, then within that rootfs container we can use our own build tooling (boulder) to construct new builds of our packages in a nested container.

The protosnek collection has been updated to include the newest versions of moss and boulder.

self hosting

As a fun experiment, we wanted to see how far along things are. Using a throwaway kernel + initrd build, we were able to get Serpent OS booting using virtualisation (qemu)

booting

Right now everyone is working in the snekpit organisation to get our base packaging in order. I’m looking to freeze protosnek, our bootstrap collection, at the latest of tomorrow evening.

We now support layered, priority based collections (repositories) and dependency solving across collection boundaries, allowing us to build our new main collection with protosnek acting as a bootstrap seed.

Throughout this week, I’ll focus on getting Avalanche, Summit and Vessel into shape for PoC so that we can enjoy automated builds of packages landing in the yet-to-be-launched volatile collection.

From there, we’re going to iterate + improve packaging, fixing bugs and landing features as we discover the issues. Initially we’ll look to integrate triggers in a stateless-friendly fashion (our packages can only ship /usr by design) - after that will come boot management.

An early target will be Qemu support via a stripped linux-kvm package to accelerate the bring up, and we encourage everyone to join in the testing. We’re self hosting, we know how to boot, and now we’re able to bring the awesome.

I cannot stress how important the support to the project is. Without it - I’m unable to work full time on the project. Please consider supporting my development work via GitHub Sponsors1.

I’m so broke they’ve started naming black holes after my IBAN.

Thank you!

You can discuss this blog post over on forums2.


  1. Since this blog post, we no longer use OpenCollective or GitHub Sponsors. For funding, please see our Funding Page and for ISOs, please see our Download Page. ↩ ↩2 ↩3

  2. Since this blog post, we now use Discourse as our forum. ↩ ↩2

Infrastructure Update

Since the last post, I’ve pivoted to full time work on Serpent OS, which is made all the more possible thanks to everyone supporting us via OpenCollective1 ❤️.

We’ve been working towards establishing an online infrastructure to support the automation of package builds, while revisiting some core components.

During the development of the Serpent OS tooling we’ve been exploring the possibilities of D Lang, picking up new practices and refining our approach as we go. Naturally, some of our older modules are somewhat … smelly. Most noticeable is our moss-db module, which was initially intended as a lightweight wrapper around RocksDB.

In practice that required an encapsulation API written in D around the C API, and our own wrapping on top of that. Naturally, it resulted in a very allocation-heavy implementation that just didn’t sit right with us, and due to the absurd complexity of RocksDB was still missing quite a few features.

Packaging Automation, Next Steps

Hot damn we’ve been busy lately. No, really. The latest development cycle saw us focus exclusively on boulder, our build tooling. As of today it features a proof of concept boulder new subcommand for the automatic generation of packaging templates from an upstream source (i.e. tarball).

Before we really start this blog post off, I’d like to thank everyone who is supporting the project! All of the OpenCollective1 contributions will make it easier for me to work full time on Serpent OS 😀 Much love ❤️

Look at all the buildiness

Alright you got me there, certain projects prefer to abstract the configuration, build and installation of packages and be provided with some kind of hint to the build system, i.e. manually setting autotools, etc.

Serpent OS packaging is declarative and well structured, and relies on the use of RPM-style “macros” for distribution integration and common tasks to ensure consistent packaging.

We prefer a self documenting approach that can be machine validated rather than depending on introspection at the time of build. Our stone.yml format is very flexible and powerful, with automatic runtime dependencies and package splitting as standard.

..Doesn’t mean we can’t make packaging even easier though.

A Word From The Founder

Well well, it’s been a long time since I personally wrote a post.. 😀 So let’s keep this short and sweet, shall we? I’m returning to full time work on Serpent OS.

The 6th of July will be my last day at my current employment having tendered my 30 day notice today. Despite having enjoyment at my current position, the reality is that my passion and focus is Serpent OS.

I’m now in a transition process and will ramp up my efforts with Serpent OS. Realistically I need to reduce the outgoing costs of the project and with your help I can gain some level of financial support as we move through the next stages of development. Worst case, I will only take on any part-time or contractual gigs, allowing my primary focus to be Serpent OS.

I’ll begin accelerating works and enabling community contribution so we can get the derailed-alpha train back on the tracks.

I have absolute faith in this project, the community and our shared ability to deliver the OS and tooling. To achieve it will require far more of my time and I’m perfectly willing to give it.

Thank you all to everyone who has been supporting the project, it is now time to deliver. Not just another run of the mill distribution but a technically competent and usable distribution that is not only different but better.

Let’s do this in the most grassroots and enjoyable way possible 😊

It All Depends

It all depends.. it really does. On shared libraries.. interpreters.. pkg-config providers and packages. It’s the same story for all “package managers”, how do we ensure that the installed software has everything it needs at runtime?

Our merry crew has been involved in designing and building Linux distributions for a very, very long time, so we didn’t want to simply repeat history.

Using updated moss

Thanks to many improvements across our codebase, including moss-deps, we automatically analyse the assets in a package (rapidly too) to determine any dependencies we can add without requiring the maintainer to list them. This is similar in nature to RPM’s behaviour.

As such we encode dependencies into our (endian-aware, binary) format which is then stored in the local installation database. Global providers are keyed for quick access, and the vast majority of packages will not explicitly depend on another package’s name, rather, they’ll depend on a capability or provider. For subpackage dependencies that usually depend on “NEVRA” equality (i.e. matching against a name with a minimum version, release, matching epoch and architecture), we’ll introduce a lockstep dependency that can only be resolved from its origin source (repo).

Lastly, we’ll always ensure there is no possibility for “partial update” woes. With these considerations, we have no need to support >= style dependencies, and instead rely on strict design goals and maintainer responsibility.

A Rolling Boulder Gathers No Moss

We actually did it. Super pleased to announce that moss is now capable of installing and removing packages. Granted, super rough, but gotta start somewhere right?

Transactional system roots + installation in moss

OK let’s recap. A moss archive is super weird, and consists of multiple containers, or payloads. We use a strongly typed binary format, per-payload compression (Currently zstd), and don’t store files in a typical archive fashion.

Instead a .stone file (moss archive) has a Content Payload, which is a compressed “megablob” of all the unique files in a given package. The various files contained within that “megablob” are described in an IndexPayload, which simply contains some IDs and offsets, acting much like a lookup table.

That data alone doesn’t actually tell us where files go on the filesystem when installed. For that, we have a specialist Layout Payload, encoding the final layout of the package on disk.

As can be imagined, the weirdness made it quite difficult to install in a trivial fashion.

Well, persistence really. Thanks to RocksDB and our new moss-db project, we can trivially store information we need from each package we “precache”. Primarily, we store full system states within our new StateDB, which at present is simply a series of package ID selections grouped by a unique 64-bit integer.

Additionally, we remember the layouts within the LayoutDB so that we can eventually recreate said layout on disk.

Moss DB Progress

I’ll try to make this update as brief as I can but it’s certainly an important one, so let’s dive right into it. The last few weeks have been rough but work on our package manager has still been happening. Today we’re happy to reveal another element of the equation: moss-db.

Putting moss-db to the test

moss-db is an abstract API providing access to simplistic “Key Value” stores. We had initially used some payload based files as databases but that introduced various hurdles, so we decided to take a more abstract approach to not tie ourselves to any specific implementation of a database.

Our main goal with moss-db is to encapsulate the RocksDB library, providing sane, idiomatic access to a key value store.

At the highest level, we needed something that could store arbitrary keys and values, grouped by some kind of common key (commonly known as “buckets”). We’ve succeeded in that abstraction, which also required us to fork a rocksdb-binding to add the Transform APIs required.

Additionally, we required idiomatic range behaviours for iteration, as well as generic access patterns. To that affect we can now foreach a bucket, pipe it through the awesomely powerful std.algorithm APIs, and automatically encode/decode keys and values through our generic APIs when implementing the mossdbEncode() and mossdbDecode() functions for a specific type.

In a nutshell, this was the old, ugly, hard way:

/* old, hard way */
auto nameZ = name.toStringz();
int age = 100;
ubyte[int.sizeof] ageEncoded = nativeToBigEndian(ageEncoded);
db.setDatum(cast(ubyte[]) (nameZ[0 .. strlen(nameZ)]), ageEncoded);

And this is the new, shmexy way:

db.set("john", 100);
db.set("user 100", "bobby is my name");
auto result = db.get!int("john");
if (result.found)
{
writeln(result.value);
}
auto result2 = db.get!string("user 100");
if (result2.found)
{
writeln(result2.value);
}

It’s quite easy to see the new API lends itself robustly to our needs, so that we may implement stateful, strongly typed databases for moss.

Let There Be Databases

We haven’t been too great on sharing progress lately, so welcome to an overdue update on timelines, progress, and database related shmexiness.

Emerging DB design

OK, so you may remember moss-format, our module for reading and writing moss binary archives. It naturally contains much in the way of binary serialisation support, so we’ve extended the format to support “database” files. In reality, they are more like tables binary encoded into a single file, identified by a filepath.

The DB archives are currently stored without compression to ensure 0-copy mmap() access when loading from disk, as a premature optimisation. This may change in future if we find the DB files taking up too much disk space.

So far we’ve implemented a “StateMetaDB”, which stores metadata on every recorded State on the system, and right now I’m in the progress of implementing the “StateEntriesDB”, which is something akin to a binary encoded dpkg selections file with candidate specification reasons.

Next on the list is the LayoutsDB (file manifests) and the CacheDB, for recording refcounts of every cached file in the OS pool.

Moss Unlocked

Well, it’s not all doom and gloom these days. We’ve actually made some significant progress in the last few days, so it seems a good time to share a progress update.

Extracting content from moss archives

Oh yeah, that totally happened. So, we can now successfully build moss packages from boulder and then extract them to disk once again with moss. This might sound totally uninteresting, but it demonstrates that our format is actually working as intended.

Admittedly the code is super rough within moss and somewhat proof of concept, however we’re able to extract the contents of the moss archive and rebuild the layout on disk.

Moss Format: Read Write Support

It’s been 8 days since our last blogpost and a lot of development work has happened in that time. Specifically, we’ve completely reworked the internals of the moss-format module to support read/write operations… Which means installation is coming soon (TM)

Development work on moss-format

So, many commits have been made to the core repositories, however the most important project to focus on right now is moss-format, which we used to define and implement our binary and source formats. This module is shared between boulder, our build tool, and moss, our package manager.

We’ve removed the old enumeration approach from the Reader class, instead requiring that it processes data payloads in-place, deferring reading the content payload streams. We’ve also enforced strong typing to allow safe and powerful APIs:

import moss.format.binary.payload.meta : MetaPayload;
auto reader = new Reader(File(argv[0], "rb"));
auto metadata = reader.payload!MetaPayload();

Right now we can read and write our MetaPayload from and to the stream, allowing us to encode & decode the metadata associated with the package, with strong type information (i.e. Uint64, String, etc.)

Unlocking Moss

Wait, what? Another blog post? In the same WEEK? Yeah totally doing that now. So, this is just another devlog update but there have been some interesting updates that we’d like to share.

Thanks to some awesome work from Peter, we now have LDC (The LLVM D Lang Compiler) present in the stage3 bootstrap. To simplify the process we use the official binary release of LDC to bootstrap LDC for Serpent OS.

In turn, this has allowed us to get to a point where we can now build moss and boulder within stage3. This is super important, as we’ll use the stage3 chroot and boulder to produce the binary packages that create stage4.

Some patching has taken place to prevent using ld.gold and instead use lld to integrate with our toolchain decisions.

A wild LDC appears

Cleanups Complete

Well, we’re officially back to working around the clock. After spending some time optimising my workflow, I’ve been busy getting the entire codebase cleaned up, so we can begin working towards an MVP.

Lots and lots of work

Since Friday, I’ve been working extensively on cleaning up the codebase for the following projects:

  • boulder
  • moss
  • moss-core
  • moss-format

As it now stands, 1 lint issue (a non-issue, really) exists across all 4 projects. A plethora of issues have been resolved, ranging from endian correctness in the format to correct idiomatic D-lang integration and module naming.

Granted, cleanups aren’t all that sexy. Peter has been updating many parts of the bootstrap project, introducing systemd 247.3, LLVM 11.0.1, etc. We now have all 3 stages building correctly again for the x86_64 target.

Build System Functional

Wow, has it been a hectic few weeks, and it definitely shows: last time we blogged it was about re-bootstrapping with glibc. Feels like ancient news already! So, what’s new in the world of Serpent OS? Apart from yours truly now being proud parent to a beautiful baby girl, work has resumed on the development of Moss, our package manager. And it builds stuff. Awesomely. Let’s quickly catch up with those updates and see where we’re headed next.

Look at all the buildiness

We’re now able to build the majority of a moss package. Notice I’ve made a distinction there. So, we’re able to run the build instructions, and use all the metadata, configuration and macros available. The only thing we’re not actually doing is dumping the built files into a binary package.

We’re able to do the build-system part rather well, now. Right now we support the following features in the build system:

  • Multiple, profile-based architecture support
  • Automatic -m32 profile based cross compilation support for 32-bit libraries on 64-bit OS
  • Basic Profile Guided Optimisation via the workload key. Set a workload, optimise accordingly.
  • LLVM Context Sensitive Profile Guided Optimisation. This is default for LLVM with the workload key, and comes for free, with multiple build stages.
  • Profile based tuning options, such as optimize for speed, size, disabling hardening, etc.
  • Trivially switch between gnu and llvm toolchain in stone.yml, with profiles knowing the right flags to use with tuning options.
  • Recursive macro support in build scripts, defined on per-profile level
  • Architecture specific profiles support in stone.yml, i.e. profiles -> aarch64 -> setup: to avoid if/else spaghetti.

Results Of The Experiment

It seems like only yesterday we announced to the world a Great Experiment. It was in fact 2 months ago, and a whole lot of work has happened since that point. A few take-homes are immediately clear, the primary one being the need to be a community-oriented Linux distribution.

To quote ourselves 2 months ago:

If the experiment is a success, which of course means having tight controls on scope and timescale, then one would assume the primary way to use Serpent OS would be through some downstream repackaging or reconfiguration, i.e. basing upon the project, to meet specific user demand.

It turns out that so far the experiment has been successful, and being forkable is still at the very heart of our goals. Others have joined us on our journey, and expressed the same passion in our goals as we have. A community has formed around the project, with individuals sharing the same ambitions for a reliable, rolling operating system with a powerful package manager.

Toolchain bootstrap

Source Format Defined

Following quickly on the heels of yesterday’s announcement that the binary format has been defined, we’ve now implemented the initial version of our source format. The source format provides metadata on a package along with instructions on how to build the package.

The next step of course, is to implement the build tool, converting the source specification into a binary package that the end user can install. With our 2 formats defined, we can now go ahead and implement the build routines.

Very trivial package recipe

The eagle-eyed among you will already see this is a derivation of the package.yml format I originally created while at Solus. Minor adaptations to the format have been made to support multiple architectures via the profiles key, and package splitting behaviour has now been grouped under a packages key to make the structure more readable.

In package.yml, one would have to redefine subpackage summaries as a key in a list of the primary summary key, such as:

rundeps:
- primary-run-dep
- dev: secondary-run-dep
summary:
- Some Summary
- dev: Some different summary

We’ve opted to group “Package Definition” behaviour into core structs, which are allowed to appear in the root-level package and subpackages:

summary: Top-level summary
packages:
- dev:
summary: Different summary
rundeps: Different rundeps

Moss Format Defined

The core team have been hard at work lately implementing the Moss package manager. We now have an initial version of the binary format that we’re happy with, so we thought we’d share a progress update with you.

Development work on moss

Briefly, the binary container format consists of 4 payloads:

  • Meta (Information on the package)
  • Content (a binary blob containing all files)
  • Index (indices to files within the binary blob)
  • Layout (How to apply the files to disk)

Each payload is verified internally using a CRC64-ISO, and contains basic information such as the length of the payload both compressed and uncompressed, the compression algorithm used (zstd and zlib supported) as well as the type and version of the payload. All multiple-byte values are stored in Big Endian order (i.e. Network Byte Order).

Payloads

Internally the representation of a Payload is defined as a 32-byte struct:

@autoEndian uint64_t length = 0; /* 8 bytes */
@autoEndian uint64_t size = 0; /* 8 bytes */
ubyte[8] crc64 = 0; /* CRC64-ISO */
@autoEndian uint32_t numRecords = 0; /* 4 bytes */
@autoEndian uint16_t payloadVersion = 0; /* 2 bytes */
PayloadType type = PayloadType.Unknown; /* 1 byte */
PayloadCompression compression = PayloadCompression.Unknown; /* 1 byte */

We merge all unique files in a package rootfs into the Content payload, and compress that using zstd. The offsets to each unique file (i.e. the sha256sum) are stored within the Index payload, allowing us to extract relevant portions from the “megablob” using copy_file_range().

These files will become part of the system hash store, allowing another level of deduplication between all system packages. Finally, we use the Layout payload to apply the layout of the package into a transactional rootfs. This will define paths, such as /usr/bin/nano, along with permissions, types, etc. All regular files will actually be created as hard links from the hash store, allowing deduplication and snapshots.

The Meta payload consists of a number of records, each with strongly defined types (such as String or Int64) along with the tag, i.e. Name or Summary. The entire format is binary to ensure greater resilience and a more compact representation. For example, each metadata key is only 8 bytes.

@autoEndian uint32_t length; /** 4 bytes per record length*/
@autoEndian RecordTag tag; /** 2 bytes for the tag */
RecordType type; /** 1 byte for the type */
ubyte[1] padding = 0;

Defining Moss

Over the past few weeks, throughout the entire bootstrap process, we’ve been deliberating on what our package manager is going to look like. We now have a very solid idea on what that’ll be, so it’s time for a blog post to share with the community.

Initial moss prototype CLI

The team has been very clear in wanting a traditional package management solution, whereby repositories and packages compose the OS as a whole. However, we also want modern features, such as being stateless by default. A common theme is wanting to reduce the complex iterations of an OS into something that is sanely versioned, but also flexible, to ensure a consistent, tested experience with multiple end targets.

Additionally, the OS must be incredibly easy for contributors and team members to maintain, with intelligent tooling and simple, but powerful formats.

One of the most recent software update trends of recent years is atomic updates. In essence this allows applying an update in a single operation, and importantly, reversing an update in a single operation, without impacting the running system.

This is typically achieved using a so-called A/B switch, which is what we will also do with moss. We won’t rely on any specific filesystem for this implementation, instead relying on a smart layout, pivot_root and a few other tricks.

Primarily we’ll update a single /usr symlink to point to the current OS image, with / being a read-only faux rootfs, populated with established mountpoints and symlinks. Mutation will be possible only via moss transactions, or in world-writable locations (/opt, /usr/local, /etc …)

Stage3 Complete

Another week, another milestone completed. We’re pleased to announce that we’ve completed the initial stage3 bootstrap. While we have some parallel works to complete for stage4, we can now begin to work on the exciting pieces! We’re now bootstrapped on ARMv8(-a) and x86_64

Build

Our immediate focus is now to implement our package manager: moss. Currently it only has a trivial CLI and does absolutely nothing. We will now shift our attention to implement this as a core part of the stage4 bootstrap. Moss is so-called as we’re a rolling release, a rolling stone gathers no moss. The package manager is being implemented in the D Programming Language1:.

Moss does not aim to be a next generation package management solution, it instead inspired by eopkg, RPM and swupd, aiming to provide a reliable modern package management and update solution with stateless design at the core. Core features and automatic dependencies will be managed through capability subscriptions. The binary format will be versioned and deduplicated, with multiple internal lookup tables and checksums. Our chief focus with moss is a reliable system management tool that is accessible even in situations where internet access is limited.

In order to complete stage3, we provided linker stubs in libwildebeest to ensure we can build. Obviously this introduces runtime errors in systemd and our stage3 isn’t bootable, just chrootable. This will be resolved when systemd is properly packaged by moss in stage4.

Looking Stage4 In The Eye

Well, we’ve made an awful lot of progress in these last few days. It wasn’t that long ago that we introduced some of the new projects required to get stage3 off the ground.

Building systemd the easy way

Our libc-support project has been growing, thanks primarily to the contributions of Jouni Roivas. We now have initial working versions of getent and getconf. The getconf program is considered feature-complete for our current requirements, and the focus is now on cleaning up getent making it more modular and easy to maintain in the long run.

Stage 3 Progress

Well, it’s been a few days since we last spoke, so now it’s time for a quick roundup. Long story short, we’re approaching the end of the stage3 bootstrap.

Fully functional chroot

In an effort to simplify our bootstrap process, we dropped the newly-introduced stage2.5 and came up with a new strategy for stage3. In order to make it all work nicely, we bind-mount the stage2 resulting runtime at /serpent within the stage3 chroot environment, executing the /serpent/usr/bin/bash shell.

In order to make this work, we build an intermediate musl package for libc.so in the very start of stage3, with all subsequent builds being performed in chroots. Part of the build is done on the host, i.e. extraction and patching, minimising the tool requirements for the chroot environment. The configuration, build and install is performed from within the initially empty chroot environment, replacing all the /serpent/usr/bin tools and /serpent/usr/lib libraries.

As we move further through stage3, towards a fully usable chroot environment, we’ve encountered a small number of blockers. Now, we could solve them by using existing patchwork and workarounds, but most have not and will not be accepted upstream. Additionally it is incredibly hard to track the origin and history of most of these, making security rather more painful.

We’re going to start working on a project to flesh out the musl runtime with some missing utilities, written with a clean-room approach. These will initially include the getconf and getent tools, which will be written only with Linux in mind, and no legacy/BSD support.

These will be maintained over at our GitHub

Stage2 Complete

Just in case you thought we were sleeping behind the wheel, we’ve got another blogpost for your viewing pleasure. In a nutshell, we completed stage2 bootstrap.

Complete build-target for ARMv8

In order to simplify life, we greatly reduced the size of the stage2 build component. This decision was taken to better support cross-compilation in the face of software that is distinctly cross-compilation unfriendly.

A support stage, stage2.5 will be added which will chroot into a copy of stage2, and natively compile a small handful of packages required to complete stage3, also within the chroot environment.

For cross-compilation, we’ll be relying on qemu-static to complete 2.5 and 3. However, at this point in time, we have the following:

  • Working cross-compilation of the entire bootstrap
  • Complete LLVM based toolchain: clang, llvm, libc++, lib++abi, libunwind
  • Entirety of stage2 built with musl libc.
  • Working, minimal, chroot environment as product of stage2, with working compiler (C & C++)

x86_64

This is a major milestone for the project, as it is an early indication that we’re self hosting.

Stage1 Complete

Short and sweet, stage1 of the bootstrap is complete. As I indicated on the Lispy Snake1 blog, I’m still in the process of settling into new accommodation. This is going well, but still awaiting proper broadband connectivity. Work has begun, however, and we’re now moving onto stage2 of the bootstrap.

This is handled via our bootstrap-scripts2 project and can be run by anyone on a relatively modern Linux distribution.

Validating stage1 cross-compiler

Next on the list is completing stage2, which we’ve already started on. This is simply a cross-compiled chroot environment with all the basic bits in place to build stage3, sanitizing and cleansing the toolchain.

Even though it is early days, we can already, automatically produce a working cross-compiler that targets LLVM’s libc++, the musl libc.so, and x86_64-serpent-linux-musl host triplet.

Probably a dull update for most, but an update it is.


  1. Since this blog post, the Lipsy Snake website no longer exists. ↩

  2. Since this blog post, the bootstrap-scripts project no longer exists. ↩

Welcome: Project Manager

The Serpent OS team wishes to formally welcome Aydemir Ulaş Şahin to the core team in the primary capacity of Project Manager. Aydemir will help us to coordinate and manage the project goals and infrastructure, being responsible for certain “keys to the kingdom”.

Aydemir has extensive experience with Linux, both as a user and developer, having worked in the media industry for many years. Additionally he will be contributing to the core system, code and packaging.

Our initial trio has been announced, and you can always access information on the current team at the new About page. Now that much of our groundwork has been laid, we can get on with building the project. Check progress at the new Roadmap1 page.


  1. Since this blog post, we have removed our Roadmap page. ↩

Welcome Peter

Recently we did reveal on Twitter that long time friend and colleague-of-many-projects, Peter O’Connor, has formally joined the core team for Serpent OS. Many of you know him as sunnyflunk. He will be working on core toolchain optimisations, benchmarking, and eventually Plasma inclusion.

Peter has long wanted to get the most out of his computing experience, starting from Gentoo back in the 2000’s to ensure there was no wasted resources on features he didn’t use. The desire to over-tinker and use unstable software often led to poor outcomes. While the itch laid dormant for over a decade, it has always been there.

Peter O’Connor (sunnyflunk)

There’s not much about Linux that currently excites me, but I’ve always wanted to see what’s possible with a full LLVM/Clang stack. There’s never been a better time to find out, but it can be difficult to push the envelope when limited with backwards compatibility and supporting older CPU architectures. After experimenting with some tool-chain customisations the chance to test them out more broadly and with a new tool-chain was too hard to pass up. This is the research project I’ve always wanted to undertake and looking forward to using it as my daily driver

We’d like to officially extend our welcome to Peter, as the ramp up gets in place. It should be noted, he is partly to blame for Serpent OS becoming manifest, as the core team got together after discussions on IRC1. He has some fantastic ideas, and is a brilliant distribution engineer with proven success.

As an aside, we’re working on improving the website to incorporate a team page, ready for the third team member reveal.


  1. aerynOS now uses a Zulip server for community discussion. ↩

The Great Experiment

So as many have come to realise, we had to rush out a website super quick yesterday as the cat was already out of the bag. One thing that should also be clarified, is our approach to development.

Primarily (but not exclusively) Serpent OS (future: Serpent Linux) is an endeavour worked on by the Lispy Snake, Ltd1 crew. So we’re all about creating awesome technology and trying to further the playing field.

Serpent OS is not, however, owned by Lispy Snake. It is a contributor-based open source project, with specific roles. Those will be clarified over the coming week, but there is a dedicated Project Manager. Hint: It’s not me.

So, how does an experimental project… measure success? Quite simply - if others adopt our methodologies or technology, we’ve succeeded. We’re really not looking at “number of installs” or “size of userbase” as a metric, as our chief aim is to build technology.

If the experiment is a success, which of course means having tight controls on scope and timescale, then one would assume the primary way to use Serpent OS would be through some downstream repackaging or reconfiguration, i.e. basing upon the project, to meet specific user demand.

This is the heart and soul of all Linux development. Developers should enjoy the involvement and seeing the end result. If we build something more in the end, then that’s a journey we can create together.


  1. Since this blog post, the Lipsy Snake website no longer exists. ↩

First Post

In a nutshell, this post wasn’t meant to be coming out for a few weeks. As it turns out, we had to get ahead of the cat coming out of the bag, and ensure clarity of goals and ambition.

We’ve got an awful lot of work ahead of us across the next few months and weeks, and we strongly suggest checking out the About page which should answer most questions.

TL;DR This is not your typical user-facing Linux distribution.

So go ahead, check out the About page.

Posts will follow in the coming weeks, introducing the core team. It should be pointed out, internally, we’re in a stage1 bootstrap phase, with stage2 ready. Tooling is next on the agenda.

If you’re looking for a modern, lightweight, user-friendly, privacy-focused Linux desktop distribution, then you’re in the wrong place.

Otherwise, stay tuned for the updates. We know our website is insanely wbasic, but it will be improved when it becomes a priority. You’re more than welcome to join us on #serpentOS on freenode1 to become part of the community discussion in the meantime.

Til next time.


  1. aerynOS now uses a Zulip server for community discussion. ↩