Ikey

44 posts by Ikey

🏗️ AerynOS: The OS As Infrastructure

Mar 30, 2025

Ikey

Founder

Taking a break from our usual release-oriented updates, I thought it was high time to dive into what AerynOS actually is and what sets it apart from other distros. Fair warning, this is a meaty, in depth post, and still only scratches the surface. If you’re interested in the future of Linux distributions, read on. It may also help to visit our work-in-progress documentation site for more information.

🧱 The OS As Infrastructure

Firstly, AerynOS isn’t Yet Another Linux Distribution. It’s a platform, a foundation, and a set of tools engineered in accordance with a vision and design that just so happens to produce a Linux distribution. Were we to go back in time and build a design brief for AerynOS, the initial question might be:

What if the operating system itself behaved like modern infrastructure?

AerynOS is the answer to that question. What if, instead of doing things the way they’ve always been done, we started from the ground up and produced a well designed system, rather than the traditional model of in-place mutation internal to a distribution?

🏗️ The AerynOS Foundation

Leveraging years of experience across the industry, including some notable pioneering projects such as Solus, Clear Linux, and others, we set out to build things the hard way, in order to make them easy.. eventually.

🔧 Toolchain

At the very core, is the decision to not be another GNU/Linux distribution. We default to the LLVM toolchain, using libc++ and compiler-rt by default. This isn’t just a case of “we like LLVM”, but rather a strategic decision to leverage the superior diagnostics, and ensure correctness / portability of packages. Whilst in our very early stages a few years ago we did trial musl, we do default to glibc for compatibility reasons and its superior performance characteristics. The performance advantage of glibc over musl is well-documented, particularly for compute-intensive workloads and applications requiring optimal threading performance. We’re here to build a working, usable system, for a multitude of use cases and verticals, and glibc provides the best balance of compatibility and performance.

Note, we do still package gcc and it’s trivial to enforce its use in a package recipe by setting the toolchain field in stone.yaml to gnu.

Lastly, it’s worth noting we build all packages for x86_64-v2 as our minimum baseline, and perform targeted optimisation in our package recipes using the extensive tuning options configurable in boulder.

📄 Statelessness

Our packages are forbidden from containing any files outside of /usr. In order to enable this, packages and/or configurations are altered in AerynOS to ensure they can operate in the absence of a user provided configuration. This forces us to ensure sane-defaults are baked in at all levels, and eliminates dreaded 3-way merge conflicts on package updates. There are no conflicts, because everything in /etc and /var is yours. Likewise, /usr belongs exclusively to the system.

This approach was coined and developed during Clear Linux and Solus days, and we’re refining it further in AerynOS. Other projects have adopted similar approaches, termed “hermetic /usr”.

You might be wondering how files end up in /etc automatically. To this end, we support two forms of package triggers:

🔄 Transaction Triggers

These triggers are run at the end of a transaction in an ephemeral container (linux namespace) and may affect the contents of the transaction-specific /usr tree. This is useful for interdependent packages that need to dynamically produce plugin registries, for example.

🖥️ System Triggers

These triggers do not run in an isolated container, but instead are run in the context of the host system after the transaction has been successfully built and applied. It is these (minimally used) triggers that invoke systemd-tmpfiles, systemd-sysusers, etc. Even for these cases we take special care to ensure that our default configs are sane and that a rebuild is always possible.

👤 System Accounts

Another important aspect is the handling of local system accounts. Traditionally these are snippets/shell scripts that invoke useradd, groupadd, etc. In AerynOS we default to systemd userdb for any users/groups that do not explicitly need groups that users would join. Thus, using drop-ins in /usr/lib/userdb, most accounts are defined and made available via NSS. We use this already in gdm, polkit, colord, and many others.

For the other cases, we utilise systemd-sysusers using snippets in /usr/lib/sysusers.d to create the necessary system accounts. In time, when GNOME and other desktops (as well as shadow et al.) are more adapted to userdb and systemd-homed, we won’t need to rely so much on sysusers. The goal of course is elimination of /etc/passwd and /etc/group entirely, facilitating quicker recovery, provisioning, etc. For now some packages will ship sysusers-based groups, ie docker group, etc.

⚛️ Atomic Updates

Every moss transaction is atomic. Very high level: we produce a new usr tree (rootfs essentially due to mandated usr-merge) very quickly using hardlinks from a deduplicated cache. Once successfully built and primed, we atomically swap the new tree into place. Effectively, the staged transaction is swapped with the real /usr using renameat2 with RENAME_EXCHANGE. It either works or it doesn’t. Nothing in between.

🥾 Boot Management

Leaning heavily on our blsforme and disks-rs projects, we’ve taken a very different approach to boot management. As part of applying a transaction, we produce a Boot Loader Specification (BLS) entry for the new transaction. The tooling will discover the EFI System Partition (ESP), and mount it if necessary. It then synchronises the bootloader itself (systemd-boot), the relevant BLS entries, and the kernel/initramfs. Additionally it will automatically garbage collect older entries, currently retaining 5 of the last transactions.

It’s not just a case of copy/paste and hope for the best. We dynamically produce the root parameters for the kernel command line by directly reading the superblocks (natively, in Rust) of the devices all the way up the chain for the root filesystem. It means there’s no configuration file anywhere in AerynOS containing your root= parameter, and we’re even able to read LUKS2 encrypted devices to add the rd.luks.uuid parameter.

If that wasn’t cool enough, we also encode the moss transaction ID into the kernel command line. This is picked up during early boot in our initramfs, before /sysroot is pivoted to. Long story short it means that every kernel is correctly synchronised with the right rootfs, and that rollback is cheap, easy, and accessible directly from the boot menu.

In addition, we also automatically support XBOOTLDR partitions. In the absence of LoaderDevicePartUUID EFI variable, we’ll scan the GPT table itself relative to the rootfs to find the ESP, and we’ll always scan GPT relative to ESP to find the XBOOTLDR partition.

Long story short? No /etc/default/grub. In fact, if you wipe your ESP, moss can rebuild it from scratch.

📦 The Stone Format

At the heart of moss lies our .stone format. At a basic level, it’s our binary package format. Using a version-agnostic header, we’ve ensured that we design for the future. We also version every payload within the stone, allowing us to refine and evolve the format over time. Currently we support 4 payloads in a single stone:

• 📄 Content payload: A sequential blob of deduplicated data
• 🔍 Index payload: Contains offsets for the content payload, keyed by the XXH128 hash of the content
• 🗂️ Layout payload: Describes the intended filesystem layout when the stone is applied
• 📋 Metadata payload: Sequence of strongly typed, tagged metadata entries such as name, providers, etc.

Right now we default to XXH128 for hashing, but Blake3 is on the horizon (and used in blsforme). We compress all payloads using Zstd, offering great decompression performance whilst still providing a decent compression ratio.

The process for “installing” a .stone is quite different to other systems.

• 📥 The stone is fetched according to the repository data
• 🧠 Index payload is unpacked in memory
• 🗜️ Content payload is decompressed in a single run to a temporary location
• 🔗 Using the index payload, the content payload is spliced into the content addressable storage (CAS)
• 📂 The layout payload is loaded and merged into the LayoutDB, keyed by the unique package ID (SHA256 for the entire .stone, per the repo)
• 📊 Using the same key, the metadata payload is loaded into the “InstallDB”.

Notice that at no point are we actually “installing” anything. We cache into the CAS, and store metadata and layout details. This is used when producing a transaction. Also note there are various safe guards, integrity checks and such in place. For example every payload contains a “CRC” in the payload header, verified on read. We actually use XXH64 for this.

🔁 What is a transaction?

In AerynOS, a transaction is an entirely self-contained rootfs produced by moss. Right now we have to emulate imperative package management, by producing a new dependency graph seeded from the previous system state as recorded in the StateDB. Alterations are made and validated, and then the new DAG selects the packages to be included in the transaction.

At a high level, the transaction is produced by:

• 📊 Producing a valid dependency graph, seeded from the previous system state
• 🧮 Determine cache availability for each package
• 📥 Fetch and cache every missing .stone to produce the transaction
• 📂 Load all layouts for the stones by ID
• 🌐 Using our multi-layered vfs approach, we build an arena graph of the target filesystem, detecting conflicts ahead of time, whilst also precomputing reparented nodes (i.e symlink redirection) and produce an optimal iteration order for transaction application
• 🚶 Walking the vfs iterator to produce the new rootfs in a staging location, using linkat, mkdirat, etc, to optimally produce the new filesystem, linked from the deduplicated cache
• 📦 Binding an ephemeral container to the new rootfs, and running transaction triggers
• 💾 Recording the transaction in the StateDB.

As mentioned above, once a successful transaction is produced, it is atomically swapped into place. If the transaction fails, no ill effects are observed on the system, and your work day continues as normal. In the event of a successful transaction, the system is updated and ready to go, along with system trigger execution and boot management updates.

🔮 The Future

All of this is just the start. It’s taken a few years, and perhaps now it’s clear why. We’ve not even covered our package build tool, boulder, or indeed our build infrastructure! With that said, lets skip forward a short while and see what’s coming down the pipe.

🌍 Beyond the distro model

To reiterate, we emulate imperative package management. And to be honest, it’s totally pointless. It actually introduces more bugs than it solves. Given that we produce a new rootfs for every transaction, we could just as easily produce an entirely new graph each time too.

That’s exactly what we’re planning to do. In a similar vein to Gentoo or Nix, the intent is a global file to explicitly state the desired state of the system, whilst the tooling will simply fulfil it based on the moss plugin cascade. Whilst we have no intention of conflating state and configuration, we will in time extend this system model to support variants of packages in order to provide a kind of “slots” system, and to implement a richer, saner alternative to the infamous update-alternatives.

Obvious candidates include mutually exclusive packages like coreutils and uutils-coreutils (a gnu variant weight, perhaps?) or coinstallables such as clang, ensuring a default vendor version whilst allowing overrides at the local install or indeed package recipe level.

🔒 Immutability

Often AerynOS is described as an immutable OS, and that’s not strictly true. Granted, every transaction results in a new /usr tree, so local changes won’t persist and recovery is immediately available. However, we’re not immutable in the sense of read-only.

Having produced a composition-first, developer&user friendly atomic update implementation, we want to take this further in order to also support immutability without compromise: no unnecessary reboots. To do so we’ll implement something similar to composefs, without the drawbacks. Using the same transaction driven approach we have now, we’ll simply produce an erofs metadata image dynamically instead of the exploded filesystem tree. Utilising trusted.overlay.redirect xattr, and an overlayfs mount, we’ll expose our own CAS through the layouts in erofs, and also support fsverity hashes. Icing on the cake? We’ll leverage mount stacks to retain our composition-first, no unnecessary reboot ethos.

📚 Versioned repositories

We’ve touched on this a few times in our blog posts. Essentially the binary repository will be produced as a result of available build artefacts correlating to the state of our git repo manifest files (boulder proofs of build). The main repository index will link releases to moss/stone format versions, such that moss would only update to the latest release that it supports. This allows us to stagger breaking changes in a way that nothing breaks at all.

📊 Current Status

• 🚀 Actively shipping GNOME ISOs
• 🎮 Quite usable for gaming with NVIDIA drivers, Steam, Flatpak, etc
• 👥 Real users are already praising stability and innovation
• 🛠️ Focused heavily on disks-rs and lichen-installer to leverage disk strategy files (automatic provisioning described in KDL)

🎉 Conclusion

This isn’t just another distro. It’s us redefining distributing Linux. We’ve achieved a tremendous amount, having successfully integrated all of the above into a cohesive, singular whole. The amusing part of course being that the resulting system is “boring” - it just works.

Now, we are alpha. We’re not done, we’re not without issues. That said, we’re building the future and with your support, we’ll get there even faster. This post has only scratched the surface of AerynOS, but if you like what we’re doing, please do get involved or support us.

Or see other ways in which you can support the project financially.

Tags:

• news

🚀 Hello AerynOS

Mar 25, 2025

Ikey

Founder

Hello from AerynOS! ✨ As you recall from February, we set about rebranding Serpent OS into what you now see, AerynOS. It’s not been without challenges, and where possible we’ve ensured continuity. Additionally we silently dropped a few ISOs, and can now take our first formal steps under our new identity. TLDR: the transition has been entirely fluid and you only need to keep updating, no manual intervention is necessary! 🎉

Progress has been steady but quiet lately, as due to unfortunate circumstances I’ve been working in bursts from my wifes hospital bedside.

💿 But first, an ISO drop!

OK, lets get right to it. We’ve released AerynOS 2025.03, with the following shinies:

• 🖥️ GNOME 48.0
• 🐧 Linux 6.13.8
• 🦊 Firefox 136.0.2
• 🎮 Mesa 25.0.2
• 🚀 Vulkan SDK 1.4.309.0
• 🛠️ LLVM 19.1.7

We’ve also decoupled the internal tooling version from the ISO versions to make them a little bit more.. well.. readable, for humans.

Grab it now from the download page.

💰 Funding

I want to thank everyone for their support of the project since we got more transparent around goals. It’s been a huge help and has facilitated massive progress on the project! 🙏 AerynOS no longer feels like a hacked together PoC, but a very solid daily runner. Yes, it’s certainly alpha, and the installer leaves a lot to be desired. For daily use and updates? It’s really quite something.

Please note that I never realised the Ko-fi goals don’t automatically reset, so the currently listed goal has been running for way more than a month.. xD However, over a period of time, we did manage to achieve it! 🎉 I’ll be resetting it shortly on a fixed date for each month ahead.

🔄 Rebranding, the sane way.

Rebranding can be more challenging than one expects. The easy stuff is out of the way, HTTP redirects and DNS trickery to ensure old URLs continue to work without manual intervention. In the case of the repos, we’ve updated the scheme:

🧪 Unstable presync repos

packages.serpentos.com -> packages.aerynos.dev

🌟 Post-sync user-facing repos

dev.serpentos.com -> packages.aerynos.com

📚 Documentation site

docs.serpentos.com -> aerynos.dev

Note the former scheme made no sense at all whereas now we take advantage of the dotdev for unstable work. A future release of moss will automatically handle the transition on installs for the sake of consistency.

📊 Introducing os-info

Previously we had moss generate the /usr/lib/os-release file on demand using compiled in defaults, which was somewhat inflexible. Now, we ship a JSON file (/usr/lib/os-info.json) containing a description of the OS, the composition of technologies, and the capabilities. While os-release and lsb_release exist, they provide very primitive identification and metadata. os-info is designed to provide compatibility with those formats while being far more expressive. Importantly, it also contains a mechanism for identifying the former identities of an operating system:

...
"former_identities": [
  {
    "id": "serpentos",
    "name": "Serpent OS",
    "start_date": "2020-06-15T00:00:00Z",
    "end_date": "2025-03-17T00:00:00Z",
    "end_version": "0.24.6",
    "announcement": "https://aerynos.com/blog/2025/02/14/evolve-this-os/"
  }
]
...

moss now utilises os-info to generate the os-release file, as well as to provide identification and history for our blsforme crate to manage each entry on the boot partition. This has allowed us to sync the branding on a per transaction basis, while still “owning” the legacy branded transactions on the ESP too (i.e /EFI/serpentos) and ensure they are correctly garbage collected. Interestingly this means that for a short period of time the boot menu will still show some of the older Serpent OS entries, allowing you to roll back to it.

Please visit the os-info repository for more details, the schema, etc. We’re very keen to open collaboration and adoption of this project, envisaging use cases in installers, welcome apps, and indeed even container introspection tooling.

🔧 Installer progress

We’ve officially begun the revamp of lichen-installer using a proper split between the privileged backend and frontend. In order to permit a number of frontends and use cases in future, we’re using tonic gRPC messages to communicate, currently just on a UNIX domain socket. In future, we’ll have use cases for TCP using WASM frontend.

More importantly than that, we’ve now implemented the foundational aspects for disks-rs. Long story short - we’re using provisioning strategies defined in custom .kdl files that are dynamically tested/validated against an input storage pool to produce a working set of changes. Such as, create a new partition table and add some partitions to it, using some constraints logic.

It is early days, but its quite awesome that we’re able to automatically partition disks based on these strategy files! ✨ These will form the backbone of lichen, allowing us to offer rich automatic storage strategies as well as manual partitioning options. Notable is the ability to set the volid or uuid of a specific filesystem, and partuuid, etc. This in essence allows shallow reproduction of an installation/disk topology using a configuration file.

🙏 Acknowledgments

This transition wouldn’t have been possible without our amazing community. Both long-time contributors and newcomers have stepped up to the plate to make AerynOS a reality, pushing us across the finish line during this critical transition.

Special thanks to Cameron for his instrumental work in porting our websites to a more maintainable, Astro-based infrastructure. This migration has not only improved our development workflow but also enhanced the user experience across all our web properties.

Whether you’ve contributed code, reported bugs, tested releases, spread the word, or supported us financially - thank you. AerynOS is truly a community effort, and we’re excited to continue this journey together.

As always, you can join our community on Matrix or contribute on GitHub.

🔄 Migration Guide

For existing Serpent OS users, migrating to AerynOS is straightforward:

1. Simply run the following command in your terminal:

   sudo moss sync -u

2. Some branding changes may require a second moss operation to fully “take” effect (as they’ll be using the new moss binary). After the initial sync, run one of these commands to complete the transition:

   sudo moss sync
   # or
   sudo moss install some-package

That’s it! Your system will seamlessly transition to AerynOS while maintaining all your existing configurations and installed packages.

🔮 Next up

We’ll continue on the path we set at the start of the year. By and large we’ll improve the core tooling to continue delivering a better experience for users, developers, gamers, etc.

By the way, we do have nvidia-open-gpu-kernel-modules / nvidia-graphics-driver for the linux-desktop users wanting to game on their shiny AerynOS installs. Our Steam package works great, and any feedback is appreciated on improving it (ie udev/controller stuff, 6.14 kernel is planned).

• Shifting more towards the installer images, using Slint for the installer frontend, and dropping unnecessary packages from the installer image. Live ISOs will of course be available, but our primary target is installer images.
• Integration of upstreams-rs and ABI tracking into the tooling in order to vastly lighten the load for maintainers.
• Accelerated delivery of milestone ISOs.

Tags:

• news

Evolve This OS

Feb 14, 2025

Ikey

Founder

A long overdue, and highly requested update today. We’re finally rebranding the project!

🤔 Why the change?

The “Serpent OS” name was a quickly chosen name that stuck. Unfortunately “serpents” are often associated with negative connotations, and we’ve had a lot of feedback over the years that the name was off-putting. Let’s be completely honest, it’s not the most inviting name for a project. Who wants to trust a serpent? Generally speaking they’re considered dangerous at best.

It’s fair to say we’ve spent a long time in prototype and alpha phases. In order to move forward, our identity needs to be more befitting of the project we’re building. A move into the real world. This isn’t a hobby project, it’s a full blown Linux distribution with serious technical underpinnings, achievements and goals. Getting the tone right from day dot is critical.

Do note, there is no change to the internal core team, so we’re aiming for full continuity and a seamless transition. Had enough of these snakes on this plane? We have too.

✨ What’s the new name?

Hah, it’s not “Evolve OS”, despite the clickbait title. Our new name is… drum roll please…

🌟 AerynOS

Pronounced like “Erin”, it’s a name that we feel is more befitting of the project. Pulling from multiple etymologies, it’s a name that better describes the project now versus the project that started as Serpent OS.

“Aer” is rather obvious, Latin in origin. The phonetic “Erin” is a nod to the Irish roots of the project, and of course a home. There are a number of reasons for the name, which will form part of the initial documentation on the new website.

Our intent is to have a name that is more inviting, and more descriptive of the project’s goals and aspirations. We’re not anti-establishment or anti-corporation - if anything, we’re a statement that without the fiscal handcuffs, we can produce a technically sound and user-friendly operating system.

⏰ Timeline

Having already secured AerynOS.com, AerynOS.dev, plus the associated social media accounts, we’re now in the process of rebranding the project. For sheer irony, we’ll make the final transition day the 17th of March, 2025. Yes, St. Patrick’s Day. The Matrix space has already been updated, and any areas where a new name isn’t feasible will see almost immediate renames.

A new GitHub organisation has been created, with our first priority there to create a proper organisation structure for once. Over the coming weeks everything will be migrated over, and we’ll mark the old organisation read-only/deprecated. Can’t have folks thinking Serpent OS was abandoned, after all.

Stick with us, we’ve got plenty of exciting news to come over the next days and weeks, as we embrace new transparency and shed our old skin. Plus, we’ll be starting to onboard for web properties, translations, artwork, and more. Cherry on the cake? Admitting that we’re actually building a full distribution will allow us to put relevant priorities in place to expose Serpent OS.. I mean AerynOS .. functionality to the wider world. Get ready for a reboot of the grassroots, independent Linux scene. We’re here to stay.

Tags:

• news

Hello 2025

Feb 6, 2025

Ikey

Founder

TL;DR: Serpent OS is facing funding challenges but development continues. Alpha2 is coming soon with an improved installer. We’re seeking community support through donations and volunteers for key roles. Our technical roadmap includes versioned repositories, immutable OS features, and improved package management workflows.

---

We had a flurry of activity around the Christmas period, including our first alpha release as well as enabling offline rollbacks early in January. We’re actively working on alpha2, but we also need to talk about the elephant in the room.

🐘 The elephant in the room

Recently I posted a tweet that has been covered by a few news outlets. The long and short is quite simple: distinct lack of funds. A very long story short: a change in my contract led to a loss of income protection, combined with a herniated disc (C6-C7), culiminated in having to resign. Naturally I’ve been working exclusively on Serpent OS since then, but I’ve also got to keep the lights on at home.

This doesn’t mean the project is closing or anything like that, but it does mean that I actively need to keep costs down whilst using daytime hours for financing. This is what’s led to the significant slowdown in the last few weeks, and I’m just trying to be as transparent as possible and manage expectations.

💼 But couldn’t you just get a job?

Sure. The reality is that with a job, and kids, there’s very little time left for Serpent OS. I personally believe very strongly in Serpent OS, and I want to see it succeed. As a truly independent project, we’re in a unique position to do things that other projects can’t. We haven’t got to answer to shareholders, or concern ourselves with market share. We can focus on making the best possible OS for our users.

But it’s not just that. We’re building the tools so that others can build their vision of the best OS. Or provide applicances. Or <insert your idea here>. We’re building a set of tools and principles to facilitate innovation and disruption, for projects of any size. We’re not just building an OS, we’re building a platform. Bring your own distro, if you will.

My sincere hope is that others also believe in this vision, and can help us get there. It’s make or break time, and you have the power to help us make it.

💰 Funding

OK, so how can folks help? The most obvious one is funding, so we’ll get that out of the way first. Right now it’s possible to support us via Ko-fi or GitHub Sponsors. For the sake of transparency, as noted on the Sponsor page, we haven’t got an established company or business bank account. In time, we’re looking towards something more cooperative. For now, the reality is the daily running and development largely falls on me.

🏗️ Infrastructure

One way that we could certainly cut some costs, and improve scaling, is through additional infrastructure. We’re currently hosted on a single Hetzner server, and we’re looking to expand that to a more distributed setup. Rune Morling currently hosts 2 of the builders in his home at his own expense, and the main server is also the primary builder, which is less than ideal.

We need to split the repository hosting as its grown enormously, and could do with a few more builders to facilitate “try builds” of pull requests.

Lastly we’re looking at CDN solutions to improve the speed of package downloads (efforts currently ongoing).

👥 Roles and responsibilities

We’re growing quickly and that comes with some pain points. We’re looking for immediate help in the following key areas:

• Community management: Someone to manage social media, forums, and other community platforms.
• Documentation: Once we pivot to Astro + Starship, we’ll need to focus on adding high quality documentation.
• Translation: We’re enabling gettext + fluent where appropriate, and will need extensive translation work.
• Web development: We need to improve our web presence, management of web properties, transition to Astro, etc.

While of course we’d also love more developers and packagers, most of the time is spent in these areas already, and the other roles desperately need attention.

🚀 Latest developments

Before we get into the future, let’s talk about what we’ve been working on lately.

💾 Disk Management (disks-rs)

A couple of weeks back we broke the news that we were working on a new disk management library, disks-rs. The scope wasn’t made quite clear at the time, but it’s another building block to facilitate smarter applications. Eventually we want to support the manipulation of various filesystems and disk structures, for use in CI pipelines, etc.

Another requirement for disk-rs is to not only expose sane partitioning logic, but also to provide a way to declaratively define disk layouts. This will be used in the installer, but also for reproducing entire installations. It forms the basis for our future provisioning system.

Right now we’re working on a very cool .kdl based format for defining disk layouts. It takes advantage of kdl-rs streaming parser to implement a procedural syntax complete with miette error handling for a very nice developer experience:

// Create a partition table. Defaults to GPT
create-partition-table type="gpt" disk="root_disk"

// Create the ESP
create-partition disk="root_disk" id="esp" {
    constraints {
        min (GB)1
        max (GB)2
    }
    type ...
}

🛠️ Installer

Our installer has quite a nice backend, but honestly the frontend experience leaves a lot to be desired. Given that our current focus is being able to not only onboard developers but build tooling against an installed system, we need to make the core of this effectively scriptable. To do so will build upon ongoing IPC work for moss and disks-rs, and the introduction of a new GUI frontend based on Slint. While early days we’re trying to think 10 miles down the road, where we can utilise the LinuxKMS backend for minimal installer ISOs.

Early preview:

🔮 The road to stability

So we have some exploratory works in progress, and it’s time to discuss how they all relate to the future of Serpent OS. You’ll immediately notice that these are highly circular in nature, meaning we need to switch between different areas routinely. In order to design an effective architecture, whilst minimising disruption, we need to have a clear vision of the future that we’re referring to as “10 miles down the road”.

📚 Versioned repositories

As previously explained, one of the built-in mechanisms to support a rolling release with breaking changes to the format is to introduce explicitly versioned repositories. Long story short, a local moss client will only “see” the latest version of the repository that it can actually use. Breaking changes will mean format bumps, and newer repository versions will be unavailable until the client has updated to a client/Serpent OS version that actually supports it.

This work requires changes to the repository format, infrastructure, and a migration path from the current repository to a new archive. However, it is also dependent on having a new format version being available for the versioned-indices.

🔒 Immutable OS

Frequently Serpent OS is described as an immutable OS, but in actuality it is an atomic OS. That means we’ve focused entirely on providing a reliable update mechanism, rollbacks, etc. We have not yet enforced immutability, aka a read-only root filesystem.

Our plan is heavily inspired by composefs. Currently moss produces new filesystem trees using a hardlink farm approach, and relies on renameat2() to atomically switch the root filesystem. This is a very fast operation, but does not provide immutability.

Leveraging the ability of moss to dynamically construct filesystems from packages, it will be abstracted into a driver mechanism. This will allow the classic hardlink approach for buildroots, but employ a new erofs strategy for immutable installs. The newly generated erofs image will essentially rely on extended attributes to “point” to the underlying content addressable storage, and utilise overlayfs to present a read-only view of the filesystem. Further building on this we’ll use so-called “mount tucking” to provide the mechanism for atomic updates without the need for a reboot.

🤔 Why not use composefs?

We’re actually going to make moss support exports to composefs format, making it an ideal match for podman use cases, deduplicating the image content. Whereas composefs produces content from an existing tree, we’ve already got content addressable storage, independent transactions and databases to facilitate this. This means we’ll be able to retain our composition features and very quickly produce the new erofs images. Using composefs directly would significantly slow down the time to produce each transactions, whereas we can instead build a similar internal functionality and integrate at higher levels.

We also plan to integrate the file digests in a new revision of the moss stone format, which will open the door to supporting SELinux xattrs, as well as fsverity for the immutable images.

Note that this goal will inform the requirements for the next stone format, which will in turn require versioned repositories. Of course, this also impacts the build pipeline infrastructure.

👷 Maintainer burden

Truly the most time consuming task in most distributions is the cost of maintaining packages. This is being approached, as with all of the planned goals, in an incremental fashion. However, it often pays to have a view 10 miles down the road to determine exactly how to get there.

🧬 ABI Tracking

In order to provide sanity for developers, and make guarantees about the stability of the system, we need to track the binary ABI for all packages. Furthering on this, we can of course immediately detect any breakage and schedule rebuilds as appropriate. This will also add a higher degree of confidence for pull requests as we’ll know if its consistent or introducing problems.

🔄 Pull Request workflows

The workflow for packaging right now is for someone to open a pull request, which contains their recipes, the automatic “build records” (manifest) and our assumption that they built it using a local repo based on top of the latest repository. What is actually needed here is for the infrastructure to perform automatic (low priority) builds of each pull request in a transient repository, making those builds available for verification purposes.

⚛️ Atomic repo merges

Another limitation with the current infrastructure is the reliance on a global build queue. Simply put, once a PR is merged, all of those builds are scheduled and individually published to the target repository upon completion. This is very naive, and leads to a lot of “nannying” in the event of a failed merge.

In future, those builds will be scheduled in a transient repository, and upon successful completion the resulting artefacts would be published to the target repository in a single atomic operation. This will also allow for the introduction of “try builds” for pull requests.

🩸 The bleed-through problem

Another issue which partially relates to bootstraps, is the bleed through problem. This is an artefact of using layered repositories, such as a local repository, to build updates that would change the providers available in the underlying repository upon merge. Take for example, LLVM.

main repo:
    ├── rust [runtime-depends: libLLVM-18.so]
    └── build-dep: binary(rust) [runtime-depends: libLLVM-18.so]

local repo:
    ├── rust [runtime-depends: libLLVM-19.so]
    └── build-dep: binary(rust) [runtime-depends: libLLVM-18.so]

After completion of libLLVM-19.so, the local repository would contain the new version, but the main repository would still contain the old version. Locally this would allow the circular build dependencies to work as rust builddep for rust could still resolve libLLVM-18.so. However, upon, merging to the repository, we have a new problem. The main repository would now contain libLLVM-19.so, but the rust package would still be built against libLLVM-18.so. This then breaks the dependency chain for rust, requiring manual intervention.

To solve this, we’ll ensure that filtered repository views are introduced both for local builds and our build infrastructure. This will mean that the index of the local repository will occlude the main repository, and eliminate bleed-through.

⭕ The bootstrap problem

Whilst ABI tracking is very important, it doesn’t actually solve the bootstrap problem of circular dependencies. This is a consequence of using an incremental development model for a binary-first distribution. In reality the only real way to solve this is by eventually altering the tooling to support a “world view”, with intermediate recipes that are not published to the final repository. In effect, combined with ABI tracking for dependency chain invalidation, we can determine which recipes need to run again as part of an invaldation-solver loop that relies on “cached artifacts”, i.e. the final packages.

This isn’t an immediate issue to solve, but it will inform our next steps to ensure we get to that point without requiring extensive overhauls or “stop the world” rewrites. Our eventual vision is that building any new package will feel much like it does now, but changing a reverse dependency will evaluate the dependent chain for rebuilds. To make this streamlined, the infrastructure will by this time have grown to a point where we can share the compute power by way of so-called “try builds”, and a shared cache of artifacts.

📦 Keeping the packages up to date

A large portion of the burden is in reality chasing updates. Currently we have a very simplistic utility that cross-references release-monitoring.org to spit out a list of recipes that have available updates. Truthfully we could easily generate diffs for these updates, and even automate the process of updating the recipes. This is dependent however on having confidence in the pull request, which in turn is dependent on ABI tracking being in place.

🗂️ Organising it all

We have a complex series of interdependent goals, which require research and frequently shifting between focus areas. Not all of them will be immediately realised, and some are simply informing the direction for our future. Additionally, we need to keep moving without any “rewrite the world” pauses.

It’s important to highlight that none of this is a reinvention of the project, but a series of steps to “1.0” and beyond to ensure that the project is sustainable and reliable. This means that delivery continues, and the delivery itself becomes simpler and more trustworthy.

Despite the interdependencies, once we’ve finished scoping the end requirements we can work towards the immediate concerns:

• Pull request workflow
• Repository consistency (initially ABI story)
• Update automation
• Immutability

Note: These timelines are estimates and may adjust based on available resources and community support.

Scale-up considerations

We’re at the point where we need a functional OS for contributors to be able to work on the project. It also presents a paradox because we wish to keep the project lean whilst we work on extending the scale-out capabilities. In reality this means that we’re going to permit a limited amount of repository growth:

• Addition of the Plasma desktop for daily use
• Container-management tooling (podman, docker, etc)
• Development tools (IDEs, etc)
• Onboarding requirements (such as Matrix clients, documentation, etc)

This in turn also requires a more reliable installation experience, which is why we’ve been spending cycles on the installer project. Internally we want the installer to support a variety of scriptable configurations in order to provide better test coverage for our updates and boot management, whilst we also intend to incrementally improve the user experience.

📅 Conclusion

Alpha2 is coming, and will feature a preliminary version of the updated installer. The plan initially is for a simple “use whole disk” strategy and will be followed up with supported for encrypted installations (post alpha2). Quite simply, some employers require their developers have encrypted installs, and not supporting this configuration will restrict our ability to onboard developers.

We’re doing the Alpha2 as a baseline, from which all update testing, infrastructure adjustments, etc, will be measured against. This is the point at which the project will formally “open the doors” for more developers and testers, with the explicit view that growth is sustainable and manageable. Per the requirements set above, we’re not going to “explode”, but enable specific workflows that in turn will permit us to scale out the project and inform direction.

It’s important to me that we’re explicit in how we take our next steps. As stated earlier, pausing the world is not an option. Continued delivery whilst enabling an incremental path to the future is the only way to ensure that we can deliver on our promises. Contrary to a reboot, this is a solidification of our vision and a commitment to the future. Sure, new features will also appear in time, such as easier packaging recipe formats, but no decision is being made that will require a rewrite of the project. However, these goals require stating and explaining up front so that the high level vision is clear and provides the context for any short-term deliverables. The last thing we need is to be in a position where we’re “stuck” and unable to deliver, due to a lack of foresight.

🙏 Thank you

On behalf of the project, I’d like to thank everyone who has supported us so far. Whether that’s through donations, code contributions, or even having the time of day to try out the ISO, it’s deeply appreciated. Being the underdog comes with its own set of challenges, and of course the “indie cost”, but it’s also an opportunity to challenge the status quo and find new, more optimal ways, to do things.

We’re not just building “Serpent OS”, we’re building a set of tools to solve the problem of distributing an OS. Way down the line this will be powerful enough to support a number of use cases, from appliances to desktops, and even servers. In short, we want to bring high quality tooling to the masses, and we’re doing it in a way that’s sustainable and reliable.

Tags:

• news

Offline Rollbacks Enabled

Jan 4, 2025

Ikey

Founder

If a picture tells a thousand words… well then this video .. Yknow what, sod it. Here’s a video of the offline rollback support in action, as tested in a newly installed VM using xfs for the root filesystem. We jokingly refer to this as the “LTT” test due to LinusTechTips’ “interesting encounters” with package management systems in the past.

This has already landed in our volatile repository, so simply sudo moss sync -u and grab the latest updates. Any subsequent transactions will automatically generate the boot entries for you, no intervention required. Or mounting. Or anything. Just update and go. Srsly.

We actually saw this as a golden standard for the feature, as it’s a real-world scenario where you’d want to roll back to a known good state. The video shows the system booting and recovering in multiple ways, from complete system nuke (via glibc) and simpler scenarios like removal of the GNOME desktop environment.

The intent is to ensure a user can quickly revert to the last transaction that worked in the instance an update goes awry, without needing to boot into a rescue environment or live CD.

Like what we’re doing?

Developing a project with the scope of Serpent OS is no small feat. In order to maintain our current cadence, and to land huge features like the offline rollback, or impending system model work, we need your support. I’m as happy as the next person to return to the job market once my medical conditions ease up, but the simple matter is this will greatly slow down our recent progress.

Our aim is to entirely redefine the distribution of Linux, and the years of prior effort are finally coming together rapidly to make this a reality. If you like what we’re doing, consider supporting us!

Visit our sponsor page for more information on how you can help us grow and deliver the future of Linux distributions.

Technical Details

This is all achieved using moss’s internal content addressable storage for deduplication, where each old transaction is swapped with the live /usr at the time into the archive.

The behaviour is implemented in the initramfs (dracut) by invoking moss state activate $transaction_id -D /sysroot, ensuring that the activation is performed before the new root is mounted and the system is booted. We actually shrunk our moss and uutils-coreutils builds to half their usual size so that we can fit them comfortably into the initramfs.

For now, we’ll automatically generate entries on the ESP (or XBOOTLDR) for the last 5 transactions, and you can select them during early boot to perform the rollback. It’s not slow, as it’s just doing yet another renameat2() to atomically exchange the live /usr with the archived transaction.

TLDR: Boot time rollback, no network required, no live CD required, no rescue environment required. Just a working system.

What’s next?

The system model is coming next, which will expose the non-imperative core of the architecture to users. The usual commands will continue to emulate an imperative OS, by mutating the system model and applying the internal sync operation. This ensures dependencies are resolved only according to the layered repository configuration, without special exceptions for “installed” (which.. we spent a lot of effort faking this behaviour).

It’s good to remember that moss is effectively a huge cache, and that each state has no relation to the other. We simulate upgrades by building a new state with the same selections as the old one, then begin to upgrade those candidates.

The new model (somewhat Gentoo inspired ❤️) will allow moss to build the dependency graph for the finished system state. As a sneak preview of what we’re enabling, imagine the newly landed transactional reboot code linked with multiple system models… such that you might decide to moss sync from your existing model to an entirely different one, rebooting from your GNOME desktop to a KDE desktop, for example. Cleanly, atomically, and without any manual intervention.

Tags:

• news

Investing in the Future

Dec 31, 2024

Ikey

Founder

Here we are, at the end of 2024, and what a year it’s been! We’ve made significant progress on Serpent OS, and we’re excited to share some of the highlights with you.

Attained Alpha

Virtually 5 years in the making, we recently attained alpha status. Our tooling and concepts have aligned, allowing us to now rapidly iterate on the core deliverable itself: Serpent OS. We’ve seen an explosion in cadence, with our tooling enabling us to quickly and easily deliver updates, new packages and enabling new features.

Since the Alpha 1 ISO (0.24.5), we’ve fixed a number of issues in relation to hardware and booting. We’ve also added new features to moss (refined boot management) and boulder (our build system). Even now we have multiple PRs open to tidy up moss, add automatic generation of monitoring.yaml files, and more.

Keeping the cadence

We’ve been working hard to keep the cadence up, and we’re now at a point where we can deliver new ISOs on a regular basis. Our main focus now is the tooling, in enabling a far more automatic delivery pipeline. The vision is having automated pull requests generated for package updates, automatically handling changes in dependencies and ABI, and ordered by build tier. This will significantly reduce the maintainer burden, in conjunction with planned security monitoring.

Short term deliverables

In the short term we’re going to land the following features:

• Offline Rollbacks - We’re going to land offline rollbacks in the boot menu, allowing you to easily revert to a previous system state.
• Versioned Repositories - We’re going to introduce versioned repositories, allowing us to gracefully handle format changes in moss and boulder, permitting an “update forever” epoch-staggering approach. Additionally, this will allow users to stay on an older repo version if they’re awaiting a fix to a regression in a newer version.
• Improved Documentation - We’re going to improve our documentation, making it easier for new users to get started with Serpent OS.
• Tools as a service - We’re adding IPC layers to our tooling, allowing them to be launched as helper services for integration in other applications. This will allow us to integrate moss with GNOME Software and COSMIC Store, for example. Additionally it will ensure the latest version of the moss binary is always used by the system, gracefully handling version repo changes. Chiefly, this will allow moss to cheaply integrate anywhere we need it, including for lichen.
• Lichen Improvements - Lichen will also be converted to an IPC based backend, allowing it to be executed via pkexec for integration into a GUI (or even CLI) frontend. This is a necessary step to allow the frontend to operate without privileges, facilitating live updates to the NetworkManager configuration, or indeed keyboard layouts. Long term we’re planning a cage based kiosk with a full screen installer mode, on a far smaller ISO.

Long term vision

Our long term vision is to deliver a system that is easy to use, reliable, and secure. We’re building Serpent OS to be a system that updates forever, confidently. We’re also shaking things up by being first-adopter of new technologies as default solutions, with a view to enhancing the robustness, origin-diversity and security of the system through Rust based replacements and alternatives to key components. You can already see this with our adoption of sudo-rs, coreutils (uutils), ntpd-rs, and not to forget rustls by default for our curl build.

Outside of that, moss, boulder and the supporting crates are also written in Rust, giving us a strong foundation for the future.

Serpent OS is a project that is built to last, and challenge the status quo. Deliver a Linux distribution for use on multiple verticals, with the feeling of a conventional Linux distribution, but with key management facilities and architecture designed to make a rolling release distribution that can be trusted for long term use, extensive updates, and the ability to entirely re-engineer the entire OS, delivering in a safe atomic update without breaking client installations.

How you can help

In order to give Serpent OS the development time it needs, we’re looking for sponsors to help fund development. If you’re interested in becoming a sponsor or investor, please do not hesitate to get in touch. We want to secure funding for development and infrastructure for the next 2 years.

• Sponsors
• Discuss sponsorship/investment

While of course we need to iterate on our own tooling, we also need to be able to work more upstream and with kindred projects to drive the ecosystem forward. In time, we wish to become the incubator for new technologies and solutions, and we need your help to do that.

With your support, we can expand our builder network and enable more hosting, bandwidth, and compute resources. This will in turn enable our automated rebootstraps, which will of course allow downstreams to extend upon Serpent OS as a kit-distro. Indeed, it’ll allow us to expand beyond x86_64 when the time is right.

While Serpent OS currently delivers a desktop ISO, this is not the limit of our potential. As indicated, our priorities in the short term are designed to massively aimplify project maintainer bandwidth and free up more time for feature iteration and development. Once we’re quite happy with the base (and we’re not far from that point) - we’ll quite happily extend to other desktops, and even server/cloud offerings.

Nothing is bullet proof, but with the architecture and tooling of Serpent OS we can get pretty close. Gotta admit, having a transactional, atomic OS with rollbacks, and the full granularity of a conventional package manager, is pretty cool. 😎 Especially when things “Just work” out of the box thanks to the stateless (“hermetic usr”) requirements of the system.

Know what’s even cooler? When we land our system-model implementation, you’ll be able to completely duplicate the setup between local development and production. When we also land our export/compose functionality in moss, you’ll be able to reap these benefits in containers too. 🐍

Here’s to a great 2025, and we hope you’ll join us on this journey. 🚀

Tags:

• news

Alpha Refresh Available

Dec 29, 2024

Ikey

Founder

In response to feedback from the community, we’re issuing a refresh of the Alpha 1 release, 0.24.6. This refresh includes a number of key updates and improvements, including:

• Now works with etcher
• Fractional scaling enabled by default
• Prebuild icon theme caches to speed up trigger execution (except for hicolor)
• Add new moss boot status command for introspection into blsforme functionality
• Fix amdgpu initialisation for older devices, previously leading to black screen on boot

Clarification on moss boot management

Frequently in Team Serpent we joke that moss is the “systemd of package managers”. It actually does a whole lot more than is easily visible.. for example:

• Content addressable storage for the entire OS
• Atomic updates of /usr by renameat2() w/ RENAME_EXCHANGE
• Executes transactional triggers in a namespace (container) with RW /usr and RO binds of /etc and /var …
• Executes system triggers postblit, with dependency ordering..
• Via blsforme, manages the ESP, XBOOTLDR, boot entries, sync and promotion of bootloader, kernel and assets, initrds..
• Entirely zero configuration. No /etc/default/grub, no remembering your root={}, it’s determined automatically.
• Basically does all the work of OS provisioning, barring disk mounting and partitioning. Our installer, lichen, is basically a thin wrapper of moss.

We’ve had no debugging capabilities around the boot management, so we’ve landed moss boot status command. It’s super primitive but does offer an insight into the introspection capabilities of blsforme. In this simple screenshot, moss/blsforme are supporting the Boot Loader Protocol, querying the systemd-boot entries in efivars. When found, LoaderDevicePartUUID is used to determine the boot ESP, but moss/blsforme will quite happily read the GPT directly (relative to / mount) and find the ESP and XBOOTLDR…

Remember - we happily use systemd-boot. We do not use bootctl or kernel-install. Entries are automatically managed and we’re extending this to support offline rollbacks via boot menu (feel free to extend with /etc/kernel/cmdline.d/*.cmdline drop-in snippets!) Also note we do not allow systemd to automount the ESP or XBOOTLDR via the GPT autogenerator, instead moss/blsforme will automatically discover and if necessary, mount these partitions, when dealing with bootloader or kernel updates.

With this explicit control over the boot management, we’ll happily be landing secure boot and fwupd support in the near future, without the downsides of trying to provide EFI assets to /boot in a package itself. 😬

A note on virtual machine usage

It must be clearly stated that Serpent OS does not support anything other than UEFI booting on x86_64-v2+ capable hardware. Please note that GNOME Boxes will default to BIOS, not UEFI, and will need hand-mangling of the XML file. Also note in other solutions such as VirtualBox you will need to specifically select EFI in the VM settings.

In order to maximise compatibility with other tooling and firmware, we’ve modified the generation of our ISO to include the isolinux MBR/GPT hybrid components, however we deliberately did not include a functioning bootloader for BIOS boot. Instead, you will see a boot failure indicating that ldlinux.c32 could not be found, which is a sure-fire way to know your VM or hardware is incorrectly configured.

Installer notes

Lichen is due to receive some love in the next few cycles, so please be aware of the following limitations:

• A pre-configured GPT disk is required before lichen is launched
• A working internet connection is also required, this is a network installer
• A FAT32 formatted EFI System Partition is required (marked as esp in gparted)
• For optimal usage, or when your ESP is tiny, please create an XBOOTLDR partition:
  • Create an FAT32 partition in gparted of 2GB in size
  • Mark it as bls_boot in gparted
• Further filesystem support for XBOOTLDR will be added to blsforme when we’ve finished packaging efifs.
• XFS or F2FS are strongly recommended for the rootfs, due to the limitations of ext4 with hardlink counts.

COSMIC Desktop

We’ve made the decision to pause our offering of COSMIC Desktop ISOs. We’re huge fans of the project, and we strive to keep it up to date and functional. However, it is very early in alpha, and so are we. It doesn’t make a lot of sense for us to extend our workload to two alpha codebases!

We still offer COSMIC as an option in the installer for those who wish to try it out, and do encourage testing and engaging upstream with the developers. For now, we will offer just the GNOME ISO.

Next up

In the new year we’ll be focusing heavily on our tooling to facilitate a scale up of our packaging efforts. We’re quite proud of our tooling, but we’re not oblivious to the warts and issues. The next couple of weeks will be spent improving our documentation (or making it exist!) and making sure that we have the ideal workflow in place to permit large stack updates along with the ABI consistency verification we need. This will also pool work for ent, moss, boulder, by streamlining packaging, and making it trivial to update an entire stack of packages in one go. Eventually this will be an automatic thing, with successful PRs being verified and then merged into the volatile repository.

Don’t forget, versioned repositories are also coming your way, and very soon we’re landing offline rollbacks in the boot menu! blsforme has been significantly restructured for cmdline handling, and understands entries in the context of a distinct system root (ie moss filesystem transaction) - which is the bulk of the work needed to facilitate offline rollbacks. The last piece will be running moss at an early stage in the initrd to swap the /usr pointer to the older transaction, which is instanteous and atomic.

Final thoughts

5 years and no release, and now two alpha ISOs before the end of the year. We’re pretty happy with that, and we’re going to keep surprising you next year with more releases, more features, and more stability. Happy new year from all of us at Team Serpent! 🐍

If you like what we’re doing, please consider sponsoring us - this will allow us to grow our infrastructure and deliver builds quicker. Due to health issues in the last few months, I was forced into a position whereby I had to resign from my job. Any and all contributions to the project are greatly appreciated, and will help us to continue delivering what folks are frequently now calling “the cleanest Linux implementation they’ve ever seen”.

Try it. You might like it. 🚀

Tags:

• news

Serpent OS Enters Alpha

Dec 23, 2024

Ikey

Founder

Following a successful prealpha phase, we’re excited to announce that Serpent OS has now entered the alpha stage of development. This milestone marks a significant step forward in the project’s journey. While the usual disclaimers apply, as a certain level of project fluidity is to be expected, we invite you to explore the latest alpha release and experience the cutting-edge features and improvements we’ve introduced.

Supporting Serpent OS

Serpent OS is a community-driven project, and we welcome contributions from developers, testers, and users alike. If you’re interested in helping shape the future of Linux distributions, we invite you to join our growing community and get involved in the project. Whether you’re interested in packaging, development, documentation, testing, or community support, there are many ways to contribute to Serpent OS.

Lastly, you can support the project by becoming a sponsor. Your support helps cover our hosting costs and allows us to dedicate more time to making Serpent OS exceptional. We’re grateful for the support we’ve received so far and look forward to continuing to build something meaningful in the Linux ecosystem.

Sponsor Serpent OS

What is Serpent OS?

Serpent OS is a heavily engineering-led Linux distribution seeking to redefine how we distribute Linux. It is a stateless OS that leverages atomic updates, cutting-edge tooling, and rock-solid reliability to deliver a safe and efficient system. Built by industry veterans with decades of experience, Serpent OS represents the next evolution in Linux distributions.

We currently offer x86_64-v2 desktop builds, for both GNOME and COSMIC desktop environments. We only support UEFI systems, and plan secure boot support via shim in the near future. The majority of the distribution (including the kernel) is built against the LLVM toolchain, with libc++ as the default C++ standard library.

Through our unique architecture, we’ve combined proven concepts into a cohesive system where updates either fully complete or safely roll back, keeping your system running reliably. Our atomic updates let you see changes immediately, without requiring reboots just to apply updates. While certain updates like kernels will still need a reboot, we’ve designed Serpent OS to be as seamless and user-friendly as possible.

What’s new in the alpha release?

The alpha release introduces several key features and improvements that enhance the overall user experience and system performance. Here are some highlights of what you can expect from Serpent OS alpha:

Hardware support

We’ve rounded out our hardware support, now including patches for ASUS devices and Surface devices. Additionally, we now have the NVIDIA driver available in the repositories. Note we only offer open-gpu-kernel-modules, which we prebuild against our linux-desktop package to streamline installation.

Steam

You can now install Steam from our repos, making use of our newly enabled multilib drivers:

• mesa-32bit
• nvidia-graphics-driver-32bit

More Rust

As part of our efforts to modernise and strengthen the Serpent OS base, we’ve switched some components out for Rust alternatives:

• uutils-coreutils replaces coreutils
• sudo-rs replaces sudo
• ntpd-rs replaces timesyncd
• curl is built with rustls support (and hyper but this is being dropped upstream)

Additionally we include starship by default on our GNOME ISO, along with zed, loupe, resources and other Rust applications.

Packages

Our recipes repository has seen over 1600 commits since our prealpha, with some of the highlights including:

• Linux 6.12.6
• Firefox 133.0.3
• LLVM 18.1.8
• GNOME 47.2
• COSMIC 1.0.0_alpha4

Tooling updates

We’ve made significant improvements to our tooling, including updates to moss, boulder, and blsforme. These tools are essential for managing packages, building software, and handling boot management in Serpent OS.

moss - Package Management

Our atomic package manager swaps the entire /usr directory during system updates, ensuring a stateless, bulletproof upgrade process. Updates either succeed completely or not at all. Recently we made improvements to moss to handle certain special cases:

• EMLINK - ext4 is limited to 65k hardlinks per inode, and we were hitting this limit on some systems due to empty files. We now handle this case gracefully by creating new inodes for empty files.
• ENOSPC - We now mitigate the risk of ENOSPC on the boot partition by performing automatic cleanup of old kernels and initrds. This paves the way for us to land transaction-specific entries for the offline rollback feature.

blsforme - Boot Management

Automated boot management that just works. Seamlessly handles your EFI System Partition and boot entries without manual intervention.

We’ve improved blsforme to support drop-in command line snippets for kernel parameters, allowing for easy customisation of boot entries. These can be supplied by packages, or by the user directly. This has allowed us to make plymouth work out of the box, in tandem with our prebuilt reproducible initrd images.

We also now support multiple initrd images, which in turn allowed us to land support for early KMS in the initrd for NVIDIA users.

lichen - Installer

We’ve revamped the UI for Lichen, and now allow picking between xfs, ext4, and f2fs for the root filesystem. The team advises against ext4 for systems where more than a few hundred transactions will need to be retained, due to ext4 limitations with hardlink counts.

Sync protections

In preparation for “versioned repositories”, we’ve added a gate to prevent the automatic builds on our infrastructure from directly reaching users. Once we’re happy with our validation, we then “sync” the repositories to the public mirrors. This ensures that users are not exposed to any potential issues with the latest builds.

What’s next?

With us now entering the alpha phase, we’re looking to grow our contributor base and community. Our immediate focus is improving/creating documentation to enable more people to use Serpent OS as a daily driver in the near future. We’re also going to keep iterating on our tooling, ensuring a reliable and efficient system for all users.

We’ll begin deeper integration of the tooling, such as moss with packagekitd for GNOME Software and COSMIC Store use.

Issues and feedback

If you encounter any issues or have feedback to share, please don’t hesitate to reach out to us. You can report bugs on our GitHub issue tracker or join our Matrix space to chat with other users and developers.

We are aware of some known issues in the alpha release, and we’re actively working to address them in future updates. Your feedback is invaluable to us, and we appreciate your help in making Serpent OS the best it can be.

• Performance of moss reduced significantly in VMs on first run
• NVIDIA driver not working on some systems
• Lichen still deliberately limited to “seeing” pre-formatted partitions. Requires GPT disk, with ESP and / partitions pre-created.
• Lichen locked to net install mode, a moss export functionality will be added in the future to address this.
• XBOOTLDR partition (fat32, bls_boot flag in gparted) of 2GB recommended for ideal boot management
• Small repos - deliberate, to ensure maintainer story is solid before we scale up
• Secure boot support pending secure key distribution support within the build infrastructure
• Due to how our atomic upgrades work using renameat2, GNOME Shell doesn’t currently detect new applications (.desktop) files when installed, requiring a reboot. We’re working on a fix for this.

Finally

We’re excited to share the alpha release with you and look forward to your feedback and contributions. Thank you for your continued support and interest in Serpent OS. We’re committed to building a safe, efficient, and reliable system that empowers users to get the most out of their computing experience.

Tags:

• news

Serpent OS Prealpha0 Released

Aug 1, 2024

Ikey

Founder

Well, it didn’t take us that long, really … Our technical preview, prealpha0, is now available for testing!

Head to our download page to grab serpent-os-prealpha-0.iso now!

This is a super rough version of Serpent OS that is capable of being installed on baremetal hardware and VMs that support UEFI and OpenGL acceleration. It is however not recommended for use daily due to the early nature and a bunch of packages being wildly out of date. It features a minimal GNOME desktop with the Firefox web browser and a terminal.

Right now it contains a CLI installer that can be accessed from the terminal by typing:

sudo lichen

To detract from casual use it is necessary to manually partition the disk before running the (net) installer. You can use fdisk to create a GPT disk with a (mandatory) EFI System Partition and an optional XBOOTLDR partition. This currently also needs to be FAT32 until we integrate systemd-boot driver support. Lastly of course you will need a large enough root partition.

You will need a working network connection, so make sure you’re connected before starting the installer!

Technologies involved

This will of course appear to be a very rough (crap) prealpha ISO. Underneath the surface it is using the moss package manager, our very own package management solution written in Rust. Quite simply, every single transaction in moss generates a new filesystem tree (/usr) in a staging area as a full, stateless, OS transaction.

moss

When the package installs succeed, any transaction triggers are run in a private namespace (container) before finally activating the new /usr tree. Through our enforced stateless design, usr-merge, etc, we can atomically update the running OS with a single renameat2 call.

As a neat aside, all OS content is deduplicated, meaning your last system transaction is still available on disk allowing offline rollbacks.

Translated: Like A/B swaps? Don’t like rebooting? … Ok that explained it.

blsforme

A few crates deep we find blsforme - a library for automatically managing the ESP and XBOOTLDR partitions. Whenever kernels and associated files are present in the OS filesystem, they are synchronised to the boot partitions along with automatically generated boot entries and cmdlines. Specifically this means moss can discover and mount necessary partitions according to the disk topology, GPT entries, and Boot Loader Specification, generating configurations by scanning your local rootfs to build the proper root= parameters for you.

Translated: Magic boot management makes way for offline rollbacks.

Warning

Super pre-alpha. Will 100% break! We just wanted you for the journey <3

Start of public iterations

Ok you have this super rough ISO, what next? We now have an actual startpoint and will continue to iterate on the ISOs, delivering new installer updates/improvements and removing the unnecessary live mode from the ISO completely.

The next release will also feature more installer options so you can fresh-install the Cosmic Desktop (in repos now) For the brave, go forth and sudo moss sync -u ! (ok you want moss help first.)

Closing word

As a distro, it’s kinda crap right now. The tooling has been our focus for years and now we can actually build something with it. With only a handful of packages, flatpak is your best friend. Or you could swing a PR into our recipes repo!

The project is only possible with your support. Something tells me that putting out this ISO is going to somewhat increase our hosting costs and stress the datacenter hamsters.

Feel free to sponsor to support our work and increase our capacity!

Tags:

• news

Calm Before The Storm

Apr 30, 2024

Ikey

Founder

It’s that time of month again, and we have some details to share with you on boot management, as well as plans for real installs landing in May. Additionally, plans for “deferred updates” via mandatory reboots have been dropped!

Project priorities

To better understand the technical deep-dive on boot management further below, we should clarify a slight change in priorities for the project as a whole. For some time now we’ve served as a technical beacon for Solus to guide them towards better strategy and technical solutions whilst awaiting a situation where a rebase is feasible.

While we still look forwards to that future, the present day is the most important to us. Serpent OS must now transition from a collection of tools and promises into a daily driver with actual users. We still aim to cater to the more developer/technical oriented end of the spectrum rather than a “desktop distro”, and want to start this journey yesterday.

The boot management code is currently landing over the coming days, and will be immediately followed by an incredibly rudimentary system installer to facilitate an initial adoption of Serpent OS for the general public. Our intent is to ship some early test ISOs for supporters, and by the end of May have a feasible option for people to use and contribute to Serpent OS on real hardware.

We acknowledge this is an exchange of the pursuit of perfection for a pragmatic rapid iteration cycle and believe the best way forward is to open the doors wide, fixing issues on the job. The initial system will likely be rough and buggy, but it is an official starting point from which we can all build Serpent OS together.

Moss + Boulder

Our Rust tooling has now been in “production” use in our development environment and build servers for the last month without any major hiccup. We have recently discovered some minor issues in our emul32 (32-bit) package generation which are being resolved, chiefly our move from the improperly named ISA x86, now updated to 386 affecting 32-bit dependency chains.

Boot

Our main topic of conversation this month is the enabling of boot management for Serpent OS installations. To be crystal clear, we are not talking about a boot loader. In fact, Serpent OS uses systemd-boot for UEFI. Instead, we are talking about the automated management of boot loader entries, their corresponding kernel/initrd assets, cleanup from old states, etc.

In a former life I authored the clr-boot-manager tool to manage this in an agnostic fashion and this served well for a number of years. However, existing as an externally invoked binary meant that the tool had to become responsible for all accountancy and asset lifetimes.

Enter … moss-boot

Ok - technically it’s crates/boot in the moss repo. This is a Rust crate that will be integrated into the main moss project to provide the aforementioned boot management facilities. A huge advantage for us is that moss can only allow a single version of a package to exist within a given state, and all states are numerically identified due to the transaction nature of our package manager.

In simple terms, one “state” == “one kernel candidate” (per type..) so all we really need to do for now is identify the ESP in use by way of the Boot Loader Interface and any XBOOTLDR partition. Then, map our “boot environment” into something moss-boot can convert into boot entries, and voila..

At a more technical level, moss-boot will handle the following:

• form candidate cmdline from filesystem snippets
• auto-generate the root= parameter by probing the / partition (LVM2, etc)
• record the initrd needed
• compare and atomically write “missing” files in FAT-respecting fashion (copy/unlink/rename)
• garbage collect unused boot entries

Live atomic updates and kernels…

An interesting, perhaps not widely considered result of upgrading a kernel, is filesystem path changes. If for any reason the kernel’s module tree becomes inaccessible (say, due to updating to a new version) then it becomes impossible to load the old kernel modules. To remedy this, distributions will tend to leave multiple versions of the kernel on disk, invariably within mounted /boot.

The approach Serpent OS is taking will mean that like Solus, /boot will not need to be premounted nor will any package ship files in that directory. Unlike the existing design in Solus 4, we will not leave kernels behind on upgrade. Instead, they become locked to the system state (transaction) in which they are used.

To solve the “missing version” issue, we will have moss record the qualified snapshot path (in /.moss) within the /run tree so that once the /usr tree has been swapped with a new transaction atomically, a patched kmod package will fall back to probing the suddenly orphaned kernel tree in the cached /run location.

This will mean that the /lib/modules tree may not have the current kernel version, but the OS will still be usable while having had a live atomic update. Of course, to use the new kernel you must reboot. Unlike other atomic OS implementations, it will be up to you when you do so: no more deferred updates!

Next time

We are aiming to get back to monthly blogs on the 19th, but the cycle for the coming 6 weeks may be quite interesting. Expect out of schedule posts looking for testers and volunteers while we use Serpent OS to build Serpent OS, enabling a community of pioneers benefiting from the unique design of our architecture: a full-featured OS featuring hermetic /usr, offline rollbacks, and live atomic upgrades with containerised system triggers.

Tags:

• news

The March Of Progress

Mar 31, 2024

Ikey

Founder

Despite a brief excursion out of the country for a first-in-a-lifetime holiday, I’m happily back at the desk to bring you up to date with the latest goings in Serpent OS. TLDR: Loads of awesome, baremetal is enabled, ISO cycle in next couple of weeks.

Tooling Improvements

OK, we all know you want to know about the baremetal stuff, but c’mon, check out this changeset!

Boulder

We’re officially at a point where the Rust boulder implementation has become our “blessed” build tool. This is actually a crate within the moss repository, allowing the two tools to share huge chunks of code. Our focus over the past few weeks has been to ensure we can drop the tool into our existing build network and “just work”, while making it significantly faster.

One of the most important high level changes is performance, in some cases virtually 50% faster packaging times. Our package payloads are now compressed using multi-threaded zstd compression, and a bunch of internal components were reworked to use faster code paths leading to a significant speed up in the raw generation of each .stone package.

!!! warning “Legacy boulder tool has been retired”

As of March 31st, 2024, the legacy D implementation of `boulder` is no longer supported. The runtime component, `mason`, has been removed from the
volatile repository. Our new tool is written in Rust and contains the `moss` code, enabling a self-contained all-in-one approach to buildroot  management.

It relies on clone-based user namespaces, so is fully "rootless". Please migrate immediately to the new Rust based tool. Not only is it more advanced,
it is significantly faster at generating the raw `.stone` packages.

Other, quite awesome changes:

• Fix manifest encoding for 0 nul byte, interop with build systems
• Explicit --update flag to refresh the repositories prior to build
• Defaults to multithread zstd compression
• Slick timing reports
• Base-enabling landed for “meta” packages (contain no files or content)
• Added BuildRelease control via tooling for automated package rebuilds

Moss

Moss also got some love these past few weeks, including a full migration over to diesel.rs. Combined with a limiting of async to largely network bound operations, and some clever performance optimisations under the hood, moss is now faster than ever. In fact, we’re able to generate desktop ISOs in under 20 seconds, and install the full GNOME desktop to a virtiofs VM in almost no time.

• Dropped some unnecessary dependencies
• Significant code cleanups
• Further refined our system + transaction triggers to fully respect dependency ordering
• Augmented state management with new functionality to activate/rollback to a specific transaction by ID using offline store.
• Various optimisations:
  • Drop expensive Path usage in vfs crate
  • Optimise queries in our transaction management to speed up DAG initialisation and filesystem blit
  • And more to be covered in a future blog post

Hardware/Desktop Enablings.

Rather than extensively widen our repository at this point we’ve opted for further low-level enabling of components required for the developer experience desktop. This includes Network Manager, the desktop kernel package, Vulkan + mesa integration, etc.

As well as enabling core applications and features such as GNOME Software and Flatpak, we’ve also enabled preliminary usage on baremetal booting hardware. Note, this isn’t yet installable but we do plan to offer early access installable images to our sponsors!

Also worth noting.. we’ve begun to package the Cosmic Epoch desktop from the cool guys over at System76. We’re highly interested in this project and plan to track progress closely, offering it within Serpent OS as a fully featured development focused desktop experience.

The Agenda

This post was finished moments before the March deadline, so needless to say we’ve been super busy! Thankfully a lot of pivotal pieces have landed now and we can focus on some big ticket items. Our primary concern is to ship one last “pre-alpha” image for the public, and a test installable pre-alpha ISO for our sponsors to help gain early feedback. The hope is this will help fund a widening of our infrastructure to support more concurrent repository users and our own debuginfod compatible extensions to our repository.

The April (and beyond) agenda:

• Finish pre-alpha stage with some shiny ISOs!
• Enter alpha by adding versioned repository support to moss for an “upgrade forever” guarantee
• Enhance our ABI tracking story to vastly simplify huge rebuild queues and automation
• Expand our native hardware support
• Add some missing basic triggers such as Fedora-compatible SSL certificate management
• Begin work on the all-important boot management in moss! (not boot LOADER, this will be systemd-boot)
• Add battery testing of our tooling to our CI pipelines to protect users
• Ensure all of our code is fully documented
• Greatly improve our onboarding and documentation for packaging and usage of Serpent OS, focusing on stone.yaml in depth
• Get core team running Serpent OS daily. WiFi and GPU support are in place already.
• Build an amazing developer experience

On behalf of the Serpent OS team, I look forward to sharing some amazing updates with you towards the end of April! Keep your eyes peeled for the ISO drops in the coming weeks.

Tags:

• news

End of February Update

Feb 29, 2024

Ikey

Founder

import { Aside } from ‘@astrojs/starlight/components’;

This update came a little later in the month, as we’ve got a lot of exciting news to share. Everything from boulder in Rust, to the GNOME 45 Desktop complete with moss triggers built atop a rebootstrapped toolchain.

Rusty Boulder

We’re pleased to announce that over the course of this weekend, once testing has completed we’ll deploy the latest version of boulder, our packaging build tool. This has been given the Rust treatment, directly sharing the codebase with moss.

Unlike our PoC implementation, the new boulder makes use of the clone syscall to execute portions of itself under user namespaces, eliminating a longstanding deployment irritation of having the main build binary inside packaging.

Just like moss it makes use of tokio for async right where we need it, and is rootless to make life easier for our contributors. Far from being just a direct port of the proof of concept, it adds new tools to empower developers, such as breakpoints in packaging recipes.

Triggers, integrated!

After much discussion we finally integrated support for triggers in moss. These are actions that are executed at different stages of package management operations to finalize or “bake” some state based on the unified components of the installation.

In order for triggers to be useful, they need to be shipped in packages installed on client machines. These are easy-to-write yaml files that live in either /usr/share/moss/triggers/sys.d or within /usr/share/moss/triggers/tx.d, depending on their scope.

Learn More

Note that right now triggers are re-executed for each transaction, and no caching support is yet in place. With that said, they’re still very fast and we plan to add a store based cache to prevent unnecessary execution. One of the greatest benefits of our triggers is ensuring these actions run within the right context under a namespace, or container.

In pursuit of stateless

We’ve iterated many times over the years our belief in the stateless philosophy, and as a design decision we mandated that packages can only contain /usr files. This has meant we’ve had to get a bit inventive in shipping working packages out of the box.

One such strategy has been to abolish the reliance on package triggers for system users and groups. In the past we’ve relied on systemd-sysusers to construct these accounts, such as gdm. A downside is there is no simple way to handle the cleanup of these accounts.

We’re now happily using systemd userdb user and group drop-in records via nss-systemd to provide the default users + groups (including users) group with centrally managed, fixed UIDs and GIDs in our git recipes.

In order to make management of users easier, we employ a specific systemd mechanism that makes drop-in user and group records automatically "appear" to applications that use the proper libc APIs. These are JSON format files that live in `/usr/lib/userdb` and are exposed to applications via `nss`. Learn More

GNOME Desktop

Thanks to triggers now being fully integrated in Serpent OS via moss, we’ve actually managed to package up most of the basic parts of GNOME 45. This includes a functioning GDM, GNOME Shell and a handful of applications.

Packaging work

Outside of the heavy work on the tooling and GNOME, we’ve also been tidying up our recipes repo and recently performed a fresh rebootstrap using:

• glibc 2.39
• binutils 2.42
• GCC 13.2.0
• LLVM 17.4.0
• Rust 1.78.0

Note that our primary toolchain is glibc, lld, clang / clang++, libc++, libunwind, etc. The majority of packages are built with this LLVM + glibc toolchain, and a subset of packages currently build with the GNU toolchain due to various wrong assumptions or use of GCC extensions.

Our kernel has always been built with clang.

Images

Our shortterm plan is to deliver a prealpha image for the general public to test, and to get a “feel” for Serpent OS and associated tooling. As soon as this is out of the door we will immediately pivot to getting Serpent OS fully functioning on baremetal, with explicit targets in mind: the hardware that core developers are currently using.

These images will be available but not promoted for general use - given their intended use as dogfooding systems. This is part of our shift to the oxide-alpha-1 target milestone, which will be discussed further in our next post announcing the prealpha image.

Next time

Our next monthly blog post should return to its regular slot, with updates on the following topics:

• Limiting async in moss to targeted areas (PR ready and pending review)
• Life on Serpent: How baremetal testing + enabling is going.

Until then, please, stay tuned!

Tags:

• news

January Updates

Jan 19, 2024

Ikey

Founder

Precisely one month since our end of year summary, so, what have we been up to? With a new year, a new start was long overdue. We’re pleased to announce that we’ve finally shifted our butts over to our own server (again).

While this isn’t quite as chunky a news update as the last post, we rather hope you appreciate the regularity of the news.

Infrastructure, revisited

‘tarkah’ has been tirelessly porting moss-service, the core foundation of our automated build system, to a shiny new Rust codebase. While we don’t have an immediately pressing need for the port to come online, it will certainly address some “v1 issues” and ensure the best-in-class development experience for our contributors.

Licensing

In our pursuit for a sustainable, community oriented project, we’ve been doing a lot of in house organisation. Very importantly, we’ve now relicensed our packaging recipes to MPL-2.0, bringing them in line with the license chosen for our current engineering efforts (moss, etc).

To reiterate a critical point, we require that our contributions are attributed to the virtual collective “Serpent OS Developers” to ensure we’re unable to relicense any contributions without express consent of the contributors involved.

Additionally we picked MPL-2.0 over Zlib due to some distinct advantages in relation to patent trolls. In combination we believe both the project and our users will have the greatest protection from patent trolls and fiscally-motivated relicensing.

As I Raise My Trigger Finger

As you may recall from the last post, we announced our intention to add system triggers to moss. We also stated that our triggers must run in isolation thanks to the unique architecture of our package management. After much consideration, we’ve landed on a strategy that will work for us.

Post-blit triggers

Every mutable operation in moss results in a fresh internal transaction, which is present as a full /usr tree inside a staging tree. The majority of triggers can be executed here, just before we “activate” the new rootfs. This allows us to detect potential trigger failures, and abort the activation.

These YAML-defined triggers will be run in an isolated environment (clone-based container) with the new /usr and read-only access to /etc.

Post-activation triggers

Once we have a freshly activated /usr (ie new system view) we can run a very limited set of post-activation triggers. An obvious example may be systemctl daemon-reexec. Note that by splitting post-blit and post-activation triggers, we leave the door open to multiple activation strategies for non-simple updates, including soft reboots and kernel patching.

Packaging updates!

With all of the migration work, our builds have also resumed. In the space of a few short days we’ve had 32 package builds. Ok, not a huge number, but we also have 15 outstanding pull requests to our new recipes repo!

In order to make life not only simpler for us, but for Solus in its future role as a source-derivative of Serpent, we’ve unified all of our git recipe repos into a new single repo, recipes.

Improved hosting

Our devops team has been working hard to mitigate bus factor with our transition to the new infrastructure, Bitwarden account, shared DNS, and opentofu configuration.

Self Hosting. Again

As part of the recovery process for Solus, the two projects shared resources, including a large Hetzner node. This has actually worked remarkably well for a long time now, however with Serpent growing we do not wish to impact the availability and stability of the Solus update experience.

Long story short, money from GitHub Sponsors (You guys! < 3) is now paying for an AX52 server from Hetzner. It’s sufficiently powerful that we’re running a backup builder there as well as repo management and build controller.

El Cheapo Web Hosting

Looking at our project’s use of static generators - we decided to review our hosting for web content. Long story short, we deemed it entirely unnecessary. We’re now using GitHub pages to deploy this website and our documentation site via GitHub actions. Don’t worry, we use git, we have backups, and we manage the infrastructure using IaC practices.

Clarifying our baseline target

Lately there have been some questions in both Serpent and Solus as to what our baseline will be. In Serpent OS, this will be at minimum x86_64-v2, with x86_64-v3x packages automatically offered if the system base criteria are met.

This decision has been made to offer the widest baseline compatibility without compromising heavily on system performance (Such that x86_64-generic is unsupported in Serpent OS). We will continue offering both v2 and v3x until it is no longer feasible from a storage/financial perspective. Source derivatives (Including Solus) are free to chart their own course here, if necessary, and still benefit from tight upstream integration.

Towards Next Month

In our next update, we’ll have quite a bit to cover, including:

• Integrated triggers
• Progress with GNOME packaging/readiness
• Boulder / Infrastructure porting progress
• Actually useful Serpent images

Tags:

• news

End Of Year Summary

Dec 19, 2023

Ikey

Founder

Ermo

Co-founder

Burning question - how long before we can use Serpent OS on our development systems? It’s a fair question - and one that requires context to answer. Let’s start our new monthly update cycle with a recap of progress so far.

Introducing tarkah

We’d like to extend a huge official welcome, and thank you, to newest team member Cory Forsstrom (aka tarkah)!

Firstly, a quote from Cory:

I’ve been a long time Solus user and was very excited about what serpent os was doing. Really got invested once I started diving into the D code and seeing how powerful the tools and ideas were. The Rust rewrite was just a perfect storm for me with my experience there and my desire to make contributions. Getting to know you and ermo has just been icing on the cake, you’ve both been so welcoming and friendly. So lots of fun times.

On the personal side, I’m on the west coast in the States, have a lovely wife, just had a baby girl and am enjoying my time with fatherhood and coding

Chances are you know tarkah for his contributions to iced-rs, and to Solus. His contributions to the moss-rs project have been absolutely critical, and his patience essential as we got ourselves quickly up to speed with Rust. It is entirely fair to say that our Rust efforts would not have been possible without him!

Rust Progress

I think it is fair to say that people collectively facepalmed when we announced our plans to adopt Rust - assuming this would be a huge set back. We’re really happy to report this is not the case, and we’ve made tremendous progress in this area.

moss

Our Rust based moss system package tool is now leaps and bounds ahead of its predecessor, supporting the original features and more!

• State management
• Garbage collection of old transactions (via state prune)
• Optimised write order of transactions (see more below)
• File conflict detection
• Parallel fetching and decompression of packages
• Update support (as sync)
• Local and remote repositories with priority-based layering
• Support for building with musl for fully self-contained, static builds of moss with zero system dependencies.

We now have a version-agnostic “stone” crate that is more efficient at reading packages than our original D code. In addition, it benefits from the memory safety promises of Rust.

What is moss?

moss aims to feel like a traditional package manager while offering features typically only found in tools like ostree. It is still package based and offers the same level of granularity and composition as classic package managers while still managing to offer full OS deduplication, offline rollbacks and atomic updates.

At a technical level, we have a custom binary package format that focuses on read performance, deduplication, strong types and error reduction. Every operation in moss creates an entirely new filesystem tree composed of hard links to content addressable storage (Hash-based) which is then “promoted” by swapping the host /usr with the staging tree using the renameat2 system call. Phew!

Virtual filesystem management

OK, lets look at the problem: Every transaction in moss requires us to generate a new staging tree containing all of the directories and files for the /usr tree, using hard links, from content addressable storage. As one might expect, creating tens of thousands of new nodes is really, really slow!

To alleviate this issue we created a vfs crate in moss to optimise how we construct each transaction tree:

• Organise all incoming registered paths into a specific order
• Inject all missing implicitly created directories for accounting
• Re-parent nodes in tree which live under redirected symlinks
• Compile into a recursive data-structure using relative names.

There is a multipart “build” for the tree that detects any filesystem conflicts before any files have been written to disk, preventing broken transactions. With the fully built data structure we can recurse a virtual filesystem, making use of the at family of functions like linkat, mkdirat, to “blit” a filesystem tree without requiring any path resolution, in the most efficient order possible.

The net result? New transaction roots are created in milliseconds, not seconds.

Upgrade support - kind of

moss now offers a sync command in place of the conventional upgrade one might expect. Rather than upgrading to the latest version of a package, we rely on moss and our infrastructure tooling work in conjunction to ensure we can simply synchronize the package selection with the the “tip” candidates in the configured repositories, as ordered by priority. From a distro-builder PoV, we can now revert and pull builds. For a user PoV, you can remove a test repository and moss sync to go back to the official versions of packages.

Generally speaking, when you moss sync you will be seeing new packages, however. :smile: This feature is being built to enable a future feature: exported states. Want to try a new edition? Sync to the lockfile. Need to quickly and reproducibly deploy containers from a locked down configuration? You get the idea.

Fully async

The moss tool is now completely asynchronous, making efficient use of both coroutines and OS threads to perform as much work possible in the shortest space of time. This allows us to download, fetch, check and unpack many packages at the same time without being blocked on network or disk, greatly speeding up the fetch part of a transaction.

This is achieved by building on the famous tokio runtime and :material-github: reqwest crates, among others.

boulder

At the time of writing our port of boulder hasn’t quite yet caught up with the original implementation, given all of the ground we covered with moss. With that said, we decided to step back and evaluate where we could improve upon the first version of boulder.

Recap: What is boulder?

Boulder is the tool we use to generate installable .stone packages from a YAML recipe, stone.yml. The recipe contains machine-readable metadata in a declarative format, along with “steps” (instructions) used to build the package.

Leveraging a series of intelligent patterns and helpers, boulder can analyse and collect the build artefacts into the correct subpackages and automatically discover most runtime dependencies, thus vastly decreasing the workload for the developer.

Learn More

User namespaces (rootless)

Arguably one of the most important changes: boulder no longer requires running as root. Even better, we make use of user namespaces and execute using clone - meaning we no longer need a separate binary (mason) within the container environment!

Superior error handling

Thanks to everything being a self-contained binary, we can tell you up front that you’re using unknown action or definition macros (i.e. %(datadir)) before attempting any builds, saving you valuable time.

Breakpoints

When this was first proposed, I did a double take. Breakpoints.. in recipes… ? No, seriously. Not only was it proposed, but between ermo and tarkah, they only went and implemented it. You can now use special macros within your recipe to add debugging breakpoints in your recipe to drop to a shell within the container and debug the more difficult builds!

Our documentation will be updated when the new boulder ships a stable build.

Unified codebase

We now build both moss and boulder from the moss-rs repository, allowing them to share the same code and mechanisms for dealing with dependencies, system roots, etc., allowing for far higher levels of integration than were previously possible.

A feature we’re currently working on allows you to use a single directory for the system “root”, where all the unpacking, fetching and databases happen, while almost instantly creating new build roots!

What We Already Have

I get it. “All this rewriting - how will you ever make progress” ? It’s a fair question. We didn’t throw anything away, quite the opposite. Our repos, tooling, are all still in use enabling us to build the distribution in parallel to the ongoing rewrites.

Build infrastructure

For a long time, we’ve had our :material-view-dashboard: dashboard up and running for our build infrastructure. Sure, we’re going to rewrite it beyond the “Proof of concept” stage when required - but for now it still serves us well. Admittedly there is a deep issue within the druntime causing thread leaks over time, but we have it restarting via systemd every 24 hours as a bandaid. Needless to say, we’re big fans of memory safety now.

Our current deployment watches git repos in our :material-github: snekpit collection, and automatically forms build queues from the pending builds, ordering them by dependency. Using two of ermo’s beefy builders, the packages are automatically built and deployed to our repository. Our workflow means that maintainers only have to merge a pull request for builds to appear in the volatile repository.

Legacy tooling

Despite having some warts, the build infrastructure is still able to use legacy moss and boulder to make our new builds available, and we’ve been minimally patching these old tools to support our transition to the Rust based tooling. As of this moment, moss-rs officially supersedes the D version, and our new boulder is quickly catching up.

Long story short, we’re still building on all fronts and aren’t blocked anywhere.

Live repository

Builds are published to our volatile repository, and we do have a fully functioning software repository despite being quite small. Once we’re happy with the new moss we’ll speed up in our packaging efforts. With that said, we do have two kernels (kvm and desktop) and even gtk-4 at this early stage!

The immediate future

OK let’s tldr this:

• Rust ports are coming along fantastically
• moss has been replaced
• still making use of existing tools and infrastructure to build the OS
• actually building the fully bootstrapped, self-contained OS, built on libc++, clang, glibc, etc.
• actively pushing updates to binary repository with fully automated pipeline
• new stuff is :rocket: blazing fast

System triggers

We’re experimenting with system triggers right now, execution units that run to fully bake the OS state after each transaction completes. In keeping with our stateless philosophy (and pending adoption of hermetic /usr), we need triggers that understand the union state of the operating system, not individual per-package triggers.

A YAML-based trigger recipe has been baked up, which deliberately avoids the use of shell helpers and execution security weaknesses by using an extended variant of globs:

    let pattern = "/usr/lib/modules/(version:*)/*".parse::<fnmatch::Pattern>()?;

Our new :material-github: fnmatch crate, heavily inspired by Python fnmatch, compiles specially format glob strings to Regex, extending with capture groups for named variables. In YAML, it looks a bit like this:

handlers:
    depmod:
        run: /sbin/depmod
        args: ["-a", "$(version)"]

## Link paths to handlers and filter on the type
paths:
    "/usr/lib/modules/(version:*)/kernel" :
        handlers:
            - depmod
        type: directory

Once we roll out the trigger support, we unblock the packaging of critical items like gtk-3 for gdm, as well as enable usable installations and ISOs. Note that they differ from classical triggers due to the architecture of moss: they need to run in an isolated namespace (container) so we can ensure the staged transaction won’t break. Additionally we cache and collect the trigger artefacts to speed up each filesystem operation, giving us a first: versioned state artefacts.

Boot management

Our plan is to build on my prior work in the Solus boot management design and :material-github: clr-boot-manager, addressing some long standing shortcomings. Most importantly, our new module will not need to inspect or manage the OS filesystem as moss will be able to provide all of the relevant information (full accounting for all used paths in all transactions).

Initially we will focus on UEFI and supporting the :material-github: Discoverable Partitions Specification by way of XBOOTLDR partitions, the ESP and the Boot Loader Interface. Currently we have no plans to support Unified Kernel Images as the approach taken by CBM (and soon, moss) alleviates the data concerns of dealing with vfat. However, as and when UKIs gain configurable, standardised behaviour for cmdline control we will investigate their use. Until that point please note we prebuild our initrd images and ship them directly in our packages, as Solus has done so for years already.

The net gain for taking control of boot management will be the deduplication and garbage collection of assets installed to either the ESP or XBOOTLDR partitions, along with generation of boot entries relevant to Serpent OS: Transaction specific entries allowing us to directly boot back to older installs in case upgrades go wrong.

Last, but certainly not least, this approach will make dual-booting with another OS less irritating by no longer being bound by a fixed ESP size.

Desktop

Our plan is to produce a GNOME Shell based live ISO containing the bare basics to get everything else validated, and open the path to installations. In short, it will include:

• Desktop.
• moss tooling
• web browser (firefox)
• terminal (potentially gnome-console)
• flatpak
• Eventually: GNOME Software (integrated with flatpak+moss)

It should be obvious we’re not building a general purpose home-use distribution. However, we also won’t stop people using Serpent OS for their needs, and we plan to beef up the repository, fully support flatpak and eventually custom moss source recipe repos. (ala AUR)

Safety is important

Our journey over the last few years through D, DIP1000, Rust and memory safety has been incredibly eye opening for us. To our own surprise, memory safety has become a huge interest and concern for the team. Not only have we embarked on a full transition of our tooling to Rust, we’ve started looking at the OS itself.

As an early indication of our intent, our curl package is now built with the rustls backend and not OpenSSL. Needless to say, we’re very keen to reduce the surface area for attacks by adopting alternative, safer implementations where possible. Other projects we’re keeping a very close eye on include:

• :material-github: sudo-rs
• :material-github: uutils/coreutils

Looking to next year

With most of us entering our holiday leave, and the new year practically around the corner, the Serpent OS team wishes you many happy returns and a wonderful new year. With one blocker being worked on (triggers), and the last on the horizon (boot management), your dream installation of Serpent OS is firmly within reach.

In the spirit of the season… if you feel like supporting our project, you can help with the rising cost of living and our expanding hosting requirements via GitHub Sponsors!

Sponsor us!

Tags:

• news

Oxidised Moss

Sep 6, 2023

Ikey

Founder

Allow me to start this post by stating something very important to me: I absolutely love the D programming language, along with the expressivity and creative freedom it brings. Therefore please do not interpret this post as an assassination piece.

For a number of months now the Serpent OS project has stood rather still. While this could naively be attributed to our shared plans with Solus - a deeper, technical issue is to be acredited.

D isn’t quite there yet. But it will be, some day.

Again, allow me to say I have thoroughly enjoyed my experience with D over the last 3 or so years, it has been truly illuminating for me as an engineer. With that said, we have also become responsible for an awful lot of code. As an engineering-led distribution + tooling project, our focus is that of secure and auditable code paths. To that extent we pursued DIP1000 as far as practical and admit it has a way to go before addressing our immediate needs of memory safety.

While we’re quite happy to be an upstream for Linux distributions by way of release channels and tooling releases, we don’t quite have the resources to also be an upstream for the numerous D packages we’d need to create and maintain to get our works over the finish line.

With that said, I will still continue to use D in my own personal projects, and firmly believe that one day D will realise its true potential.

Rewarding contributors

Our priorities have shifted somewhat since the announcement of our shared venture with Solus, and we must make architectural decisions based on the needs of all stakeholders involved, including the existing contributor pool. Additionally, care should be taken to be somewhat populist in our choice of stacks in order to give contributors industry-relevant experience to add to their résumé (CV).

Playing to strengths

Typically Solus has been a Golang-oriented project, and has a number of experienced developers. With the addition of the Serpent developers, the total cross-over development team has a skill pool featuring Rust and Go, as well as various web stack technologies.

Reconsidering the total project architecture including our automated builds, the following decisions have been made that incorporate the requirements of being widely adopted/supported, robust ecosystems and established tooling:

• Rust, for low level tooling and components. Chiefly: moss, boulder, libstone
• ReactJS/TypeScript for our frontend work (Summit Web)
• Go - for our web / build infrastructure (Summit, Avalanche, Vessel, etc)

Lets start… 2 days ago!

The new infrastructure will be brought up using widely available modules, and designed to be scalable from the outset as part of a Kubernetes deployment, with as minimal user interaction as needed. Our eventual plans include rebuilding the entire distribution from source with heavy caching once some part of the dependency graph changes.

This infrastructure will then be extended to support the Solus 4 series for quality of life improvements to Solus developers, enabling a more streamlined dev workflow: TL;DR less time babysitting builds = more Serpent development focus.

Our priority these past few days has been on the new moss-rs repository where we have begun to reimplement moss in Rust. So far we have a skeleton CLI powered by clap with an in-progress library for reading .stone archives, our custom package format.

The project is organised as a Rust workspace, with the view that stone, moss and boulder will all live in the same tree. Our hope is this vastly improves the onboarding experience and opens the doors (finally) to contributors.

Licensing

It should also be noted that the new tooling is made available under the terms of the Mozilla Public License (MPL-2.0). After internal discussion, we felt the MPL offered the greatest level of defence against patent trolls while still ensuring our code was widely free for all to respectfully use and adapt.

Please also note that we have always, and continue to deliberately credit copyright as:

Copyright © 2020-2023 Serpent OS Developers

This is a virtual collective consisting of all whom have contributed to Serpent OS (per git logs) and is designed to prevent us from being able to change the license down the line, i.e. a community protective measure.

Glad to be back

Despite some sadness in the change of circumstances, we must make decisions that benefit us collectively as a community. Please join us in raising a virtual glass to new pastures, and a brave new blazing fast 🚀 (TM) future for Serpent OS and Solus 5.

Cheers!

Tags:

• news

Snakes On A Boat

Apr 21, 2023

Ikey

Founder

We had intended to get a blog post out a little bit quicker, but the last month has been extremely action packed. However, it has paid off immensely. As our friends at Solus recently announced it is time to embark on a new voyage.

General gist of it

There is very little point in rehashing the post, so I’ll cover the essentials

• We assisted the Solus team in deploying a complete new infrastructure. Some of this work is ongoing
• The package delivery pipeline is in place, meaning updates will soon reach users.
• We’re working towards a mutually beneficial future whereby Solus adopts our tooling.

Is Serpent OS over?

Oh no, no. The most practical angle to view this from is that Serpent OS is free to build and innovate to create the basis of Solus 5. Outside of the distribution we’re still keen to continue development on our tooling and strategies.

It is therefore critical that we continue development, and find the best approach for both projects, to make the transition to Solus 5 as seamless as possible. To that end we will still need to produce ISOs and have an active community of users and testers. During this transition period, despite being two separate projects, we’re both heading to a common goal and interest.

Finances

We have an exciting journey ahead for all of us, and there are many moving parts involved. Until we’re at the point of mutual merge, we will keep the entities and billing separate. Thus, funds to the Solus OpenCollective are intended for use within Solus, whereas we currently use GitHub Sponsors for our own project needs.

Currently Solus and Serpent OS share one server, which was essential for quick turnaround on infrastructure enabling. At the end of this month Serpent OS will migrate from that server to a new, separate system, ensuring the projects are billed separately.

Long story short, if you wish to sponsor Serpent OS specifically, please do so via our GitHub sponsors account as our monthly running costs will immediately rise at the end of this month. If you’re supporting Solus development, please do visit them and sponsor them =)

Next steps?

Glad you asked! Now that the dust is settling, we’re focusing on Serpent OS requirements, and helping Solus where we can. Our immediate goals are to build a dogfooding system for a small collection of developers to run as an unsupported prealpha configuration, allowing us to flesh out the tooling and processes.

This will include a live booting GNOME ISO, and sufficient base packages to freely iterate on moss, boulder, etc as well as our own infrastructure. Once we’ve attained a basic quality and have an installer option, those ISOs will be made available to you guys!

Tags:

• news

Infrastructure Launched!

Mar 18, 2023

Ikey

Founder

After many months and much work, our infrastructure is finally online. We’ve had a few restarts, but it’s now running fully online with 2 builders, ready to serve builds around the clock.

Firstly, I’d like to apologise for the delay since our last blog post. We made the decision to move this website to static content, which took longer than expected. We’re still using our own D codebase, but prebuilding the site (and fake “API”) so we can lower the load on our web server.

Introducing the Infra

Our infrastructure is composed of 3 main components.

Summit

This is the page you can see over at dash.serpentos.com. It contains the build scheduler. It monitors our git repositories, and as soon as it discovers any missing builds it creates build tasks for them. It uses a graph to ensure parallel builds happen as much as possible, and correctly orders (and blocks) builds based on build dependencies.

Avalanche

We have 2 instances of Avalanche, our builder tool, running. This accepts configuration + build requests from Summit, reporting status and available builds.

Vessel

This is an internal repository manager, which accepts completed packages from an Avalanche instance. Published packages land at dev.serpentos.com

Next steps

Super early days with the infra but we now have builds flowing as part of our continuous delivery solution. Keeping this blog post short and sweet… we’re about to package GNOME and start racing towards our first real ISOs!

Don’t forget

If you like what the project is doing, don’t forget to sponsor!

Tags:

• news

Lift Off

Dec 24, 2022

Ikey

Founder

Enough of this “2 years” nonsense. We’re finally ready for lift off. It is with immense pleasure we can finally announce that Serpent OS has transitioned from a promise to a deliverable. Bye bye, phantomware!

We exist

As mentioned, we spent 2 years working on tooling and process. That’s .. well. Kinda dull, honestly. You’re not here for the tooling, you’re here for the OS. To that end I made a decision to accelerate development of the actual Linux distro - and shift development of tooling into a parallel effort.

Infrastructure .. intelligently deferred

I deferred final enabling of the infrastructure until January to rectify the chicken/egg scenario whilst allowing us to grow a base of contributors and an actual distro to work with. We’re in a good position with minimal blockers so no concern there.

A real software collection

This is our term for the classical “package repository”. We’re using a temporary collection right now to store all of the builds we produce. In keeping with the Avalanche requirements, this is the volatile software collection. Changes a lot, hasn’t got a release policy.

A community.

It goes without saying, really, that our project isn’t remotely possible without a community. I want to take the time to personally thank everyone that stepped up to the plate lately and contributed to Serpent OS. Without the work of the team, in which I include the contributors to our venom recipe repository, an ISO was never possible. Additionally contributions to tooling has helped us make significant strides.

It should be noted we’ve practically folded our old “team” concept and ensured we operate across the board as a singular community, with some members having additional responsibilities. Our belief is all in the community have equal share and say. With that said, to the original “team”, members both past and present, I thank for their (long) support and contributions to the project.

An ISO.

We actually went ahead and created our first ISO. OK that’s a lie, this is probably the 20th revision by now. And let’s be brutally honest here:

It sucks.

We expected no less. However, the time is definitely here for us to begin our public iteration, transitioning from suckness to a project worth using. In order to do that, we need to get ourselves to a point whereby we can dogfood our work and build a daily driver. Our focus right now is building out the core technology and packaging to achieve those aims.

So if you want to try our uninstallable, buggy ISO, chiefly created as a brief introduction to our package manager and toolchain, head to our newly minted Download page. Set your expectations low, ignore your dreams, and you will not be disappointed!

All jokes aside, it took a long time to get to point where we could even construct our first, KVM-focused, UEFI-only snekvalidator.iso. We now have a baseline to improve on, a working contribution process, and a booting, self-hosting system.

The ISO is built using 2 layered collections, the protosnek collection containing our toolchain, and the new volatile collection. Much of the packaging work has been submitted by venom contributors and the core team. Note you can install neofetch which our very own Rune Morling (ermo) patched to support the Serpent OS logo.

Boot it in Qemu (or certain Intel laptops) and play with moss now! Note, this ISO is not installable, and no upgrade path exists. It is simply the beginnings of a public iteration process.

Next steps

In January we’ll launch our infrastructure to scale out contributions as well as to permit the mass-rebuilds that need to happen. We have to enable our -dbginfo packages and stripping, which were disabled due to a parallelism issue. We need to introduce our boot management based around systemd-boot, provide more kernels, do hardware enabling, introduce moss-triggers, and much more. However, this is a pivotal moment for our project as we’ve finally become a real, if not sucky, distro. The future is incredibly bright, and we intend to deliver on every one of our promises.

As always, if you want to support our development, please consider sponsoring the work, or engaging with the community on Matrix or indeed our forums.

You can discuss this blog post, or leave feedback on the ISO, over at our forums.

Tags:

• news

The Big Update

Sep 14, 2022

Ikey

Founder

Well - we’ve got some big news! The past few weeks have been an incredibly busy time for us, and we’ve hit some major milestones.

Funding

After much deliberation - we’ve decided to pull out of Open Collective. Among other reasons, the fees are simply too high and severely impact the funds available to us. In our early stages, the team consensus is that funds generated are used to compensate my time working on Serpent OS.

As such I’m now moving funding to my own GitHub Sponsors page - please do migrate! It ensures your entire donation makes it and keeps the lights on for longer =) Please remember I’m working full time on Serpent OS exclusively - I need your help to keep working.

Moved to GitHub

We’ve pretty much completed our transition to GitHub. We’ve now got the following organisations:

• snekpit - Packaging work
• serpent-os - Main code development (moss/boulder/etc)

Forums

Don’t forget - our forums are live over at forums.serpentos.com - please feel free to drop in and join in with the community =)

Rehash on the tooling

OK so what exactly are moss and boulder? In short - they’re the absolute core pieces of our distribution model.

”What is moss?”

On the surface, moss looks and feels roughly the same as just about any other traditional package manager out there. Internally, however, its far more modern and has a few tricks up its sleeve. For instance, every time you initiate an operation in moss, be it installation, removal, upgrade, etc, a new filesystem transaction is generated. In short, if something is wrong with the new transaction - you can just boot to an older transaction when things worked fine.

Now, it’s not implemented using bundles or filesystem specific functionality, internally its just intelligent use of hardlinks and deduplication policies, and we have our own container format with zstd based payload compression. Our strongly typed, deduplicating binary format is what powers moss.

Behind the scenes we also use some other cool technology, such as LMDB for super reliable and speedy database access. The net result is a next generation package management solution that offers all the benefits of traditional package managers (i.e. granularity and composition) with new world features, like atomic updates, deduplication, and repository snapshots.

boulder

It’s one thing to manage and install packages, it’s another entirely to build them. boulder builds conceptually on prior art such as pisi and the package.yml format used in ypkg. It is designed with automation and ease of integration in mind, i.e. less time spent focusing on packaging and more time on actually getting the thing building and installing correctly.

Boulder supports “macros” as seen in the RPM and ypkg world, to support consistent builds and integration. Additionally it automatically splits up packages into the appropriate subpackages, and automatically scans for binary, pkgconfig, perl and other dependencies during source analysis and build time. The end result is some .stone binary packages and a build manifest, which we use to flesh out our source package index.

Major moss improvements

We’ve spent considerable time reworking moss, our package manager. It now features a fresher (terminal) user interface, progress bars, and is rewritten to use the moss-db module encapsulating LMDB.

It’s also possible to manipulate the binary collections (software repositories) used by moss now. Note we’re going to rename “remote” to “collection” for consistency.

At the time of writing:

$ mkdir destdir
$ sudo moss remote add -D destdir protosnek https://dev.serpentos.com/protosnek/x86_64/stone.index
$ sudo moss install -D destdir bash dash dbus dbus-broker systemd coreutils util-linux which moss nano
$ sudo systemd-nspawn -b -D destdir

This will be simplified once we introduce virtual packages (Coming Soon ™)

Local Collections

Boulder can now be instructed to utilise a local collection of stone packages, simplifying the development of large stack items.

sudo boulder bi stone.yml -p local-x86_64

Packages should be moved to /var/cache/boulder/collections/local-x86_64 and the index can be updated by running:

sudo moss idx /var/cache/boulder/collections/local-x86_64

Self Hosting

Serpent OS is now officially self hosting. Using our own packages, we’re able to construct a root filesystem, then within that rootfs container we can use our own build tooling (boulder) to construct new builds of our packages in a nested container.

The protosnek collection has been updated to include the newest versions of moss and boulder.

Booting

As a fun experiment, we wanted to see how far along things are. Using a throwaway kernel + initrd build, we were able to get Serpent OS booting using virtualisation (qemu)

ISO When?

Right now everyone is working in the snekpit organisation to get our base packaging in order. I’m looking to freeze protosnek, our bootstrap collection, at the latest of tomorrow evening.

We now support layered, priority based collections (repositories) and dependency solving across collection boundaries, allowing us to build our new main collection with protosnek acting as a bootstrap seed.

Throughout this week, I’ll focus on getting Avalanche, Summit and Vessel into shape for PoC so that we can enjoy automated builds of packages landing in the yet-to-be-launched volatile collection.

From there, we’re going to iterate + improve packaging, fixing bugs and landing features as we discover the issues. Initially we’ll look to integrate triggers in a stateless-friendly fashion (our packages can only ship /usr by design) - after that will come boot management.

An early target will be Qemu support via a stripped linux-kvm package to accelerate the bring up, and we encourage everyone to join in the testing. We’re self hosting, we know how to boot, and now we’re able to bring the awesome.

I cannot stress how important the support to the project is. Without it - I’m unable to work full time on the project. Please consider supporting my development work via GitHub Sponsors.

I’m so broke they’ve started naming black holes after my IBAN.

Thank you!

You can discuss this blog post over on our forums

Tags:

• news

Infrastructure Update

Jul 10, 2022

Ikey

Founder

Since the last post, I’ve pivoted to full time work on Serpent OS, which is made all the more possible thanks to everyone supporting us via OpenCollective <3.

We’ve been working towards establishing an online infrastructure to support the automation of package builds, while revisiting some core components.

moss-db

During the development of the Serpent OS tooling we’ve been exploring the possibilities of D Lang, picking up new practices and refining our approach as we go. Naturally, some of our older modules are somewhat … smelly. Most noticeable is our moss-db module, which was initially intended as a lightweight wrapper around RocksDB.

In practice that required an encapsulation API written in D around the C API, and our own wrapping on top of that. Naturally, it resulted in a very allocation-heavy implementation that just didn’t sit right with us, and due to the absurd complexity of RocksDB was still missing quite a few features.

Enter LMDB

We’re now using the Lightning Memory-Mapped Database as the driver implementation for moss-db. In short, we get rapid reads, ACID transactions, bulk inserts, you name it. Our implementation takes advantage of multiple database indexes (MDB_dbi) in LMDB to partition the database into internal components, so that we can provide “buckets”, or collections. These internal DBs are used for bucket mapping to permit a key-compaction strategy - iteration of top level buckets and key-value pairs within a bucket.

Hat tip, boltdb

The majority of the API was designed with the boltdb API in mind. Additionally it was built with -preview=dip1000 and -preview=in enabled, ensuring safely scoped memory use and no room for memory lifetime issues. While we prefer the use of generics, the API is built with immutable(ubyte[]) as the internal key and value type.

Custom types can simply implement mossEncode or mossDecode to be instantly serialisable into the database as keys, values or bucket identifiers.

Example API usage:

Database db;
/* setup the DB with lmdb:// URI */

/* Write transaction */
auto err = db.update((scope tx) @safe
{
    auto bucket = tx.bucket("letters");
    return tx.set(bucket, "a", 1);
});

/* do something with the error */

err = db.view((in tx) @safe
{
    foreach (name, bucket ; tx.buckets!int)
    {
        foreach (key, value ; tx.iterator!(string,string)(bucket))
        {
            /* do something with the key value pairs, decoded as strings */
        }
    }

    /* WILL NOT COMPILE. tx is const scope ref :) */
    tx.removeBucket("numbers");

    return NoDatabaseError;
}

Next for moss

Moss will be ported to the new DB API and we’ll gather some performance metrics, while implementing features like expired state garbage collection (disk cleanup), searching for names/descriptions, etc.

Avalanche

Avalanche is a core component of our upcoming infrastructure, providing the service for running builds on a local node, and a controller to coordinate a group of builders.

Summit will be the publicly accessible project dashboard, and will be responsible for coordinating incoming builds to Avalanche controllers and repositories. Developers will submit builds to Summit and have them dispatched correctly.

So far we have the core service process in place for the Controller + Node, and now we’re working on persistence and handshake. TLDR; fancy use of moss-db and JSON Web tokens over mandated SSL. This means our build infra will be scalable from day 1 allowing multiple builders to be online very early on.

Timescale

We’re planning to get an early version of our infrastructure up and running within the next 2 weeks, and get builds flowing =)

Tags:

• news

Packaging Automation, Next Steps

Jun 22, 2022

Ikey

Founder

Hot damn we’ve been busy lately. No, really. The latest development cycle saw us focus exclusively on boulder, our build tooling. As of today it features a proof of concept boulder new subcommand for the automatic generation of packaging templates from an upstream source (i.e. tarball).

Before we really start this blog post off, I’d like to thank everyone who is supporting the project! All of the OpenCollective contributions will make it easier for me to work full time on Serpent OS =) Much love <3

But, but build flavours …

Alright you got me there, certain projects prefer to abstract the configuration, build and installation of packages and be provided with some kind of hint to the build system, i.e. manually setting autotools, etc.

Serpent OS packaging is declarative and well structured, and relies on the use of RPM-style “macros” for distribution integration and common tasks to ensure consistent packaging.

We prefer a self documenting approach that can be machine validated rather than depending on introspection at the time of build. Our stone.yml format is very flexible and powerful, with automatic runtime dependencies and package splitting as standard.

..Doesn’t mean we can’t make packaging even easier though.

Build discovery

Pointing boulder at an upstream source will perform a deep analysis of the sources to determine the build system type, build dependencies, metadata, licensing etc. Right now it’s just getting ready to leave POC stage so it has a few warts, however it does have support for generating package skeletons for the following build systems:

• cmake
• meson
• autotools

We’re adding automation for Perl and Python packaging (and Golang, Rust, etc) so we can enforce consistency, integration and ease without being a burden on developers. This will greatly reduce the friction of contribution - allowing anyone to package for Serpent OS.

We’re also able to automatically discover build time dependencies during analysis and add those to the skeleton stone.yml file. We’ll enhance support for other build systems as we go, ensuring that each new package is as close to done on creation as possible, with review and iteration left to the developer.

License compliance

A common pain in the arse when packaging for any Linux distribution is ensuring the package information is compliant in terms of licensing. As such we must know all of the licensing information, as well as FSF and OSI compliance for our continuous integration testing.

…Finding all of that information is truly a painful process when conducted manually. Thankfully boulder can perform analysis of all licensing files within the project to greatly improve compliance and packaging.

Every license listed in a stone.yml file must use a valid SPDX identifier, and be accurate. boulder now scans all license files, looking for matches with both SPDX IDs as well as fuzzy-matching the text of input licenses to make a best-guess at the license ID.

This has so far been highly accurate and correctly identifies many hundreds of licenses, ensuring a compliant packaging process with less pain for the developers. Over time we’ll optimise and improve this process to ensure compliance for our developers rather than blocking them.

As of today we support the REUSE specification for expressing software licenses too!

Next on the list

The next steps are honest-to-goodness exciting for us. Or should I say.. exiting?

bill

Work formally begins now on Bootstrap Bill (Turner). Whilst we did successfully bootstrap Serpent OS and construct the Protosnek repository, the process for that is not reproducible as boulder has gone through massive changes in this time.

The new project will leverage boulder and a newly designed bootstrap process to eliminate all host contamination and bootstrap Serpent OS from stone.yml files, emitting an immutable bootstrap repository.

Layering support will land in moss and boulder to begin the infrastructure projects.

Build Submission (“Let’s Roll”)

The aim is to complete bill in a very short time so we can bring some initial infrastructure online to facilitate the automatic build of submitted build jobs. We’ll use this process to create our live repository, replacing the initial bootstrap repository from bill.

At this point all of the tooling we have will come together to allow us all to very quickly iterate on packaging, polish up moss and race towards installed systems with online updates.

Tags:

• news

A Word From The Founder

Jun 6, 2022

Ikey

Founder

Well well, it’s been a long time since I personally wrote a post.. :) So let’s keep this short and sweet, shall we? I’m returning to full time work on Serpent OS.

The 6th of July will be my last day at my current employment having tendered my 30 day notice today. Despite having enjoyment at my current position, the reality is that my passion and focus is Serpent OS.

I’m now in a transition process and will ramp up my efforts with Serpent OS. Realistically I need to reduce the outgoing costs of the project and with your help I can gain some level of financial support as we move through the next stages of development. Worst case, I will only take on any part-time or contractual gigs, allowing my primary focus to be Serpent OS.

I’ll begin accelerating works and enabling community contribution so we can get the derailed-alpha train back on the tracks.

I have absolute faith in this project, the community and our shared ability to deliver the OS and tooling. To achieve it will require far more of my time and I’m perfectly willing to give it.

Thank you all to everyone who has been supporting the project, it is now time to deliver. Not just another run of the mill distribution but a technically competent and usable distribution that is not only different but better.

Let’s do this in the most grassroots and enjoyable way possible =)

Tags:

• news

It All Depends

Nov 23, 2021

Ikey

Founder

It all depends.. it really does. On shared libraries.. interpreters.. pkg-config providers and packages. It’s the same story for all “package managers”, how do we ensure that the installed software has everything it needs at runtime?

Our merry crew has been involved in designing and building Linux distributions for a very, very long time, so we didn’t want to simply repeat history.

Thanks to many improvements across our codebase, including moss-deps, we automatically analyse the assets in a package (rapidly too) to determine any dependencies we can add without requiring the maintainer to list them. This is similar in nature to RPM’s behaviour.

As such we encode dependencies into our (endian-aware, binary) format which is then stored in the local installation database. Global providers are keyed for quick access, and the vast majority of packages will not explicitly depend on another package’s name, rather, they’ll depend on a capability or provider. For subpackage dependencies that usually depend on “NEVRA” equality (i.e. matching against a name with a minimum version, release, matching epoch and architecture), we’ll introduce a lockstep dependency that can only be resolved from its origin source (repo).

Lastly, we’ll always ensure there is no possibility for “partial update” woes. With these considerations, we have no need to support >= style dependencies, and instead rely on strict design goals and maintainer responsibility.

First and foremost!

The rapid move we’re enjoying from concept, to prototype, and soon to be fully fledged Linux distribution, is only possible with the amazing community support. The last few months have seen us pull off some amazing feats, and we’re now executing on our first public milestones. With your help, more and more hours can be spent getting us ready for release, and would probably help to insulate my shed office! (Spoiler: its plastic and electric heaters are expensive =))

The Milestones

We have created our initial milestones that our quite literally our escape trajectory from bootstrap to distro. We’re considerably closer now, hence this open announcement.

v0.0: Container (systemd-nspawn)

Our first release medium will be a systemd-nspawn compatible container image. Our primary driver for this is to allow us to add encapsulation for our build tool, boulder, permitting us to create distributable builder images to seed our infrastructure and first public binary repository.

v0.1: Bootable “image”

Once our build infra is up and running (honestly a lot of work has been completed for this in advance) we’ll work towards our first 0.1 image. This will initially target VM usage, with a basic console environment and tooling (moss, boulder, etc).

And then..

We have a clear linear path ahead of us, with each stage unlocking the next. During the development of v0.0 and v0.1 we’ll establish our build and test infrastructure, and begin hosting our package sources and binaries. At this point we can enter a rapid development cycle with incremental, and considerable improvements. Such as a usable desktop experience and installer.. :)

Recent changes

I haven’t blogged in quite a while, as I’ve been deep in the trenches working on our core features. As we’ve expressed before, we tend to work on the more complex systems first and then glue them together after to form a cohesive hole. The last few days have involved plenty of glue, and we now have distinct package management features.

Refactor

• Replaced defunct InstallDB with reusable MetaDB for local installation of archives as well as forming the backbone of repository support.
• Added ActivePackagesPlugin to identify installed packages
• Swapped non cryptographic hash usage with xxhash

Dependencies

• Introduced new Transaction type to utilise a directed acyclical graph for dependency solving.
• Reworked moss-deps into plugins + registry core for all resolution operations.
• Locally provided .stone files handled by CobblePlugin to ensure we depsolve from this set too.
• New Transaction set management for identifying state changes and ensuring full resolution of target system state.
• Shared library and interpreter (DT_INTERP) dependencies and producers automatically encoded into packages and resolved by depsolver.

Package Installation

We handle locally provided .stone packages passed to the install command identically to those found in a repository. This eliminates a lot of special casing for local archives and allows us to find dependencies within the provided set, before looking to the system and the repositories.

Dependency resolution is performed now for our package installation and is validated at multiple points, allowing a package like nano to depend on compact automatic dependencies:

  Dependency(DependencyType.SharedLibraryName, "libc.so.6(x86_64)");

Note our format and database are binary and endian aware. The dependency type only requires 1 byte of storage and no string comparisons.

List packages

Thanks to the huge refactor, we can now trivially access the installed packages as a list. This code will be reused for a list available command in future.

Example list installed output:

                   file (5.4) - File type identification utility
             file-32bit (5.4) - Provides 32-bit runtime libraries for file
       file-32bit-devel (5.4) - Provides development files for file-32bit
             file-devel (5.4) - Development files for file
                   nano (5.5) - GNU Text Editor

Inspect archives

For debugging and development purposes, we’ve moved our old “info” command to a new “inspect” command to work directly on local .stone files. This displays extended information on the various payloads and their compression stats.

For general users - the new info command displays basic metadata and package dependencies.

Package Removal

Upon generating a new system state, “removed” packages are simply no longer installed. As such no live mutation is performed. As of today we can now request the removal of packages from the current state, which generates a new filtered state. Additionally we remove all reverse dependencies, direct and transitive. This is accomplished by utilising a transposed copy of the directed acyclical graph, identifying the relevant subgraph and occluding the set from the newly generated state.

Lastly

The past few weeks have been especially enjoyable. I’ve truly had a fantastic time working on the project and cannot wait for the team and I to start offering our first downloads, and iterate as a truly new Linux distribution that borrows some ideas from a lot of great places, and fuses them into something awesome.

Keep toasty - this train isn’t slowing down.

Tags:

• news

A Rolling Boulder Gathers No Moss

Aug 10, 2021

Ikey

Founder

We actually did it. Super pleased to announce that moss is now capable of installing and removing packages. Granted, super rough, but gotta start somewhere right?

OK let’s recap. A moss archive is super weird, and consists of multiple containers, or payloads. We use a strongly typed binary format, per-payload compression (Currently zstd), and don’t store files in a typical archive fashion.

Instead a .stone file (moss archive) has a Content Payload, which is a compressed “megablob” of all the unique files in a given package. The various files contained within that “megablob” are described in an IndexPayload, which simply contains some IDs and offsets, acting much like a lookup table.

That data alone doesn’t actually tell us where files go on the filesystem when installed. For that, we have a specialist Layout Payload, encoding the final layout of the package on disk.

As can be imagined, the weirdness made it quite difficult to install in a trivial fashion.

Databases

Well, persistence really. Thanks to RocksDB and our new moss-db project, we can trivially store information we need from each package we “precache”. Primarily, we store full system states within our new StateDB, which at present is simply a series of package ID selections grouped by a unique 64-bit integer.

Additionally we remember the layouts within the LayoutDB so that we can eventually recreate said layout on disk.

Precaching

Before we actually commit to an install, we try to precache all of the stone files in our pool. So we unpack the content payload (“megablob”), split it into various unique files in the pool ready for use. At this point we also record the Layouts, but do not “install” the package to a system root.

Blitting

This is our favourite step. When our cache is populated, we gather all relevant layouts for the current selections, and then begin applying them in a new system root. All directories and symlinks are created as normal, whereas any regular file is hardlinked from the pool. This process takes a fraction of a second and gives us completely clean, deduplicated system roots.

Currently these live in /.moss/store/root/$ID/usr. To complete the transaction, we update /usr to point to the new usr tree atomically assuming that a reboot isn’t needed. In future, boot switch logic will update the tree for us.

Removal

Removal is quite the same as installation. We simply remove the package IDs from the new state selections (copied from the last state) and blit a new system root, finally updating the atomic /usr pointer.

Tying it all together

We retain classic package management traits such as having granular selections, multiple repositories, etc, whilst sporting advanced features like full system deduplication and transactions/rollbacks.

When we’re far enough along, it’ll be possible to boot back to the last working transaction without requiring an internet connection. Due to the use of pooling and hardlinks, each transaction tree is only a few KiB, with files shared between each transaction/install.

On the list..

We need some major cleanups, better error handling, logging, timed functions, and an eventloop driven process to allow parallel fetching/precaching prior to final system rootfs construction.

It’s taken us a very long time to get to this point, and there is still more work to be done. However this is a major milestone and we can now start adding features and polish.

Once the required features are in place, we’ll work on the much needed pre alpha ISO :) If you fancy helping us get to that stage quicker, do check out our OpenCollective! (We won’t limit prealpha availability, don’t worry :))

Tags:

• news

Moss DB Progress

Aug 3, 2021

Ikey

Founder

I’ll try to make this update as brief as I can but it’s certainly an important one, so let’s dive right into it. The last few weeks have been rough but work on our package manager has still been happening. Today we’re happy to reveal another element of the equation: moss-db.

Putting moss-db to the test

moss-db is an abstract API providing access to simplistic “Key Value” stores. We had initially used some payload based files as databases but that introduced various hurdles, so we decided to take a more abstract approach to not tie ourselves to any specific implementation of a database.

Our main goal with moss-db is to encapsulate the RocksDB library, providing sane, idiomatic access to a key value store.

High level requirements

At the highest level, we needed something that could store arbitrary keys and values, grouped by some kind of common key (commonly known as “buckets”). We’ve succeeded in that abstraction, which also required us to fork a rocksdb-binding to add the Transform APIs required.

Additionally we required idiomatic range behaviours for iteration, as well as generic access patterns. To that affect we can now foreach a bucket, pipe it through the awesomely powerful std.algorithm APIs, and automatically encode/decode keys and values through our generic APIs when implementing the mossdbEncode() and mossdbDecode() functions for a specific type.

In a nutshell, this was the old, ugly, hard way:

  /* old, hard way */
  auto nameZ = name.toStringz();
  int age = 100;
  ubyte[int.sizeof] ageEncoded = nativeToBigEndian(ageEncoded);
  db.setDatum(cast(ubyte[]) (nameZ[0 .. strlen(nameZ)]), ageEncoded);

And this is the new, shmexy way:

    db.set("john", 100);
    db.set("user 100", "bobby is my name");

    auto result = db.get!int("john");
    if (result.found)
    {
        writeln(result.value);
    }

    auto result2 = db.get!string("user 100");
    if (result2.found)
    {
        writeln(result2.value);
    }

It’s quite easy to see the new API lends itself robustly to our needs, so that we may implement stateful, strongly typed databases for moss.

Next Steps

Even though some APIs in moss-db may still be lacking (remove, for example) we’re happy that it can provide the foundation for our next steps. We now need to roll out the new StateDB, MetaDB and LayoutDB, to record system states, package metadata, and filesystem layout information, respectively.

With those 3 basic requirements in place we can combine the respective works into installation routines. Which, clearly, warrants another blog post … :)

For now you can see the relevant projects on our GitLab project.

Tags:

• news

Let There Be Databases

May 18, 2021

Ikey

Founder

We haven’t been too great on sharing progress lately, so welcome to an overdue update on timelines, progress, and database related shmexiness.

OK, so you may remember moss-format, our module for reading and writing moss binary archives. It naturally contains much in the way of binary serialisation support, so we’ve extended the format to support “database” files. In reality, they are more like tables binary encoded into a single file, identified by a filepath.

The DB archives are currently stored without compression to ensure 0-copy mmap() access when loading from disk, as a premature optimisation. This may change in future if we find the DB files taking up too much disk space.

So far we’ve implemented a “StateMetaDB”, which stores metadata on every recorded State on the system, and right now I’m in the progress of implementing the “StateEntriesDB”, which is something akin to a binary encoded dpkg selections file with candidate specification reasons.

Next on the list is the LayoutsDB (file manifests) and the CacheDB, for recording refcounts of every cached file in the OS pool.

Integration with Serpent ECS

An interesting trial we’re currently implementing is to hook the DB implementation up to our Entity Component system from the Serpent Engine, in order to provide fast, cache coherent, in memory storage for the DB. It’s implemented using many nice DLang idioms, allowing the full use of std.algorithm APIs:

    auto states()
    {
        import std.algorithm : map;

        auto view = View!ReadOnly(entityManager);
        return view.withComponents!StateMetaArchetype
            .map!((t) => StateDescriptor(t[1].id, t[3].name, t[4].description,
                    t[1].type, t[2].timestamp));
    }
    ...

  /* Write the DB back in ascending numerical order */
  db.states
    .array
    .sort!((a, b) => a.id < b.id)
    .each!((s) => writeOne(s));

Tying it all together

Ok, so you can see we need basic DB types for storing the files for each moss archive, plus each cache and state entry. If you look at the ECS code above, it becomes quite easy to imagine how this will impact installation of archives. Our new install code will simply modify the existing state, cache the incoming package, and apply the layout from the DB to disk, before committing the new DB state.

In essence, our DB work is the current complex target, and installation is a <50 line trick tying it all together.

  /* Pseudocode */

  State newState...

  foreach (pkgID; currentState.filter!((s) => s.reason == SelectionReason.Explicit))
  {
    auto fileSet = layoutDB.get(pkgID);
    fileSet.array.sort!((a, b) => a.path < b.path).each!((f) => applyLayout(f));
    /* Record into new state */
    ...
  }

Til next time -

Ikey

Tags:

• news

Moss Unlocked

Mar 20, 2021

Ikey

Founder

Well, it’s not all doom and gloom these days. We’ve actually made some significant progress in the last few days, so it seems a good time to share a progress update.

moss can now extract

Oh yeah, that totally happened. So, we can now successfully build moss packages from boulder and then extract them to disk once again with moss. This might sound totally uninteresting, but it demonstrates that our format is actually working as intended.

Admittedly the code is super rough within moss and somewhat proof of concept, however we’re able to extract the contents of the moss archive and rebuild the layout on disk.

What makes extraction difficult?

Well, quirky new format for one. A moss archive currently consists of 4 “payloads”, or major sections:

• MetaPayload

  Contains all package information, with strongly typed keys.

• IndexPayload

  Contains the IDs of all unique files (hash) and their offsets within the ContentPayload

• LayoutPayload

  A sequence of specialised structs describing the final “layout” of the package on disk, with attributes, paths, and for regular files, the ID of the file in the ContentPayload to build this file from.

• ContentPayload

  A binary blob containing every unique file from the package, in an order described by the IndexPayload. The files are stored sequentially with no gaps.

Additionally, each payload is independently compressed using zstd. In order to extract files to disk, we must first decompress ContentPayload to a temporary file. Next, we blit each file from the “megablob” to the cache store, using the IndexPayload to understand the offsets. Finally, we apply the instructions in LayoutPayload to construct the final layout on disk, hardlinking the cache assets into their final locations, setting attributes, etc.

Net result? Deduplication on a per package basis, and a system-wide deduplication policy allowing sharing of identical assets on disk between multiple packages. This will also power our core update mechanism, whereby each update is atomic, and is simply the difference on disk from the previous version, permitting a powerful rollback mechanism.

Room for improvement

There are areas where we’re doing things inefficiently, and we’ll certainly improve that in future revisions of the important. For example, IndexPayload actually wastes some bytes by storing redundant information that can be calculated at runtime. Additionally, we want to use the zstd C APIs directly to gain the level of control we actually need. We’re also going to wrap the copy_file_range syscall to make extraction of the content payload more efficient and not rely on userspace copies.

However, we’re working towards a prealpha, and some inefficiencies are OK. Our first port of call will be a prealpha image constructed from .stone files, produced by boulder, installed by moss. This will validate our toolchain and tooling and serve as a jumping off point for the project.

Stay tuned, there is a whole bunch of awesome coming now that moss is officially unlocked and progressing.

And finally

I want to thank everyone who is currently supporting the project. I also want to personally thank you for your understanding of the setbacks of real life, given the difficult times myself and my family have been going through. I hope it is clear that I remain committed to the project and it’s future, which is why we’re transparently run and funded via OpenCollective.

Despite the rough times, work continues, and awesome people join our ranks on a regular basis. Stability is on the immediate horizon and work with Serpent OS grows exponentially. You can be part of our journey, and help us build an amazing community and project that outlives us all.

Tags:

• news

Moss Format: Read Write Support

Feb 17, 2021

Ikey

Founder

It’s been 8 days since our last blogpost and a lot of development work has happened in that time. Specifically, we’ve completely reworked the internals of the moss-format module to support read/write operations.. Which means installation is coming soon (TM)

So, many commits have been made to the core repositories, however the most important project to focus on right now is moss-format, which we used to define and implement our binary and source formats. This module is shared between boulder, our build tool, and moss, our package manager.

We’ve removed the old enumeration approach from the Reader class, instead requiring that it processes data payloads in-place, deferring reading the content payload streams. We’ve also enforced strong typing to allow safe and powerful APIs:

        import moss.format.binary.payload.meta : MetaPayload;

        auto reader = new Reader(File(argv[0], "rb"));
        auto metadata = reader.payload!MetaPayload();

Right now we can read and write our MetaPayload from and to the stream, allowing us to encode & decode the metadata associated with the package, with strong type information (i.e. Uint64, String, etc.)

Next Steps

We need to restore the IndexPayload, LayoutPayload and ContentPayload definitions. The first two are simply data payloads and will largely follow the design of the newly reimplemented MetaPayload. Then we restore ContentPayload support, and this will allow the next steps: unpack, install.

Many of the babysteps required are done now, which power our binary format. The design of the API is done in a way which will allow powerful manipulation via the std.algorithm and std.range APIs, enabling extremely simple and reliable installation routines.

It might seem odd that we’ve spent so much time on the core format, however I should point out that the design of the format is central to the OS design. Our installation routine is not unpacking of an archive.

With our binary format, the stream contains multiple PayloadHeaders, with fixed lengths, type, tag, version, and compression information. It is up to each Payload implementation to then parse the binary data contained within. Currently our Payloads are compressed using ZSTD, though other compression algorithms are supported.

So, we have the MetaPayload for metadata. Additionally, we encode all unique files in the package in a single compressed payload, the ContentPayload. The offset to each unique file (by hash) is stored within the IndexPayload, and the filesystem layout is encoded as data within the LayoutPayload.

In order to unpack a moss package, the ContentPayload blob will be decompressed to a temporary location. Then, each struct within the IndexPayload can be used to copy portions (copy_file_range) of the blob to the cache store for permanence. We skip each Index if its already present within the cache. Finally, the target filesystem is populated with a view of all installed packages using the LayoutPayload structs, creating a shallow installation using hardlinks and directories.

The net result is deep deduplication, atomic updates, and flexibility for the user. Once we add transactions it’ll be possible to boot an older version of the OS using the deduplication capabilities for offline recovery. Additionally there is no requirement for file deletion, rename or modification for an update to succeed.

Nutshell

Huge progress. Major excitement. Such wow. Soon alphas.

Tags:

• news

Unlocking Moss

Feb 9, 2021

Ikey

Founder

Wait, what? Another blog post? In the same WEEK? Yeah totally doing that now. So, this is just another devlog update but there have been some interesting updates that we’d like to share.

LDC present in the bootstrap

Thanks to some awesome work from Peter, we now have LDC (The LLVM D Lang Compiler) present in the stage3 bootstrap. To simplify the process we use the official binary release of LDC to bootstrap LDC for Serpent OS.

In turn, this has allowed us to get to a point where we can now build moss and boulder within stage3. This is super important, as we’ll use the stage3 chroot and boulder to produce the binary packages that create stage4.

Some patching has taken place to prevent using ld.gold and instead use lld to integrate with our toolchain decisions.

A wild LDC appears

Reworking the Payload implementation

Originally our prototype moss format only contained a ContentPayload for files, and a MetaPayload for metadata, package information, etc. As a result, we opted for simple structs, basic case handling and an iterable Reader implementation.

As the format expanded, we bolted deduplication in as a core feature. To achieve this, we converted the ContentPayload into a “megablob”, i.e. every unique file in the package, one after the other, all compressed in one operation. We then store the offsets and IDs of these files within an IndexPayload to allow splitting the “megablob” into separate, unique assets. Consequently, we added a LayoutPayload which describes the final file system layout of the package, referencing the unique ID (hash) of the asset to install.

So, while the format grew into something we liked, the code supporting it became very limiting. After many internal debates and discussions, we’re going to approach the format from a different angle on the code front.

It will no longer be necessary/possible to iterate payloads in sequence within a moss archive, instead we’ll preload the data (unparsed) and stick it aside when reading the file, winding it to the ContentPayload segment if found. After initial loading of the file is complete, the Reader API will support retrieval (and lazy unpacking ) of a data segment. In turn this will allow code to “grab” the content, index and layout payloads and use advanced APIs to cache assets, and apply them to disk in a single blit operation.

In short, we’ve unlocked the path to installing moss packages while preserving the advanced features of the format. The same new APIs will permit introspection of the archives metadata, and of course, storing these records in a stateful system database.

Thank you for the unnecessary update

Oh you’re quite welcome :P Hopefully now you can see our plan, and that we’re on track to meet our not-28th target. Sure, some code needs throwing away, but all codebases are evolutionary. Our major hurdle has been solved (mentally) - now it’s just time to implement it and let the good times roll.

Tags:

• news

Cleanups Complete

Feb 8, 2021

Ikey

Founder

Well, we’re officially back to working around the clock. After spending some time optimising my workflow, I’ve been busy getting the entire codebase cleaned up, so we can begin working towards an MVP.

Codebase hygiene

Since Friday, I’ve been working extensively on cleaning up the codebase for the following projects:

• boulder
• moss
• moss-core
• moss-format

As it now stands, 1 lint issue (a non-issue, really) exists across all 4 projects. A plethora of issues have been resolved, ranging from endian correctness in the format to correct idiomatic D-lang integration and module naming.

Bored

Granted, cleanups aren’t all that sexy. Peter has been updating many parts of the bootstrap project, introducing systemd 247.3, LLVM 11.0.1, etc. We now have all 3 stages building correctly again for the x86_64 target.

And then…

Yikes, tough audience! So we’ve formed a new working TODO which will make its way online as a public document at some point. The first stage, cleanups, is done. Next is to restore feature development, working specifically on the extraction and install routines. From there, much work and cadence will be unlocked, allowing us to work towards an MVP.

You keep saying MVP..

I know, it makes me feel all … cute and professional. At the moment we’re cooking up a high level description of how an MVP demonstration could be deployed. While most of the features are ambiguous, our current ambition is to deploy a preinstalled QCOW2 Serpent OS image as a prealpha to help validate moss/boulder/etc.

It’ll be ugly. Likely slow. Probably insecure as all hell. But it’ll be something. It’ll allow super basic interaction with moss, and a small collection of utilities that can be used in a terminal environment. No display whatsoever. That can come in a future iteration :)

ETA? Definitely not by the 28th of February. When it’s done.

Tags:

• news

Build System Functional

Oct 11, 2020

Ikey

Founder

Wow, has it been a hectic few weeks, and it definitely shows: last time we blogged it was about re-bootstrapping with glibc. Feels like ancient news already! So, what’s new in the world of Serpent OS? Apart from yours truly now being proud parent to a beautiful baby girl, work has resumed on the development of Moss, our package manager. And it builds stuff. Awesomely. Let’s quickly catch up with those updates and see where we’re headed next.

It lives!

We’re now able to build the majority of a moss package. Notice I’ve made a distinction there. So, we’re able to run the build instructions, and use all of the metadata, configuration and macros available. The only thing we’re not actually doing is dumping the built files into a binary package.

We’re able to do the build-system part rather well, now. Right now we support the following features in the build system:

• Multiple, profile-based architecture support
• Automatic -m32 profile based cross compilation support for 32-bit libraries on 64-bit OS
• Basic Profile Guided Optimisation via the workload key. Set a workload, optimise accordingly.
• LLVM Context Sensitive Profile Guided Optimisation. This is default for LLVM with the workload key, and comes for free, with multiple build stages.
• Profile based tuning options, such as optimize for speed, size, disabling hardening, etc.
• Trivially switch between gnu and llvm toolchain in stone.yml, with profiles knowing the right flags to use with tuning options.
• Recursive macro support in build scripts, defined on per-profile level
• Architecture specific profiles support in stone.yml, i.e. profiles -> aarch64 -> setup: to avoid if/else spaghetti.

Next on the agenda

Now that the huge amount of scaffolding work has been done, we can actually turn the results of builds into installable binary packages using our moss.format.binary module. We’ll add some magic sauce to have automatic subpackages + inter-package dependencies, along with the expected automatic runtime dependencies + such. Cue some linting, et voila, build tool that’s a pleasure to work with.

Once we have all those packages building, we’ll need a way to install them. Luckily some scaffolding is in place for this already, and it won’t take much effort to support moss install somepkg.stone. Then we throw a few dozen packages in the mix, add some dependency handling, repo support, Bob’s your uncle, and your aunt is downloading exclusive early-access images from our Open Collective as soon as they’re available. :O

Tags:

• news

Results Of The Experiment

Sep 22, 2020

Ikey

Founder

import FigureScreenshotOne from ”@/components/ui/FigureScreenshotOne.astro” import featuredImage from ”./Featured.webp”

It seems like only yesterday we announced to the world a Great Experiment. It was in fact 2 months ago, and a whole lot of work has happened since that point. A few take-homes are immediately clear, the primary one being the need to be a community-oriented Linux distribution.

To quote ourselves 2 months ago:

 If the experiment is a success, which of course means having tight controls on scope and timescale,
 then one would assume the primary way to use Serpent OS would be through some downstream repackaging
 or reconfiguration, i.e. basing upon the project, to meet specific user demand.

It turns out that so far the experiment has been successful, and being forkable is still at the very heart of our goals. Others have joined us on our journey, and expressed the same passion in our goals as we have. A community has formed around the project, with individuals sharing the same ambitions for a reliable, rolling operating system with a powerful package manager.

An open community

Over the past 2 months we’ve transformed from set of ideas into a transparent, community-first organisation with clear leadership and open goals. I’ve stepped into the Benevolent Dictator For Life position, and Peter has taken on daily responsibilities for the project running. Aydemir is now our treasurer on Open Collective, and many individuals contribute to our project.

Transactional package management

No need to rehash this, but the defining feature of Serpent OS has clearly become moss, something initially not anticipated when we started. A read-only root filesystem, transactional operations, rollbacks, deduplicating throughout and atomic updates. Combine that with a rolling release model, stateless policy and ease-of-use, the core feature-set is already powerful.

Deferral of musl integration

In recent weeks we’ve been working on libwildebeest and libc-support, primarily as a stop-gap to provide glibc compatibility without using glibc. While musl has many advantages, it is clear to us now that writing another libc through our support projects isn’t what we originally planned. With that in mind we’re adopting glibc and putting our musl works under community ownership, until such time as reevaluation shows that musl is what is needed in Serpent OS. Note the primary motivator here is investing our efforts where it makes sense, and obtaining the best results in the most manageable fashion for our users.

Our new toolchain configuration will be as follows:

• LLVM/clang as primary toolchain
• libc++ as C++ library
• glibc as C library
• gcc+binutils (ld.bfd) provided to build glibc, elfutils, etc.
• Permitting per-package builds using GCC, i.e. kernel to alleviate Clang IAS issues.

Thus our toolchain will in fact be a hybrid GNU/LLVM one. This will allow both source and binary compatibility with the majority of desktop + server Linux distributions, facilitating choice of function for our users.

It should be noted this decision has been made after much discussion internally, on our IRC, on our OpenCollective, etc. Our bootstrap-scripts is being improved to support both glibc and musl, so that the decision can continuously be reviewed. If we reach a position whereby musl inclusion once again makes sense, thanks to atomic updates from moss, it will be possible to switch.

Final thoughts

Initially Serpent OS emerged as a collective agreement on IRC as a set of notions as opinions. Over the past few months those opinions have solidified into tangible ideas, and a sense of community. In keeping with what is right for the community, our messaging has been reworked.

It is fair to say our initial stance appeared quite hostile, as a bullet-point list of exhaustion with past experiences. As we’ve pivoted to being a fully community-oriented distribution, we’ve established our goals of being a reliable, flexible, open, rolling Linux distribution with powerful features imbued by the package manager, and an upstream-first approach.

As such we’ve agreed to not let our own pet-peeves interfere with the direction of the project, and instead enable users to do what they wish on Serpent OS, be it devops, engineering, browsing, gaming, you name it. We’re a general purpose OS with resilience at the core.

Our focus is on the usability and reliability of the OS - thus our efforts will be invested in areas such as the package manager, hardware enabling, the default experience, etc.

So, strap yourself in, as we’re fond of saying. Development of Serpent OS is about to accelerate rapidly.

Tags:

• news

Source Format Defined

Sep 21, 2020

Ikey

Founder

Following quickly on the heels of yesterday’s announcement that the binary format has been defined, we’ve now implemented the initial version of our source format. The source format provides metadata on a package along with instructions on how to build the package.

The next step of course, is to implement the build tool, converting the source specification into a binary package that the end user can install. With our 2 formats defined, we can now go ahead and implement the build routines.

A YAML based format

The eagle-eyed among you will already see this is a derivation of the package.yml format I originally created while at Solus. Minor adaptations to the format have been made to support multiple architectures via the profiles key, and package splitting behaviour has now been grouped under a packages key to make the structure more readable.

In package.yml, one would have to redefine subpackage summaries as a key in a list of the primary summary key, such as:

    rundeps:
        - primary-run-dep
        - dev: secondary-run-dep
    summary:
        - Some Summary
        - dev: Some different summary

We’ve opted to group “Package Definition” behaviour into core structs, which are allowed to appear in the root-level package and subpackages:

    summary: Top-level summary
    packages:
        - dev:
            summary: Different summary
            rundeps: Different rundeps

Multiple architecture support

In keeping with the grouping behaviour, we’re baking multiple architecture configurations into the YML file. A common issue encountered with the older format was how to handle emul32:

    setup: |
        if [[ -z "${EMUL32BUILD}" ]]; then
            %configure --some-emul32-option
        else
            %configure
        fi

    build: |
    %make

Our new approach is to group Build Definitions into the root level struct, which may then individually be overridden for each architecture. For example:

    profiles:
        - ia32:
            setup: |
        %configure --some-emul32-option
    setup: |
    %configure

Differences

As you can see it is highly similar to package.yml - which is a great format. However, with our tooling and aims being slightly different, it was time to reevaluate the spec and bolster it where appropriate. We’re happy to share our changes, but in the interest of not causing a conflict between the 2 variants, we’ll be calling ours “stone.yml”.

Our main motivation came from the tooling, which is written in the D language. With D we were able to create a strongly typed parser and explicit schema, and with a struct-based approach it made it more trivial to group similar definitions.

Other than that, we have the same notions with the format, intelligent automatic package splitting, ease of developer experience, etc.

Tags:

• news

Moss Format Defined

Sep 20, 2020

Ikey

Founder

The core team have been hard at work lately implementing the Moss package manager. We now have an initial version of the binary format that we’re happy with, so we thought we’d share a progress update with you.

Explaining the format

Briefly, the binary container format consists of 4 payloads:

• Meta (Information on the package)
• Content (a binary blob containing all files)
• Index (indices to files within the binary blob)
• Layout (How to apply the files to disk)

Each payload is verified internally using a CRC64-ISO, and contains basic information such as the length of the payload both compressed and uncompressed, the compression algorithm used (zstd and zlib supported) as well as the type and version of the payload. All multiple-byte values are stored in Big Endian order (i.e. Network Byte Order).

Internally the representation of a Payload is defined as a 32-byte struct:

    @autoEndian uint64_t length = 0; /* 8 bytes */
    @autoEndian uint64_t size = 0; /* 8 bytes */
    ubyte[8] crc64 = 0; /* CRC64-ISO */
    @autoEndian uint32_t numRecords = 0; /* 4 bytes */
    @autoEndian uint16_t payloadVersion = 0; /* 2 bytes  */
    PayloadType type = PayloadType.Unknown; /* 1 byte  */
    PayloadCompression compression = PayloadCompression.Unknown; /* 1 byte */

We merge all unique files in a package rootfs into the Content payload, and compress that using zstd. The offsets to each unique file (i.e. the sha256sum) are stored within the Index payload, allowing us to extract relevant portions from the “megablob” using copy_file_range().

These files will become part of the system hash store, allowing another level of deduplication between all system packages. Finally, we use the Layout payload to apply the layout of the package into a transactional rootfs. This will define paths, such as /usr/bin/nano, along with permissions, types, etc. All regular files will actually be created as hard links from the hash store, allowing deduplication and snapshots.

The Meta payload consists of a number of records, each with strongly defined types (such as String or Int64) along with the tag, i.e. Name or Summary. The entire format is binary to ensure greater resilience and a more compact representation. For example, each metadata key is only 8 bytes.

    @autoEndian uint32_t length; /** 4 bytes per record length*/
    @autoEndian RecordTag tag; /** 2 bytes for the tag */
    RecordType type; /** 1 byte for the type */
    ubyte[1] padding = 0;

TLDR that for me…

Binary format that is self deduplicating at several layers, permitting fast transactional operations.

Up Next

Before we work any more on the binary format, we now need to pivot to the source format. Our immediate goal is to now have moss actually build packages from source, with resulting .stone packages. Once this step is complete we can work on installation, upgrades, repositories, etc, and race to becoming a self hosting distribution.

Note, the format may still change before it goes into production, as we encounter more cases for optimisation or improvement.

Tags:

• news

Defining Moss

Sep 10, 2020

Ikey

Founder

Over the past few weeks, throughout the entire bootstrap process, we’ve been deliberating on what our package manager is going to look like. We now have a very solid idea on what that’ll be, so it’s time for a blog post to share with the community.

Old, but new

The team has been very clear in wanting a traditional package management solution, whereby repositories and packages compose the OS as a whole. However, we also want modern features, such as being stateless by default. A common theme is wanting to reduce the complex iterations of an OS into something that is sanely versioned, but also flexible, to ensure a consistent, tested experience with multiple end targets.

Additionally, the OS must be incredibly easy for contributors and team members to maintain, with intelligent tooling and simple, but powerful formats.

Atomic updates

One of the most recent software update trends of recent years is atomic updates. In essence this allows applying an update in a single operation, and importantly, reversing an update in a single operation, without impacting the running system.

This is typically achieved using a so-called A/B switch, which is what we will also do with moss. We won’t rely on any specific filesystem for this implementation, instead relying on a smart layout, pivot_root and a few other tricks.

Primarily we’ll update a single /usr symlink to point to the current OS image, with / being a read-only faux rootfs, populated with established mountpoints and symlinks. Mutation will be possible only via moss transactions, or in world-writable locations (/opt, /usr/local, /etc …)

Deduplication

The moss binary package format will be deduplicated in nature, containing hash-keyed blobs in an zstd compressed payload. Unique blobs will be stored in a global cache, and hard-linked into their final location (i.e. /moss/root/$number/usr/...) to deduplicate the installed system too. This allows multiple system snapshots with minimal overhead, and the ability to perform an offline rollback.

Implications

We’ll need to lock kernels to their relevant transactions (or repo versions) to prevent untested configurations. Additionally the boot menu will need to know about older versions of the OS that are installed, so they can be activated in the initrd at boot. This will require us doing some work with clr-boot-manager to achieve our goals.

Difference from existing solutions

We’re trying to minimise the iterations of the OS to what is available in a given version of the package repositories. Additionally we wish to avoid extensive “symlink farms” as we’re not a profile-oriented distribution. Instead we focus on deduplication, atomic updates and resolution performance.

Subscriptions

Keeping a system slim is often a very difficult thing to achieve, without extensive package splitting and effort on the user’s part. An example might be enabling SELinux support on a system, or culling the locales to only include the used ones.

In Serpent OS (and moss, more specifically) we intend to address this through “subscriptions”. Well defined names will be used by moss to filter (or enable) certain configurations in packages + dependencies. In some instances this will toggle subpackages/dependencies and in others it will control which paths of a package are actually installed to disk.

Going further with this concept, we will eventually introduce modalias based capabilities to automatically subscribe users to required kernel modules, allowing slim or fullfat installations as the user chooses. This in turn takes the burden of maintenance away from developers + users, and enables an incredibly flexible, approachable system.

Mandatory reboots

Where possible we will limit mandatory reboots, preferring an in-place atomic update to facilitate high uptime. However, there are situations where a reboot is absolutely unavoidable, and the system administrator should plan some downtime to handle this case.

Certain situations like a kernel update, or security fix to a core library, would require a reboot. In these instances, the atomic update will be deferred until the next boot. In most situations, however, reboots will not be mandatory.

Until the next time

Well, we’ve given a brief introduction to our aims with moss and associated tooling, and you can get more information by checking the moss README.md.

The takeaway is we want a package-based software update mechanism that is reliable and trusted, and custom-built to handle Serpent OS intricities, with a simple approach to building and maintaining the distribution.

For now, we’re gonna stop talking, and start coding.

Tags:

• news

Stage3 Complete

Sep 7, 2020

Ikey

Founder

Another week, another milestone completed. We’re pleased to announce that we’ve completed the initial Stage3 bootstrap. While we have some parallel works to complete for Stage 4, we can now begin to work on the exciting pieces! We’re now bootstrapped on ARMv8(-a) and x86_64

Announcing Moss

Our immediate focus is now to implement our package manager: moss. Currently it only has a trivial CLI and does absolutely nothing. We will now shift our attention to implement this as a core part of the Stage 4 bootstrap. Moss is so-called as we’re a rolling release, a rolling stone gathers no moss. The package manager is being implemented in the D Programming Language.

Moss does not aim to be a next generation package management solution, it instead inspired by eopkg, RPM and swupd, aiming to provide a reliable modern package management and update solution with stateless design at the core. Core features and automatic dependencies will be managed through capability subscriptions. The binary format will be versioned and deduplicated, with multiple internal lookup tables and checksums. Our chief focus with moss is a reliable system management tool that is accessible even in situations where internet access is limited.

Ongoing Works

In order to complete stage3, we provided linker stubs in libwildebeest to ensure we can build. Obviously this introduces runtime errors in systemd and our stage3 isn’t bootable, just chrootable. This will be resolved when systemd is properly packaged by moss in stage4.

Docker Image

We now have a test container available on Docker Hub. You can install and run the simple bash environment by executing the following command with a Docker-enabled user:

    docker run -it --rm serpentos/staging:latest

Currently we only have an x86_64 image, but may experiment with multiarch builds later in stage4.

IMPORTANT: The staging image is currently only a dump of the stage3 tree with minor cleaning + tweakups. It is not, in any way, shape or form, representative of the final quality of Serpent OS. Additionally zero performance work or security patching has been done, so do not use in a production environment.

The image is provided currently as a means to validate the LLVM/musl toolchain.

Future ISO

We cannot currently say when we’ll definitely have an ISO, however we do know that some VM-specific images will arrive first. After that we’ll focus on an installer (package selection based) and a default developer experience. All we can say is strap in, and enjoy the ride.

Tags:

• news

Looking Stage4 In The Eye

Aug 28, 2020

Ikey

Founder

Well, we’ve made an awful lot of progress in these last few days. It wasn’t that long ago that we introduced some of the new projects required to get stage3 off the ground.

libc-support

Our libc-support project has been growing, thanks primarily to the contributions of Jouni Roivas. We now have initial working versions of getent and getconf. The getconf program is considered feature-complete for our current requirements, and the focus is now on cleaning up getent making it more modular and easy to maintain in the long run.

libwildebeest

We began work on libwildebeest to quickly unlock building systemd. Remember, our ambition with this project is to provide a sane, centralised way of maintaining source compatibility with various projects that rely on features currently only available in the GNU toolchain + runtime. This is our alternative vision to patching every single package that fails to build against our LLVM+musl toolchain, ensuring our work scales.

Right now, libwildebeest implements some missing APIs, and indeed, some replacement APIs, for when GNU behaviours are expected. It does so in a way that doesn’t impact the resulting binary’s license, or any of the system ABI. We provide some pkg-config files with explicit cflags and libs fields set, such as:

      -L${libdir} -lwildebeest-ftw -Wl,--wrap=ftw -Wl,--wrap=nftw -I${includedir}/libwildebeest --include=lwb_ftw.h

Our headers take special care to mask the headers provided by musl to avoid redefinitions, and instruct the linker to replace calls to these functions with our own versions, i.e.:

int __wrap_ftw(const char *dir, int (*funcc)(const char *, const struct stat *, int),
               int descriptors)

It then becomes trivial to enable wildebeest for a package build, i.e.:

        export CFLAGS="${CFLAGS} $(pkg-config --cflags --libs libwildebeest)"

Right now - we’ve only provided stubs in libwildebeest to ensure we can build our packages. Our next focus is to actually implement those stubs using MIT licensed code so that applications and libraries can rely on libwildebeest to provide a basic level of GNU compatibility in a reliable fashion.

Until such point as all the APIs are fully and safely implemented, it would be highly ill-advised to use libwildebeest in any project. We’ll announce stability in the coming weeks.

systemd

We’ve made great progress in enabling systemd in Serpent OS. Where libwildebeest is in place, it now enables our currently required level of source compatibility to a point where systemd is building with networkd, resolved and a number of other significant targets enabled.

In a small number of cases, we’ve had to patch systemd, but not in the traditional sense expected to make it work with musl.

The only non-upstreamable patching we’ve done (in combination with libwildebeest enabling) was to the UAPI headers, as the musl provided headers clash with the upstream kernel headers in certain places (if_ether.h, if_arp.h) - but this is a tiny cost to bear.

The other patches, were simply portability fixes, ensuring all headers were included:

• partition/makefs: Include missing sys/file.h header #16876
• login/logind: Include sys/stat.h for struct stat usage #16887

It should be noted both of these (very trivial) pull requests were accepted and merged upstream, and will be part of the next systemd release.

Next On The Agenda

Our major ticket items involve fleshing out stage3 with some missing libraries to further enable systemd, rebuilds of util-linux to be systemd-aware, and continue fleshing out dbus, systemd and libwildebeest support to the point we have a bootable disk image.

At that point we’ll move into stage4 with package management, boot management, and a whole host of other goodies. And, if there is enough interest, perhaps some early access ISOs!

After having engaged in discussions with a variety of developers using musl as their primary libc, we’ve catalogued common pain points. We therefore encourage developers to contribute to our libwildebeest and libc-support projects to complete the tooling and experience around musl-based distributions.

Our aim for Serpent OS is a full fat experience, which means we have large ticket items on our horizon, such as NSS/IDN integration, performance improvements, increasing the default stack size, along with source compatibility for major upstream projects.

Until the next blog post, you can keep up to date on our IRC channel. Join #serpentOS on freenode!

Tags:

• news

Stage 3 Progress

Aug 20, 2020

Ikey

Founder

Well, it’s been a few days since we last spoke, so now it’s time for a quick roundup. Long story short, we’re approaching the end of the stage3 bootstrap.

In an effort to simplify our bootstrap process, we dropped the newly-introduced stage2.5 and came up with a new strategy for stage3. In order to make it all work nicely, we bind-mount the stage2 resulting runtime at /serpent within the stage3 chroot environment, executing the /serpent/usr/bin/bash shell.

In order to make this work, we build an intermediate musl package for libc.so in the very start of stage3, with all subsequent builds being performed in chroots. Part of the build is done on the host, i.e. extraction and patching, minimising the tool requirements for the chroot environment. The configuration, build and install is performed from within the initially empty chroot environment, replacing all the /serpent/usr/bin tools and /serpent/usr/lib libraries.

Mild Blockers

As we move further through stage3, towards a fully usable chroot environment, we’ve encountered a small number of blockers. Now, we could solve them by using existing patchwork and workarounds, but most have not and will not be accepted upstream. Additionally it is incredibly hard to track the origin and history of most of these, making security rather more painful.

libc-support

We’re going to start working on a project to flesh out the musl runtime with some missing utilities, written with a clean-room approach. These will initially include the getconf and getent tools, which will be written only with Linux in mind, and no legacy/BSD support.

These will be maintained over at our GitHub

libwildebeest

As a project we strive for correctness in the most pragmatic way. Some software, such as systemd, is heavily reliant on GNU GCC/GLibc extensions. In some software there are feasible alternatives when using musl, however in a good number of cases, functionality required to build certain software is missing and has no alternative.

Over time we’ll try to work with upstreams to resolve those issues, but we’re working on an interim solution called ‘libwildebeest’. This will provide source compatibility for a limited number of software packages relying on so-called ‘GNUisms’. Binary compatibility is not an aim whatsoever, and will not be implemented. This convenience library will centralise all patchwork on packages that need more work to integrate with musl, until such time as upstreams have resolved the remaining issues.

Additionally it will help us track those packages needing attention in the distribution, as they will have a build-time dependency on libwildebeest. We do not intend to use this project extensively.

This will be maintained over at our GitHub

A Word On Init Systems

Recently we’ve had many queries regarding the init system, as there is an expectation that due to our use of musl/llvm we also dislike systemd or wish to be a small OS, etc. There is a place in the world for those projects already, and we wish them much success. However from our own stance and goals, systemd has already “won the battle” and actually fits in with our design.

If it is possible in future with package manager considerations and packaging design, then we may make it possible to swap systemd for a similar set of packages. However, we only intend at this time to support systemd/udev/dbus directly in Serpent OS and leave alternatives to the community.

Other News

Just a quick heads up, we’ve been talking to the cool folks over at fosshost.org and they’ve agreed to kindly provide us with additional hosting and mirroring. This will allow us to build scale in from the very start, ensuring updates and images are always available. Once the new server is up and running we’ll push another blogpost with the details and links.

While initially we intended to avoid public bug trackers, the rate of growth within the project and community have made it highly apparent that proper communication channels need establishing. Therefore we will be setting up a public Phabricator instance for reporting issues, security flaws, and contributing packaging.

Much of our website is in much need of update, but our current priority is with building the OS. Please be patient with us, we’ll have it all sorted out in no time.

Where We At?

Well, stage3 completes fully, builds the final compiler, which has also been verified. A usable chroot system is produced, built using musl, libc++, libunwind, clang, etc. Some might say that stage3 is complete, however we wish to avoid circular dependency situations. We’ll mark stage3 as complete once we’ve integrated an initial slimmed down build of systemd and appropriately relinked the build.

As soon as this stage is done, we’ll proceed with stage4. This is the final stage where we’ll add package management and define the OS itself, with global flags, policies, etc.

With the speed we’re moving at, that really isn’t too far away.

And finally..

I personally wish to thank the Serpent OS team as a whole for the commitment and work undertaken of late. Additionally I want to thank the growing community around Serpent OS, primarily residing in our IRC channel (#serpentOS on freenode) and our Twitter account. Your input has been amazing, and it’s so refreshing to have so many people on the same page. Stay awesome.

Tags:

• news

Stage2 Complete

Aug 16, 2020

Ikey

Founder

Just in case you thought we were sleeping behind the wheel, we’ve got another blogpost for your viewing pleasure. In a nutshell, we completed stage2 bootstrap.

In order to simplify life, we greatly reduced the size of the stage2 build component. This decision was taken to better support cross-compilation in the face of software that is distinctly cross-compilation unfriendly.

A support stage, stage2.5 will be added which will chroot into a copy of stage2, and natively compile a small handful of packages required to complete stage3, also within the chroot environment.

For cross-compilation, we’ll be relying on qemu-static to complete 2.5 and 3. However, at this point in time, we have the following:

• Working cross-compilation of the entire bootstrap
• Complete LLVM based toolchain: clang, llvm, libc++, lib++abi, libunwind
• Entirety of stage2 built with musl libc.
• Working, minimal, chroot environment as product of stage2, with working compiler (C & C++)

This is a major milestone for the project, as it is an early indication that we’re self hosting.

Multiple Architecture Support

At this point in time, we now have build support for two targets: x86_64 and ARMV8a. Our intent is to support haswell and above, or zen and above, for the x86_64 target.

With our ARMv8 target, we’re currently looking to support the Pine Book Pro, if we can manage to get hold of some testing hardware. It will likely be some time after full x86_64 support that we’d officially support more hardware, however it is very important that our bootstrap-scripts can trivially target multiple platforms.

An interesting change when cross-compiling for other architectures, is the chicken & egg situation with compiler-rt and other LLVM libraries. When we detect cross-compilation, we’ll automatically bootstrap compiler-rt before building musl, and then cross-compile libc++, libc++abi and libunwind to ensure stage1 can produce native binaries for the target with correct linkage.

Next Steps

As we’ve mentioned, we’ll push ahead with 2.5 and 3, which will complete the initial Serpent OS bootstrap, producing a self-hosting, self-reliant rootfs. This is the point at which we can begin to bolt-on package management, boot management, stateless configuration utilities, etc.

Our initial focus is x86_64 hardware with UEFI, and as we gain access to more hardware we can enable support for more targets, such as ARMv8a. Our bootstrap-scripts will always remain open source, as will all processes and tooling within Serpent OS, or anything used to build and deploy Serpent OS.

This will make it much easier in future to create custom spins of Serpent OS for different configurations or targets, without derailing the core project. It should therefore be the simplest thing in the world to fork Serpent OS to one’s liking or needs.

If you want to support our work, you can jump onto our IRC channel (#serpentOS on freenode) or support us via the Team page.

Tags:

• news

Stage1 Complete

Aug 8, 2020

Ikey

Founder

Short and sweet, stage1 of the bootstrap is complete. As I indicated on the Lispy Snake blog, I’m still in the process of settling into new accommodation. This is going well, but still awaiting proper broadband connectivity. Work has begun, however, and we’re now moving onto stage2 of the bootstrap.

This is handled via our bootstrap-scripts project and can be run by anyone on a relatively modern Linux distribution.

Next on the list is completing stage2, which we’ve already started on. This is simply a cross-compiled chroot environment with all the basic bits in place to build stage3, sanitizing and cleansing the toolchain.

Even though it is early days, we can already, automatically produce a working cross-compiler that targets LLVM’s libc++, the musl libc.so, and x86_64-serpent-linux-musl host triplet.

Probably a dull update for most, but an update it is.

Tags:

• news

Welcome: Project Manager

Jul 7, 2020

Ikey

Founder

The Serpent OS team wishes to formally welcome Aydemir Ulaş Şahin to the core team in the primary capacity of Project Manager. Aydemir will help us to coordinate and manage the project goals and infrastructure, being responsible for certain “keys to the kingdom”.

Aydemir has extensive experience with Linux, both as a user and developer, having worked in the media industry for many years. Additionally he will be contributing to the core system, code and packaging.

Our initial trio has been announced, and you can always access information on the current team at the new Team page. Now that much of our groundwork has been laid, we can get on with building the project. Check progress at the new Roadmap page.

Tags:

• news

Welcome Peter

Jul 6, 2020

Ikey

Founder

Recently we did reveal on Twitter that long time friend and colleague-of-many-projects, Peter O’Connor, has formally joined the core team for Serpent OS. Many of you know him as sunnyflunk. He will be working on core toolchain optimisations, benchmarking, and eventually Plasma inclusion.

Peter has long wanted to get the most out of his computing experience, starting from Gentoo back in the 2000’s to ensure there was no wasted resources on features he didn’t use. The desire to over-tinker and use unstable software often led to poor outcomes. While the itch laid dormant for over a decade, it has always been there.

Peter O’Connor (sunnyflunk)

There’s not much about Linux that currently excites me, but I’ve always wanted to see what’s possible with a full LLVM/Clang stack. There’s never been a better time to find out, but it can be difficult to push the envelope when limited with backwards compatibility and supporting older CPU architectures. After experimenting with some tool-chain customisations the chance to test them out more broadly and with a new tool-chain was too hard to pass up. This is the research project I’ve always wanted to undertake and looking forward to using it as my daily driver

We’d like to officially extend our welcome to Peter, as the ramp up gets in place. It should be noted, he is partly to blame for Serpent OS becoming manifest, as the core team got together after discussions on IRC. He has some fantastic ideas, and is a brilliant distribution engineer with proven success.

As an aside, we’re working on improving the website to incorporate a team page, ready for the third team member reveal.

Tags:

• news

The Great Experiment

Jul 1, 2020

Ikey

Founder

So as many have come to realise, we had to rush out a website super quick yesterday as the cat was already out of the bag. One thing that should also be clarified, is our approach to development.

We’re A Lab

Primarily (but not exclusively) Serpent OS (future: Serpent Linux) is an endeavour worked on by the Lispy Snake, Ltd crew. So we’re all about creating awesome technology and trying to further the playing field.

Serpent OS is not, however, owned by Lispy Snake. It is a contributor-based open source project, with specific roles. Those will be clarified over the coming week, but there is a dedicated Project Manager. Hint: It’s not me.

Measuring Success

So, how does an experimental project.. measure success? Quite simply - if others adopt our methodologies or technology, we’ve succeeded. We’re really not looking at “number of installs” or “size of userbase” as a metric, as our chief aim is to build technology.

Options for Longevity

If the experiment is a success, which of course means having tight controls on scope and timescale, then one would assume the primary way to use Serpent OS would be through some downstream repackaging or reconfiguration, i.e. basing upon the project, to meet specific user demand.

It Should Be Fun

This is the heart and soul of all Linux development. Developers should enjoy the involvement and seeing the end result. If we build something more in the end, then that’s a journey we can create together.

Tags:

• news

First Post

Jun 30, 2020

Ikey

Founder

In a nutshell, this post wasn’t meant to be coming out for a few weeks. As it turns out, we had to get ahead of the cat coming out of the bag, and ensure clarity of goals and ambition.

We’ve got an awful lot of work ahead of us across the next few months and weeks, and we strongly suggest checking out the About page which should answer most questions.

TL;DR This is not your typical user-facing Linux distribution.

So go ahead, check out the About page.

Posts will follow in the coming weeks, introducing the core team. It should be pointed out, internally, we’re in a stage1 bootstrap phase, with stage2 ready. Tooling is next on the agenda.

If you’re looking for a modern, lightweight, user-friendly, privacy-focused Linux desktop distribution, then you’re in the wrong place.

Otherwise, stay tuned for the updates. We know our website is insanely basic, but it will be improved when it becomes a priority. You’re more than welcome to join us on #serpentOS on freenode to become part of the community discussion in the mean time.

Til next time.

Tags:

• news