Blog

Serpent OS Enters Alpha

Dec 23, 2024

Ikey

Founder

Following a successful prealpha phase, we’re excited to announce that Serpent OS has now entered the alpha stage of development. This milestone marks a significant step forward in the project’s journey. While the usual disclaimers apply, as a certain level of project fluidity is to be expected, we invite you to explore the latest alpha release and experience the cutting-edge features and improvements we’ve introduced.

Serpent OS

Supporting Serpent OS

Serpent OS is a community-driven project, and we welcome contributions from developers, testers, and users alike. If you’re interested in helping shape the future of Linux distributions, we invite you to join our growing community and get involved in the project. Whether you’re interested in packaging, development, documentation, testing, or community support, there are many ways to contribute to Serpent OS.

Lastly, you can support the project by becoming a sponsor. Your support helps cover our hosting costs and allows us to dedicate more time to making Serpent OS exceptional. We’re grateful for the support we’ve received so far and look forward to continuing to build something meaningful in the Linux ecosystem.

Sponsor Serpent OS

What is Serpent OS?

Serpent OS is a heavily engineering-led Linux distribution seeking to redefine how we distribute Linux. It is a stateless OS that leverages atomic updates, cutting-edge tooling, and rock-solid reliability to deliver a safe and efficient system. Built by industry veterans with decades of experience, Serpent OS represents the next evolution in Linux distributions.

We currently offer x86_64-v2 desktop builds, for both GNOME and COSMIC desktop environments. We only support UEFI systems, and plan secure boot support via shim in the near future. The majority of the distribution (including the kernel) is built against the LLVM toolchain, with libc++ as the default C++ standard library.

Through our unique architecture, we’ve combined proven concepts into a cohesive system where updates either fully complete or safely roll back, keeping your system running reliably. Our atomic updates let you see changes immediately, without requiring reboots just to apply updates. While certain updates like kernels will still need a reboot, we’ve designed Serpent OS to be as seamless and user-friendly as possible.

What’s new in the alpha release?

The alpha release introduces several key features and improvements that enhance the overall user experience and system performance. Here are some highlights of what you can expect from Serpent OS alpha:

Hardware support

We’ve rounded out our hardware support, now including patches for ASUS devices and Surface devices. Additionally, we now have the NVIDIA driver available in the repositories. Note we only offer open-gpu-kernel-modules, which we prebuild against our linux-desktop package to streamline installation.

Steam

You can now install Steam from our repos, making use of our newly enabled multilib drivers:

mesa-32bit
nvidia-graphics-driver-32bit

Steam on Serpent OS

More Rust

As part of our efforts to modernise and strengthen the Serpent OS base, we’ve switched some components out for Rust alternatives:

uutils-coreutils replaces coreutils
sudo-rs replaces sudo
ntpd-rs replaces timesyncd
curl is built with rustls support (and hyper but this is being dropped upstream)

Additionally we include starship by default on our GNOME ISO, along with zed, loupe, resources and other Rust applications.

Packages

Our recipes repository has seen over 1600 commits since our prealpha, with some of the highlights including:

Linux 6.12.6
Firefox 133.0.3
LLVM 18.1.8
GNOME 47.2
COSMIC 1.0.0_alpha4

Tooling updates

We’ve made significant improvements to our tooling, including updates to moss, boulder, and blsforme. These tools are essential for managing packages, building software, and handling boot management in Serpent OS.

moss - Package Management

Our atomic package manager swaps the entire /usr directory during system updates, ensuring a stateless, bulletproof upgrade process. Updates either succeed completely or not at all. Recently we made improvements to moss to handle certain special cases:

EMLINK - ext4 is limited to 65k hardlinks per inode, and we were hitting this limit on some systems due to empty files. We now handle this case gracefully by creating new inodes for empty files.
ENOSPC - We now mitigate the risk of ENOSPC on the boot partition by performing automatic cleanup of old kernels and initrds. This paves the way for us to land transaction-specific entries for the offline rollback feature.

blsforme - Boot Management

Automated boot management that just works. Seamlessly handles your EFI System Partition and boot entries without manual intervention.

We’ve improved blsforme to support drop-in command line snippets for kernel parameters, allowing for easy customisation of boot entries. These can be supplied by packages, or by the user directly. This has allowed us to make plymouth work out of the box, in tandem with our prebuilt reproducible initrd images.

We also now support multiple initrd images, which in turn allowed us to land support for early KMS in the initrd for NVIDIA users.

lichen - Installer

We’ve revamped the UI for Lichen, and now allow picking between xfs, ext4, and f2fs for the root filesystem. The team advises against ext4 for systems where more than a few hundred transactions will need to be retained, due to ext4 limitations with hardlink counts.

Sync protections

In preparation for “versioned repositories”, we’ve added a gate to prevent the automatic builds on our infrastructure from directly reaching users. Once we’re happy with our validation, we then “sync” the repositories to the public mirrors. This ensures that users are not exposed to any potential issues with the latest builds.

What’s next?

With us now entering the alpha phase, we’re looking to grow our contributor base and community. Our immediate focus is improving/creating documentation to enable more people to use Serpent OS as a daily driver in the near future. We’re also going to keep iterating on our tooling, ensuring a reliable and efficient system for all users.

We’ll begin deeper integration of the tooling, such as moss with packagekitd for GNOME Software and COSMIC Store use.

Issues and feedback

If you encounter any issues or have feedback to share, please don’t hesitate to reach out to us. You can report bugs on our GitHub issue tracker or join our Matrix space to chat with other users and developers.

We are aware of some known issues in the alpha release, and we’re actively working to address them in future updates. Your feedback is invaluable to us, and we appreciate your help in making Serpent OS the best it can be.

Performance of moss reduced significantly in VMs on first run
NVIDIA driver not working on some systems
Lichen still deliberately limited to “seeing” pre-formatted partitions. Requires GPT disk, with ESP and / partitions pre-created.
Lichen locked to net install mode, a moss export functionality will be added in the future to address this.
XBOOTLDR partition (fat32, bls_boot flag in gparted) of 2GB recommended for ideal boot management
Small repos - deliberate, to ensure maintainer story is solid before we scale up
Secure boot support pending secure key distribution support within the build infrastructure
Due to how our atomic upgrades work using renameat2, GNOME Shell doesn’t currently detect new applications (.desktop) files when installed, requiring a reboot. We’re working on a fix for this.

Finally

We’re excited to share the alpha release with you and look forward to your feedback and contributions. Thank you for your continued support and interest in Serpent OS. We’re committed to building a safe, efficient, and reliable system that empowers users to get the most out of their computing experience.

Serpent OS Prealpha0 Released

Aug 1, 2024

Ikey

Founder

Well, it didn’t take us that long, really … Our technical preview, prealpha0, is now available for testing!

wip boot code

Head to our download page to grab serpent-os-prealpha-0.iso now!

This is a super rough version of Serpent OS that is capable of being installed on baremetal hardware and VMs that support UEFI and OpenGL acceleration. It is however not recommended for use daily due to the early nature and a bunch of packages being wildly out of date. It features a minimal GNOME desktop with the Firefox web browser and a terminal.

Right now it contains a CLI installer that can be accessed from the terminal by typing:

sudo lichen

To detract from casual use it is necessary to manually partition the disk before running the (net) installer. You can use fdisk to create a GPT disk with a (mandatory) EFI System Partition and an optional XBOOTLDR partition. This currently also needs to be FAT32 until we integrate systemd-boot driver support. Lastly of course you will need a large enough root partition.

You will need a working network connection, so make sure you’re connected before starting the installer!

Technologies involved

This will of course appear to be a very rough (crap) prealpha ISO. Underneath the surface it is using the moss package manager, our very own package management solution written in Rust. Quite simply, every single transaction in moss generates a new filesystem tree (/usr) in a staging area as a full, stateless, OS transaction.

moss

When the package installs succeed, any transaction triggers are run in a private namespace (container) before finally activating the new /usr tree. Through our enforced stateless design, usr-merge, etc, we can atomically update the running OS with a single renameat2 call.

As a neat aside, all OS content is deduplicated, meaning your last system transaction is still available on disk allowing offline rollbacks.

Translated: Like A/B swaps? Don’t like rebooting? … Ok that explained it.

blsforme

A few crates deep we find blsforme - a library for automatically managing the ESP and XBOOTLDR partitions. Whenever kernels and associated files are present in the OS filesystem, they are synchronised to the boot partitions along with automatically generated boot entries and cmdlines. Specifically this means moss can discover and mount necessary partitions according to the disk topology, GPT entries, and Boot Loader Specification, generating configurations by scanning your local rootfs to build the proper root= parameters for you.

Translated: Magic boot management makes way for offline rollbacks.

Warning

Super pre-alpha. Will 100% break! We just wanted you for the journey <3

Start of public iterations

Ok you have this super rough ISO, what next? We now have an actual startpoint and will continue to iterate on the ISOs, delivering new installer updates/improvements and removing the unnecessary live mode from the ISO completely.

The next release will also feature more installer options so you can fresh-install the Cosmic Desktop (in repos now) For the brave, go forth and sudo moss sync -u ! (ok you want moss help first.)

Closing word

As a distro, it’s kinda crap right now. The tooling has been our focus for years and now we can actually build something with it. With only a handful of packages, flatpak is your best friend. Or you could swing a PR into our recipes repo!

The project is only possible with your support. Something tells me that putting out this ISO is going to somewhat increase our hosting costs and stress the datacenter hamsters.

Feel free to sponsor to support our work and increase our capacity!

Calm Before The Storm

Apr 30, 2024

Ikey

Founder

It’s that time of month again, and we have some details to share with you on boot management, as well as plans for real installs landing in May. Additionally, plans for “deferred updates” via mandatory reboots have been dropped!

wip boot code

Project priorities

To better understand the technical deep-dive on boot management further below, we should clarify a slight change in priorities for the project as a whole. For some time now we’ve served as a technical beacon for Solus to guide them towards better strategy and technical solutions whilst awaiting a situation where a rebase is feasible.

While we still look forwards to that future, the present day is the most important to us. Serpent OS must now transition from a collection of tools and promises into a daily driver with actual users. We still aim to cater to the more developer/technical oriented end of the spectrum rather than a “desktop distro”, and want to start this journey yesterday.

The boot management code is currently landing over the coming days, and will be immediately followed by an incredibly rudimentary system installer to facilitate an initial adoption of Serpent OS for the general public. Our intent is to ship some early test ISOs for supporters, and by the end of May have a feasible option for people to use and contribute to Serpent OS on real hardware.

We acknowledge this is an exchange of the pursuit of perfection for a pragmatic rapid iteration cycle and believe the best way forward is to open the doors wide, fixing issues on the job. The initial system will likely be rough and buggy, but it is an official starting point from which we can all build Serpent OS together.

Moss + Boulder

Our Rust tooling has now been in “production” use in our development environment and build servers for the last month without any major hiccup. We have recently discovered some minor issues in our emul32 (32-bit) package generation which are being resolved, chiefly our move from the improperly named ISA x86, now updated to 386 affecting 32-bit dependency chains.

Boot

Our main topic of conversation this month is the enabling of boot management for Serpent OS installations. To be crystal clear, we are not talking about a boot loader. In fact, Serpent OS uses systemd-boot for UEFI. Instead, we are talking about the automated management of boot loader entries, their corresponding kernel/initrd assets, cleanup from old states, etc.

In a former life I authored the clr-boot-manager tool to manage this in an agnostic fashion and this served well for a number of years. However, existing as an externally invoked binary meant that the tool had to become responsible for all accountancy and asset lifetimes.

Enter … moss-boot

Ok - technically it’s crates/boot in the moss repo. This is a Rust crate that will be integrated into the main moss project to provide the aforementioned boot management facilities. A huge advantage for us is that moss can only allow a single version of a package to exist within a given state, and all states are numerically identified due to the transaction nature of our package manager.

In simple terms, one “state” == “one kernel candidate” (per type..) so all we really need to do for now is identify the ESP in use by way of the Boot Loader Interface and any XBOOTLDR partition. Then, map our “boot environment” into something moss-boot can convert into boot entries, and voila..

At a more technical level, moss-boot will handle the following:

form candidate cmdline from filesystem snippets
auto-generate the root= parameter by probing the / partition (LVM2, etc)
record the initrd needed
compare and atomically write “missing” files in FAT-respecting fashion (copy/unlink/rename)
garbage collect unused boot entries

Live atomic updates and kernels…

An interesting, perhaps not widely considered result of upgrading a kernel, is filesystem path changes. If for any reason the kernel’s module tree becomes inaccessible (say, due to updating to a new version) then it becomes impossible to load the old kernel modules. To remedy this, distributions will tend to leave multiple versions of the kernel on disk, invariably within mounted /boot.

The approach Serpent OS is taking will mean that like Solus, /boot will not need to be premounted nor will any package ship files in that directory. Unlike the existing design in Solus 4, we will not leave kernels behind on upgrade. Instead, they become locked to the system state (transaction) in which they are used.

To solve the “missing version” issue, we will have moss record the qualified snapshot path (in /.moss) within the /run tree so that once the /usr tree has been swapped with a new transaction atomically, a patched kmod package will fall back to probing the suddenly orphaned kernel tree in the cached /run location.

This will mean that the /lib/modules tree may not have the current kernel version, but the OS will still be usable while having had a live atomic update. Of course, to use the new kernel you must reboot. Unlike other atomic OS implementations, it will be up to you when you do so: no more deferred updates!

Next time

We are aiming to get back to monthly blogs on the 19th, but the cycle for the coming 6 weeks may be quite interesting. Expect out of schedule posts looking for testers and volunteers while we use Serpent OS to build Serpent OS, enabling a community of pioneers benefiting from the unique design of our architecture: a full-featured OS featuring hermetic /usr, offline rollbacks, and live atomic upgrades with containerised system triggers.

The March Of Progress

Mar 31, 2024

Ikey

Founder

Despite a brief excursion out of the country for a first-in-a-lifetime holiday, I’m happily back at the desk to bring you up to date with the latest goings in Serpent OS. TLDR: Loads of awesome, baremetal is enabled, ISO cycle in next couple of weeks.

GNOME on Serpent OS

Tooling Improvements

OK, we all know you want to know about the baremetal stuff, but c’mon, check out this changeset!

Boulder

We’re officially at a point where the Rust boulder implementation has become our “blessed” build tool. This is actually a crate within the moss repository, allowing the two tools to share huge chunks of code. Our focus over the past few weeks has been to ensure we can drop the tool into our existing build network and “just work”, while making it significantly faster.

One of the most important high level changes is performance, in some cases virtually 50% faster packaging times. Our package payloads are now compressed using multi-threaded zstd compression, and a bunch of internal components were reworked to use faster code paths leading to a significant speed up in the raw generation of each .stone package.

!!! warning “Legacy boulder tool has been retired”

As of March 31st, 2024, the legacy D implementation of `boulder` is no longer supported. The runtime component, `mason`, has been removed from the
volatile repository. Our new tool is written in Rust and contains the `moss` code, enabling a self-contained all-in-one approach to buildroot  management.

It relies on clone-based user namespaces, so is fully "rootless". Please migrate immediately to the new Rust based tool. Not only is it more advanced,
it is significantly faster at generating the raw `.stone` packages.

Comparison

Other, quite awesome changes:

Fix manifest encoding for 0 nul byte, interop with build systems
Explicit --update flag to refresh the repositories prior to build
Defaults to multithread zstd compression
Slick timing reports
Base-enabling landed for “meta” packages (contain no files or content)
Added BuildRelease control via tooling for automated package rebuilds

Moss

Moss also got some love these past few weeks, including a full migration over to diesel.rs. Combined with a limiting of async to largely network bound operations, and some clever performance optimisations under the hood, moss is now faster than ever. In fact, we’re able to generate desktop ISOs in under 20 seconds, and install the full GNOME desktop to a virtiofs VM in almost no time.

Dropped some unnecessary dependencies
Significant code cleanups
Further refined our system + transaction triggers to fully respect dependency ordering
Augmented state management with new functionality to activate/rollback to a specific transaction by ID using offline store.
Various optimisations:
- Drop expensive Path usage in vfs crate
- Optimise queries in our transaction management to speed up DAG initialisation and filesystem blit
- And more to be covered in a future blog post

Hardware/Desktop Enablings.

Rather than extensively widen our repository at this point we’ve opted for further low-level enabling of components required for the developer experience desktop. This includes Network Manager, the desktop kernel package, Vulkan + mesa integration, etc.

Gnome Software

As well as enabling core applications and features such as GNOME Software and Flatpak, we’ve also enabled preliminary usage on baremetal booting hardware. Note, this isn’t yet installable but we do plan to offer early access installable images to our sponsors!

Also worth noting.. we’ve begun to package the Cosmic Epoch desktop from the cool guys over at System76. We’re highly interested in this project and plan to track progress closely, offering it within Serpent OS as a fully featured development focused desktop experience.

The Agenda

This post was finished moments before the March deadline, so needless to say we’ve been super busy! Thankfully a lot of pivotal pieces have landed now and we can focus on some big ticket items. Our primary concern is to ship one last “pre-alpha” image for the public, and a test installable pre-alpha ISO for our sponsors to help gain early feedback. The hope is this will help fund a widening of our infrastructure to support more concurrent repository users and our own debuginfod compatible extensions to our repository.

The April (and beyond) agenda:

Finish pre-alpha stage with some shiny ISOs!
Enter alpha by adding versioned repository support to moss for an “upgrade forever” guarantee
Enhance our ABI tracking story to vastly simplify huge rebuild queues and automation
Expand our native hardware support
Add some missing basic triggers such as Fedora-compatible SSL certificate management
Begin work on the all-important boot management in moss! (not boot LOADER, this will be systemd-boot)
Add battery testing of our tooling to our CI pipelines to protect users
Ensure all of our code is fully documented
Greatly improve our onboarding and documentation for packaging and usage of Serpent OS, focusing on stone.yaml in depth
Get core team running Serpent OS daily. WiFi and GPU support are in place already.
Build an amazing developer experience

On behalf of the Serpent OS team, I look forward to sharing some amazing updates with you towards the end of April! Keep your eyes peeled for the ISO drops in the coming weeks.

End of February Update

Feb 29, 2024

Ikey

Founder

import { Aside } from ‘@astrojs/starlight/components’;

This update came a little later in the month, as we’ve got a lot of exciting news to share. Everything from boulder in Rust, to the GNOME 45 Desktop complete with moss triggers built atop a rebootstrapped toolchain.

Rusty Boulder

We’re pleased to announce that over the course of this weekend, once testing has completed we’ll deploy the latest version of boulder, our packaging build tool. This has been given the Rust treatment, directly sharing the codebase with moss.

Boulder in action

Unlike our PoC implementation, the new boulder makes use of the clone syscall to execute portions of itself under user namespaces, eliminating a longstanding deployment irritation of having the main build binary inside packaging.

Just like moss it makes use of tokio for async right where we need it, and is rootless to make life easier for our contributors. Far from being just a direct port of the proof of concept, it adds new tools to empower developers, such as breakpoints in packaging recipes.

Breakpoints, in recipes

Triggers, integrated!

After much discussion we finally integrated support for triggers in moss. These are actions that are executed at different stages of package management operations to finalize or “bake” some state based on the unified components of the installation.

Note that right now triggers are re-executed for each transaction, and no caching support is yet in place. With that said, they’re still very fast and we plan to add a store based cache to prevent unnecessary execution. One of the greatest benefits of our triggers is ensuring these actions run within the right context under a namespace, or container.

In pursuit of stateless

We’ve iterated many times over the years our belief in the stateless philosophy, and as a design decision we mandated that packages can only contain /usr files. This has meant we’ve had to get a bit inventive in shipping working packages out of the box.

One such strategy has been to abolish the reliance on package triggers for system users and groups. In the past we’ve relied on systemd-sysusers to construct these accounts, such as gdm. A downside is there is no simple way to handle the cleanup of these accounts.

We’re now happily using systemd userdb user and group drop-in records via nss-systemd to provide the default users + groups (including users) group with centrally managed, fixed UIDs and GIDs in our git recipes.

GNOME Desktop

Thanks to triggers now being fully integrated in Serpent OS via moss, we’ve actually managed to package up most of the basic parts of GNOME 45. This includes a functioning GDM, GNOME Shell and a handful of applications.

GNOME on Serpent OS

Packaging work

Outside of the heavy work on the tooling and GNOME, we’ve also been tidying up our recipes repo and recently performed a fresh rebootstrap using:

glibc 2.39
binutils 2.42
GCC 13.2.0
LLVM 17.4.0
Rust 1.78.0

Note that our primary toolchain is glibc, lld, clang / clang++, libc++, libunwind, etc. The majority of packages are built with this LLVM + glibc toolchain, and a subset of packages currently build with the GNU toolchain due to various wrong assumptions or use of GCC extensions.

Our kernel has always been built with clang.

Images

Our shortterm plan is to deliver a prealpha image for the general public to test, and to get a “feel” for Serpent OS and associated tooling. As soon as this is out of the door we will immediately pivot to getting Serpent OS fully functioning on baremetal, with explicit targets in mind: the hardware that core developers are currently using.

These images will be available but not promoted for general use - given their intended use as dogfooding systems. This is part of our shift to the oxide-alpha-1 target milestone, which will be discussed further in our next post announcing the prealpha image.

Next time

Our next monthly blog post should return to its regular slot, with updates on the following topics:

Limiting async in moss to targeted areas (PR ready and pending review)
Life on Serpent: How baremetal testing + enabling is going.

Until then, please, stay tuned!