Precisely one month since our end of year summary, so, what have we been up to?
With a new year, a new start was long overdue. We’re pleased to announce that we’ve
finally shifted our butts over to our own server (again).
While this isn’t quite as chunky a news update as the last post, we rather hope you appreciate
the regularity of the news.
‘tarkah’ has been tirelessly porting moss-service, the core foundation of our automated build system,
to a shiny new Rust codebase. While we don’t have an immediately pressing need for the port to
come online, it will certainly address some “v1 issues” and ensure the best-in-class development
experience for our contributors.
In our pursuit for a sustainable, community oriented project, we’ve been doing a lot of in house organisation.
Very importantly, we’ve now relicensed our packaging recipes to MPL-2.0, bringing them in line with the license chosen for our current engineering efforts (moss, etc).
To reiterate a critical point, we require that our contributions are attributed to the virtual collective “Serpent OS Developers”
to ensure we’re unable to relicense any contributions without express consent of the contributors involved.
Additionally we picked MPL-2.0 over Zlib due to some distinct advantages in relation to patent trolls. In combination
we believe both the project and our users will have the greatest protection from patent trolls and fiscally-motivated
relicensing.
As you may recall from the last post, we announced our intention to add system triggers to moss.
We also stated that our triggers must run in isolation thanks to the unique architecture of our
package management. After much consideration, we’ve landed on a strategy that will work for us.
Every mutable operation in moss results in a fresh internal transaction, which is present as a full
/usr tree inside a staging tree. The majority of triggers can be executed here, just before we “activate”
the new rootfs. This allows us to detect potential trigger failures, and abort the activation.
These YAML-defined triggers will be run in an isolated environment (clone-based container) with the new /usr
and read-only access to /etc.
Once we have a freshly activated /usr (ie new system view) we can run a very limited set of post-activation
triggers. An obvious example may be systemctl daemon-reexec. Note that by splitting post-blit and post-activation
triggers, we leave the door open to multiple activation strategies for non-simple updates, including soft reboots
and kernel patching.
With all of the migration work, our builds have also resumed. In the space of a few short days we’ve had 32 package
builds. Ok, not a huge number, but we also have 15 outstanding pull requests to our new recipes repo!
In order to make life not only simpler for us, but for Solus in its future role as a source-derivative of Serpent, we’ve
unified all of our git recipe repos into a new single repo, recipes.
Our devops team has been working hard to mitigate bus factor with our transition to the new infrastructure, Bitwarden account,
shared DNS, and opentofu configuration.
As part of the recovery process for Solus, the two projects shared resources, including
a large Hetzner node. This has actually worked remarkably well for a long time now, however
with Serpent growing we do not wish to impact the availability and stability of the Solus
update experience.
Long story short, money from GitHub Sponsors (You guys! < 3) is now paying for an AX52 server
from Hetzner. It’s sufficiently powerful that we’re running a backup builder there as well as
repo management and build controller.
Looking at our project’s use of static generators - we decided to review our hosting for web content.
Long story short, we deemed it entirely unnecessary. We’re now using GitHub pages to deploy this
website and our documentation site via GitHub actions. Don’t worry,
we use git, we have backups, and we manage the infrastructure using IaC practices.
Lately there have been some questions in both Serpent and Solus as to what our baseline will be.
In Serpent OS, this will be at minimum x86_64-v2, with x86_64-v3x packages automatically
offered if the system base criteria are met.
This decision has been made to offer the widest baseline compatibility without compromising heavily
on system performance (Such that x86_64-generic is unsupported in Serpent OS). We will continue
offering both v2 and v3x until it is no longer feasible from a storage/financial perspective.
Source derivatives (Including Solus) are free to chart their own course here, if necessary,
and still benefit from tight upstream integration.
Burning question - how long before we can use Serpent OS on our development systems?
It’s a fair question - and one that requires context to answer. Let’s start our
new monthly update cycle with a recap of progress so far.
We’d like to extend a huge official welcome, and thank you, to newest team member Cory Forsstrom (aka tarkah)!
Firstly, a quote from Cory:
I’ve been a long time Solus user and was very excited about what serpent os was doing. Really got invested once I started diving into the D code and seeing how powerful the tools and ideas were. The Rust rewrite was just a perfect storm for me with my experience there and my desire to make contributions. Getting to know you and ermo has just been icing on the cake, you’ve both been so welcoming and friendly. So lots of fun times.
On the personal side, I’m on the west coast in the States, have a lovely wife, just had a baby girl and am enjoying my time with fatherhood and coding
Chances are you know tarkah for his contributions to iced-rs, and to Solus. His contributions to
the moss-rs project have been absolutely critical, and his patience essential as we got ourselves quickly up to
speed with Rust. It is entirely fair to say that our Rust efforts would not have been possible without him!
I think it is fair to say that people collectively facepalmed when we announced our plans
to adopt Rust - assuming this would be a huge set back. We’re really
happy to report this is not the case, and we’ve made tremendous progress in this area.
Our Rust based moss system package tool is now leaps and bounds ahead of its predecessor,
supporting the original features and more!
State management
Garbage collection of old transactions (via state prune)
Optimised write order of transactions (see more below)
File conflict detection
Parallel fetching and decompression of packages
Update support (as sync)
Local and remote repositories with priority-based layering
Support for building with musl for fully self-contained, static builds of moss with zero system dependencies.
We now have a version-agnostic “stone” crate that is more efficient at reading packages than our original D code.
In addition, it benefits from the memory safety promises of Rust.
OK, lets look at the problem: Every transaction in moss requires us to generate a new staging tree containing all of the directories and files for the
/usr tree, using hard links, from content addressable storage. As one might expect, creating tens of thousands of new nodes is really, really slow!
To alleviate this issue we created a vfs crate in moss to optimise how we construct each
transaction tree:
Organise all incoming registered paths into a specific order
Inject all missing implicitly created directories for accounting
Re-parent nodes in tree which live under redirected symlinks
Compile into a recursive data-structure using relative names.
There is a multipart “build” for the tree that detects any filesystem conflicts before any files have been written to disk, preventing broken transactions.
With the fully built data structure we can recurse a virtual filesystem, making use of the at family of functions like linkat, mkdirat, to “blit”
a filesystem tree without requiring any path resolution, in the most efficient order possible.
The net result? New transaction roots are created in milliseconds, not seconds.
moss now offers a sync command in place of the conventional upgrade one might expect. Rather than upgrading to the latest version of a package, we
rely on moss and our infrastructure tooling work in conjunction to ensure we can simply synchronize the package selection with the the “tip” candidates
in the configured repositories, as ordered by priority. From a distro-builder PoV, we can now revert and pull builds. For a user PoV, you can remove a test
repository and moss sync to go back to the official versions of packages.
Generally speaking, when you moss sync you will be seeing new packages, however. :smile: This feature is being built to enable a future feature: exported states. Want to try a new edition? Sync to the lockfile. Need to quickly and reproducibly deploy containers from a locked down configuration? You get the idea.
The moss tool is now completely asynchronous, making efficient use of both coroutines and OS threads to perform as much work possible in the shortest space
of time. This allows us to download, fetch, check and unpack many packages at the same time without being blocked on network or disk, greatly speeding up
the fetch part of a transaction.
At the time of writing our port of boulder hasn’t quite yet caught up with the original implementation, given all
of the ground we covered with moss. With that said, we decided to step back and evaluate where we could improve upon
the first version of boulder.
Arguably one of the most important changes: boulder no longer requires running as root. Even better, we make use of
user namespaces and execute using clone - meaning we no longer need a separate binary (mason) within the container
environment!
Thanks to everything being a self-contained binary, we can tell you up front that you’re using unknown action or definition
macros (i.e. %(datadir)) before attempting any builds, saving you valuable time.
When this was first proposed, I did a double take. Breakpoints.. in recipes… ? No, seriously. Not only was it proposed,
but between ermo and tarkah, they only went and implemented it. You can now use special macros within your recipe to add
debugging breakpoints in your recipe to drop to a shell within the container and debug the more difficult builds!
Our documentation will be updated when the new boulder ships a stable build.
We now build both moss and boulder from the moss-rs repository, allowing them to share the same code and mechanisms
for dealing with dependencies, system roots, etc., allowing for far higher levels of integration than were previously possible.
A feature we’re currently working on allows you to use a single directory for the system “root”, where all the unpacking, fetching
and databases happen, while almost instantly creating new build roots!
I get it. “All this rewriting - how will you ever make progress” ? It’s a fair question. We didn’t throw anything
away, quite the opposite. Our repos, tooling, are all still in use enabling us to build the distribution in parallel
to the ongoing rewrites.
For a long time, we’ve had our :material-view-dashboard: dashboard up and running for our build infrastructure.
Sure, we’re going to rewrite it beyond the “Proof of concept” stage when required - but for now it still serves us well. Admittedly there is
a deep issue within the druntime causing thread leaks over time, but we have it restarting via systemd every 24 hours as a bandaid. Needless
to say, we’re big fans of memory safety now.
Our current deployment watches git repos in our :material-github: snekpit collection, and automatically forms build
queues from the pending builds, ordering them by dependency. Using two of ermo’s beefy builders, the packages are automatically built and deployed
to our repository. Our workflow means that maintainers only have to merge a pull request for builds to appear in the volatile repository.
Despite having some warts, the build infrastructure is still able to use legacy moss and boulder to make our new builds available, and we’ve
been minimally patching these old tools to support our transition to the Rust based tooling. As of this moment, moss-rs officially supersedes
the D version, and our new boulder is quickly catching up.
Long story short, we’re still building on all fronts and aren’t blocked anywhere.
Builds are published to our volatile repository, and we do have a fully functioning software repository despite being quite small. Once we’re
happy with the new moss we’ll speed up in our packaging efforts. With that said, we do have two kernels (kvm and desktop) and even gtk-4
at this early stage!
We’re experimenting with system triggers right now, execution units that run to fully bake the OS state after each transaction completes. In keeping
with our stateless philosophy (and pending adoption of hermetic /usr), we need triggers that understand the union state of the operating system, not
individual per-package triggers.
A YAML-based trigger recipe has been baked up, which deliberately avoids the use of shell helpers and execution security weaknesses by using an extended
variant of globs:
Our new :material-github: fnmatch crate, heavily inspired by Python fnmatch, compiles specially format glob strings to Regex, extending with capture
groups for named variables. In YAML, it looks a bit like this:
handlers:
depmod:
run: /sbin/depmod
args: ["-a", "$(version)"]
## Link paths to handlers and filter on the type
paths:
"/usr/lib/modules/(version:*)/kernel" :
handlers:
- depmod
type: directory
Once we roll out the trigger support, we unblock the packaging of critical items like gtk-3 for gdm, as well as enable usable installations and ISOs.
Note that they differ from classical triggers due to the architecture of moss: they need to run in an isolated namespace (container) so we can ensure
the staged transaction won’t break. Additionally we cache and collect the trigger artefacts to speed up each filesystem operation, giving us a first:
versioned state artefacts.
Our plan is to build on my prior work in the Solus boot management design and :material-github: clr-boot-manager, addressing some long standing shortcomings. Most importantly, our new module will not need to inspect or manage the OS filesystem
as moss will be able to provide all of the relevant information (full accounting for all used paths in all transactions).
Initially we will focus on UEFI and supporting the :material-github: Discoverable Partitions Specification by way of XBOOTLDR partitions, the ESP and the Boot Loader Interface. Currently we have no plans to support Unified Kernel Images as the approach taken by CBM (and soon, moss) alleviates the
data concerns of dealing with vfat. However, as and when UKIs gain configurable, standardised behaviour for cmdline control we will investigate their
use. Until that point please note we prebuild our initrd images and ship them directly in our packages, as Solus has done so for years already.
The net gain for taking control of boot management will be the deduplication and garbage collection of assets installed to either the ESP or XBOOTLDR
partitions, along with generation of boot entries relevant to Serpent OS: Transaction specific entries allowing us to directly boot back to older installs
in case upgrades go wrong.
Last, but certainly not least, this approach will make dual-booting with another OS less irritating by no longer being bound by a fixed ESP size.
Our plan is to produce a GNOME Shell based live ISO containing the bare basics to get everything else validated, and open the path to installations.
In short, it will include:
Desktop.
moss tooling
web browser (firefox)
terminal (potentially gnome-console)
flatpak
Eventually: GNOME Software (integrated with flatpak+moss)
It should be obvious we’re not building a general purpose home-use distribution. However, we also won’t stop people using Serpent OS for their needs,
and we plan to beef up the repository, fully support flatpak and eventually custom moss source recipe repos. (ala AUR)
Our journey over the last few years through D, DIP1000, Rust and memory safety has been incredibly eye opening for us. To our own surprise,
memory safety has become a huge interest and concern for the team. Not only have we embarked on a full transition of our tooling to Rust,
we’ve started looking at the OS itself.
As an early indication of our intent, our curl package is now built with the rustls backend and not OpenSSL. Needless to say, we’re very keen to reduce the surface area for attacks by
adopting alternative, safer implementations where possible. Other projects we’re keeping a very close eye on include:
With most of us entering our holiday leave, and the new year practically around the corner, the Serpent OS team wishes you many happy returns and
a wonderful new year. With one blocker being worked on (triggers), and the last on the horizon (boot management), your dream installation of
Serpent OS is firmly within reach.
In the spirit of the season… if you feel like supporting our project, you can help with the rising cost of living and our expanding hosting
requirements via GitHub Sponsors!
Allow me to start this post by stating something very important to me: I absolutely love the
D programming language, along with the expressivity and creative freedom it brings. Therefore
please do not interpret this post as an assassination piece.
For a number of months now the Serpent OS project has stood rather still. While this could naively
be attributed to our shared plans with Solus - a deeper, technical issue is
to be acredited.
D isn’t quite there yet. But it will be, some day.
Again, allow me to say I have thoroughly enjoyed my experience with D over the last 3 or so years,
it has been truly illuminating for me as an engineer. With that said, we have also become responsible
for an awful lot of code. As an engineering-led distribution + tooling project, our focus is that of
secure and auditable code paths. To that extent we pursued DIP1000 as far as practical and admit it has a way to go before addressing our immediate needs of memory safety.
While we’re quite happy to be an upstream for Linux distributions by way of release channels and tooling
releases, we don’t quite have the resources to also be an upstream for the numerous D packages we’d need to
create and maintain to get our works over the finish line.
With that said, I will still continue to use D in my own personal projects, and firmly believe that one day
D will realise its true potential.
Our priorities have shifted somewhat since the announcement of our shared venture with Solus, and we must make
architectural decisions based on the needs of all stakeholders involved, including the existing contributor pool.
Additionally, care should be taken to be somewhat populist in our choice of stacks in order to give contributors
industry-relevant experience to add to their résumé (CV).
Typically Solus has been a Golang-oriented project, and has a number of experienced developers. With the addition
of the Serpent developers, the total cross-over development team has a skill pool featuring Rust and Go, as well as
various web stack technologies.
Reconsidering the total project architecture including our automated builds, the following decisions have been made
that incorporate the requirements of being widely adopted/supported, robust ecosystems and established tooling:
Rust, for low level tooling and components. Chiefly: moss, boulder, libstone
ReactJS/TypeScript for our frontend work (Summit Web)
Go - for our web / build infrastructure (Summit, Avalanche, Vessel, etc)
The new infrastructure will be brought up using widely available modules, and designed to be scalable from the outset
as part of a Kubernetes deployment, with as minimal user interaction as needed. Our eventual plans include rebuilding
the entire distribution from source with heavy caching once some part of the dependency graph changes.
This infrastructure will then be extended to support the Solus 4 series for quality of life improvements to Solus developers,
enabling a more streamlined dev workflow: TL;DR less time babysitting builds = more Serpent development focus.
Our priority these past few days has been on the new moss-rs repository where we
have begun to reimplement moss in Rust. So far we have a skeleton CLI powered by clap with an in-progress library for reading
.stone archives, our custom package format.
The project is organised as a Rust workspace, with the view that stone, moss and boulder will all live in the same tree.
Our hope is this vastly improves the onboarding experience and opens the doors (finally) to contributors.
It should also be noted that the new tooling is made available under the terms of the Mozilla Public License (MPL-2.0).
After internal discussion, we felt the MPL offered the greatest level of defence against patent trolls while still ensuring our code
was widely free for all to respectfully use and adapt.
Please also note that we have always, and continue to deliberately credit copyright as:
This is a virtual collective consisting of all whom have contributed to Serpent OS (per git logs) and is designed to prevent us from
being able to change the license down the line, i.e. a community protective measure.
Despite some sadness in the change of circumstances, we must make decisions that benefit us collectively as a community.
Please join us in raising a virtual glass to new pastures, and a brave new blazing fast 🚀 (TM) future for Serpent OS and Solus 5.
We had intended to get a blog post out a little bit quicker, but the last month has been extremely
action packed. However, it has paid off immensely. As our friends at Solus recently announced
it is time to embark on a new voyage.
Oh no, no. The most practical angle to view this from is that Serpent OS is free to build and innovate to create the
basis of Solus 5. Outside of the distribution we’re still keen to continue development on our tooling
and strategies.
It is therefore critical that we continue development, and find the best approach for both projects, to make
the transition to Solus 5 as seamless as possible. To that end we will still need to produce ISOs and have an
active community of users and testers. During this transition period, despite being two separate projects, we’re both
heading to a common goal and interest.
We have an exciting journey ahead for all of us, and there are many moving parts involved. Until we’re at the point
of mutual merge, we will keep the entities and billing separate. Thus, funds to the Solus OpenCollective
are intended for use within Solus, whereas we currently use GitHub Sponsors
for our own project needs.
Currently Solus and Serpent OS share one server, which was essential for quick turnaround on infrastructure enabling.
At the end of this month Serpent OS will migrate from that server to a new, separate system, ensuring the projects
are billed separately.
Long story short, if you wish to sponsor Serpent OS specifically, please do so via our GitHub sponsors account as our monthly running costs
will immediately rise at the end of this month. If you’re supporting Solus development, please do visit them and sponsor them =)
Glad you asked! Now that the dust is settling, we’re focusing on Serpent OS requirements, and helping Solus where we can.
Our immediate goals are to build a dogfooding system for a small collection of developers to run as an unsupported prealpha
configuration, allowing us to flesh out the tooling and processes.
This will include a live booting GNOME ISO, and sufficient base packages to freely iterate on moss, boulder, etc as well
as our own infrastructure. Once we’ve attained a basic quality and have an installer option, those ISOs will be made available
to you guys!
After many months and much work, our infrastructure is finally online.
We’ve had a few restarts, but it’s now running fully online with 2 builders, ready to serve builds
around the clock.
Firstly, I’d like to apologise for the delay since our last blog post. We made the decision to move
this website to static content, which took longer than expected. We’re still using our own D codebase,
but prebuilding the site (and fake “API”) so we can lower the load on our web server.
This is the page you can see over at dash.serpentos.com. It contains
the build scheduler. It monitors our git repositories, and as soon as it discovers any missing builds
it creates build tasks for them. It uses a graph to ensure parallel builds happen as much as possible,
and correctly orders (and blocks) builds based on build dependencies.
We have 2 instances of Avalanche, our builder tool, running. This accepts configuration + build requests
from Summit, reporting status and available builds.
Super early days with the infra but we now have builds flowing as part of our continuous delivery solution.
Keeping this blog post short and sweet… we’re about to package GNOME and start racing towards our first
real ISOs!
