Skip to content
AerynOS
Blog

Blog

Build System Functional

Ikey
Founder

Wow, has it been a hectic few weeks, and it definitely shows: last time we blogged it was about re-bootstrapping with glibc. Feels like ancient news already! So, what’s new in the world of Serpent OS? Apart from yours truly now being proud parent to a beautiful baby girl, work has resumed on the development of Moss, our package manager. And it builds stuff. Awesomely. Let’s quickly catch up with those updates and see where we’re headed next.

Look at all the buildiness

It lives!

Section titled “It lives!”

We’re now able to build the majority of a moss package. Notice I’ve made a distinction there. So, we’re able to run the build instructions, and use all of the metadata, configuration and macros available. The only thing we’re not actually doing is dumping the built files into a binary package.

We’re able to do the build-system part rather well, now. Right now we support the following features in the build system:

  • Multiple, profile-based architecture support
  • Automatic -m32 profile based cross compilation support for 32-bit libraries on 64-bit OS
  • Basic Profile Guided Optimisation via the workload key. Set a workload, optimise accordingly.
  • LLVM Context Sensitive Profile Guided Optimisation. This is default for LLVM with the workload key, and comes for free, with multiple build stages.
  • Profile based tuning options, such as optimize for speed, size, disabling hardening, etc.
  • Trivially switch between gnu and llvm toolchain in stone.yml, with profiles knowing the right flags to use with tuning options.
  • Recursive macro support in build scripts, defined on per-profile level
  • Architecture specific profiles support in stone.yml, i.e. profiles -> aarch64 -> setup: to avoid if/else spaghetti.

Next on the agenda

Section titled “Next on the agenda”

Now that the huge amount of scaffolding work has been done, we can actually turn the results of builds into installable binary packages using our moss.format.binary module. We’ll add some magic sauce to have automatic subpackages + inter-package dependencies, along with the expected automatic runtime dependencies + such. Cue some linting, et voila, build tool that’s a pleasure to work with.

Once we have all those packages building, we’ll need a way to install them. Luckily some scaffolding is in place for this already, and it won’t take much effort to support moss install somepkg.stone. Then we throw a few dozen packages in the mix, add some dependency handling, repo support, Bob’s your uncle, and your aunt is downloading exclusive early-access images from our Open Collective as soon as they’re available. :O

Results Of The Experiment

Ikey
Founder

import FigureScreenshotOne from ”@/components/ui/FigureScreenshotOne.astro” import featuredImage from ”./Featured.webp”

It seems like only yesterday we announced to the world a Great Experiment. It was in fact 2 months ago, and a whole lot of work has happened since that point. A few take-homes are immediately clear, the primary one being the need to be a community-oriented Linux distribution.

To quote ourselves 2 months ago:

 If the experiment is a success, which of course means having tight controls on scope and timescale,
 then one would assume the primary way to use Serpent OS would be through some downstream repackaging
 or reconfiguration, i.e. basing upon the project, to meet specific user demand.

It turns out that so far the experiment has been successful, and being forkable is still at the very heart of our goals. Others have joined us on our journey, and expressed the same passion in our goals as we have. A community has formed around the project, with individuals sharing the same ambitions for a reliable, rolling operating system with a powerful package manager.

An open community

Section titled “An open community”

Over the past 2 months we’ve transformed from set of ideas into a transparent, community-first organisation with clear leadership and open goals. I’ve stepped into the Benevolent Dictator For Life position, and Peter has taken on daily responsibilities for the project running. Aydemir is now our treasurer on Open Collective, and many individuals contribute to our project.

Transactional package management

Section titled “Transactional package management”

No need to rehash this, but the defining feature of Serpent OS has clearly become moss, something initially not anticipated when we started. A read-only root filesystem, transactional operations, rollbacks, deduplicating throughout and atomic updates. Combine that with a rolling release model, stateless policy and ease-of-use, the core feature-set is already powerful.

Deferral of musl integration

Section titled “Deferral of musl integration”

In recent weeks we’ve been working on libwildebeest and libc-support, primarily as a stop-gap to provide glibc compatibility without using glibc. While musl has many advantages, it is clear to us now that writing another libc through our support projects isn’t what we originally planned. With that in mind we’re adopting glibc and putting our musl works under community ownership, until such time as reevaluation shows that musl is what is needed in Serpent OS. Note the primary motivator here is investing our efforts where it makes sense, and obtaining the best results in the most manageable fashion for our users.

Our new toolchain configuration will be as follows:

  • LLVM/clang as primary toolchain
  • libc++ as C++ library
  • glibc as C library
  • gcc+binutils (ld.bfd) provided to build glibc, elfutils, etc.
  • Permitting per-package builds using GCC, i.e. kernel to alleviate Clang IAS issues.

Thus our toolchain will in fact be a hybrid GNU/LLVM one. This will allow both source and binary compatibility with the majority of desktop + server Linux distributions, facilitating choice of function for our users.

It should be noted this decision has been made after much discussion internally, on our IRC, on our OpenCollective, etc. Our bootstrap-scripts is being improved to support both glibc and musl, so that the decision can continuously be reviewed. If we reach a position whereby musl inclusion once again makes sense, thanks to atomic updates from moss, it will be possible to switch.

Final thoughts

Section titled “Final thoughts”

Initially Serpent OS emerged as a collective agreement on IRC as a set of notions as opinions. Over the past few months those opinions have solidified into tangible ideas, and a sense of community. In keeping with what is right for the community, our messaging has been reworked.

It is fair to say our initial stance appeared quite hostile, as a bullet-point list of exhaustion with past experiences. As we’ve pivoted to being a fully community-oriented distribution, we’ve established our goals of being a reliable, flexible, open, rolling Linux distribution with powerful features imbued by the package manager, and an upstream-first approach.

As such we’ve agreed to not let our own pet-peeves interfere with the direction of the project, and instead enable users to do what they wish on Serpent OS, be it devops, engineering, browsing, gaming, you name it. We’re a general purpose OS with resilience at the core.

Our focus is on the usability and reliability of the OS - thus our efforts will be invested in areas such as the package manager, hardware enabling, the default experience, etc.

So, strap yourself in, as we’re fond of saying. Development of Serpent OS is about to accelerate rapidly.

Source Format Defined

Ikey
Founder

Following quickly on the heels of yesterday’s announcement that the binary format has been defined, we’ve now implemented the initial version of our source format. The source format provides metadata on a package along with instructions on how to build the package.

The next step of course, is to implement the build tool, converting the source specification into a binary package that the end user can install. With our 2 formats defined, we can now go ahead and implement the build routines.

Very trivial package recipe

A YAML based format

Section titled “A YAML based format”

The eagle-eyed among you will already see this is a derivation of the package.yml format I originally created while at Solus. Minor adaptations to the format have been made to support multiple architectures via the profiles key, and package splitting behaviour has now been grouped under a packages key to make the structure more readable.

In package.yml, one would have to redefine subpackage summaries as a key in a list of the primary summary key, such as:

    rundeps:
        - primary-run-dep
        - dev: secondary-run-dep
    summary:
        - Some Summary
        - dev: Some different summary

We’ve opted to group “Package Definition” behaviour into core structs, which are allowed to appear in the root-level package and subpackages:

    summary: Top-level summary
    packages:
        - dev:
            summary: Different summary
            rundeps: Different rundeps

Multiple architecture support

Section titled “Multiple architecture support”

In keeping with the grouping behaviour, we’re baking multiple architecture configurations into the YML file. A common issue encountered with the older format was how to handle emul32:

    setup: |
        if [[ -z "${EMUL32BUILD}" ]]; then
            %configure --some-emul32-option
        else
            %configure
        fi


    build: |
    %make

Our new approach is to group Build Definitions into the root level struct, which may then individually be overridden for each architecture. For example:

    profiles:
        - ia32:
            setup: |
        %configure --some-emul32-option
    setup: |
    %configure

Permutations

Differences

Section titled “Differences”

As you can see it is highly similar to package.yml - which is a great format. However, with our tooling and aims being slightly different, it was time to reevaluate the spec and bolster it where appropriate. We’re happy to share our changes, but in the interest of not causing a conflict between the 2 variants, we’ll be calling ours “stone.yml”.

Our main motivation came from the tooling, which is written in the D language. With D we were able to create a strongly typed parser and explicit schema, and with a struct-based approach it made it more trivial to group similar definitions.

Other than that, we have the same notions with the format, intelligent automatic package splitting, ease of developer experience, etc.

Moss Format Defined

Ikey
Founder

The core team have been hard at work lately implementing the Moss package manager. We now have an initial version of the binary format that we’re happy with, so we thought we’d share a progress update with you.

Development work on moss

Explaining the format

Section titled “Explaining the format”

Briefly, the binary container format consists of 4 payloads:

  • Meta (Information on the package)
  • Content (a binary blob containing all files)
  • Index (indices to files within the binary blob)
  • Layout (How to apply the files to disk)

Each payload is verified internally using a CRC64-ISO, and contains basic information such as the length of the payload both compressed and uncompressed, the compression algorithm used (zstd and zlib supported) as well as the type and version of the payload. All multiple-byte values are stored in Big Endian order (i.e. Network Byte Order).

Payloads

Internally the representation of a Payload is defined as a 32-byte struct:

    @autoEndian uint64_t length = 0; /* 8 bytes */
    @autoEndian uint64_t size = 0; /* 8 bytes */
    ubyte[8] crc64 = 0; /* CRC64-ISO */
    @autoEndian uint32_t numRecords = 0; /* 4 bytes */
    @autoEndian uint16_t payloadVersion = 0; /* 2 bytes  */
    PayloadType type = PayloadType.Unknown; /* 1 byte  */
    PayloadCompression compression = PayloadCompression.Unknown; /* 1 byte */

We merge all unique files in a package rootfs into the Content payload, and compress that using zstd. The offsets to each unique file (i.e. the sha256sum) are stored within the Index payload, allowing us to extract relevant portions from the “megablob” using copy_file_range().

These files will become part of the system hash store, allowing another level of deduplication between all system packages. Finally, we use the Layout payload to apply the layout of the package into a transactional rootfs. This will define paths, such as /usr/bin/nano, along with permissions, types, etc. All regular files will actually be created as hard links from the hash store, allowing deduplication and snapshots.

The Meta payload consists of a number of records, each with strongly defined types (such as String or Int64) along with the tag, i.e. Name or Summary. The entire format is binary to ensure greater resilience and a more compact representation. For example, each metadata key is only 8 bytes.

    @autoEndian uint32_t length; /** 4 bytes per record length*/
    @autoEndian RecordTag tag; /** 2 bytes for the tag */
    RecordType type; /** 1 byte for the type */
    ubyte[1] padding = 0;

TLDR that for me…

Section titled “TLDR that for me…”

Binary format that is self deduplicating at several layers, permitting fast transactional operations.

Up Next

Section titled “Up Next”

Before we work any more on the binary format, we now need to pivot to the source format. Our immediate goal is to now have moss actually build packages from source, with resulting .stone packages. Once this step is complete we can work on installation, upgrades, repositories, etc, and race to becoming a self hosting distribution.

Note, the format may still change before it goes into production, as we encounter more cases for optimisation or improvement.

Defining Moss

Ikey
Founder

Over the past few weeks, throughout the entire bootstrap process, we’ve been deliberating on what our package manager is going to look like. We now have a very solid idea on what that’ll be, so it’s time for a blog post to share with the community.

Initial moss prototype CLI

Old, but new

Section titled “Old, but new”

The team has been very clear in wanting a traditional package management solution, whereby repositories and packages compose the OS as a whole. However, we also want modern features, such as being stateless by default. A common theme is wanting to reduce the complex iterations of an OS into something that is sanely versioned, but also flexible, to ensure a consistent, tested experience with multiple end targets.

Additionally, the OS must be incredibly easy for contributors and team members to maintain, with intelligent tooling and simple, but powerful formats.

Atomic updates

Section titled “Atomic updates”

One of the most recent software update trends of recent years is atomic updates. In essence this allows applying an update in a single operation, and importantly, reversing an update in a single operation, without impacting the running system.

This is typically achieved using a so-called A/B switch, which is what we will also do with moss. We won’t rely on any specific filesystem for this implementation, instead relying on a smart layout, pivot_root and a few other tricks.

Primarily we’ll update a single /usr symlink to point to the current OS image, with / being a read-only faux rootfs, populated with established mountpoints and symlinks. Mutation will be possible only via moss transactions, or in world-writable locations (/opt, /usr/local, /etc …)

Deduplication

Section titled “Deduplication”

The moss binary package format will be deduplicated in nature, containing hash-keyed blobs in an zstd compressed payload. Unique blobs will be stored in a global cache, and hard-linked into their final location (i.e. /moss/root/$number/usr/...) to deduplicate the installed system too. This allows multiple system snapshots with minimal overhead, and the ability to perform an offline rollback.

Implications

Section titled “Implications”

We’ll need to lock kernels to their relevant transactions (or repo versions) to prevent untested configurations. Additionally the boot menu will need to know about older versions of the OS that are installed, so they can be activated in the initrd at boot. This will require us doing some work with clr-boot-manager to achieve our goals.

Difference from existing solutions

Section titled “Difference from existing solutions”

We’re trying to minimise the iterations of the OS to what is available in a given version of the package repositories. Additionally we wish to avoid extensive “symlink farms” as we’re not a profile-oriented distribution. Instead we focus on deduplication, atomic updates and resolution performance.

Subscriptions

Section titled “Subscriptions”

Keeping a system slim is often a very difficult thing to achieve, without extensive package splitting and effort on the user’s part. An example might be enabling SELinux support on a system, or culling the locales to only include the used ones.

In Serpent OS (and moss, more specifically) we intend to address this through “subscriptions”. Well defined names will be used by moss to filter (or enable) certain configurations in packages + dependencies. In some instances this will toggle subpackages/dependencies and in others it will control which paths of a package are actually installed to disk.

Going further with this concept, we will eventually introduce modalias based capabilities to automatically subscribe users to required kernel modules, allowing slim or fullfat installations as the user chooses. This in turn takes the burden of maintenance away from developers + users, and enables an incredibly flexible, approachable system.

Mandatory reboots

Section titled “Mandatory reboots”

Where possible we will limit mandatory reboots, preferring an in-place atomic update to facilitate high uptime. However, there are situations where a reboot is absolutely unavoidable, and the system administrator should plan some downtime to handle this case.

Certain situations like a kernel update, or security fix to a core library, would require a reboot. In these instances, the atomic update will be deferred until the next boot. In most situations, however, reboots will not be mandatory.

Until the next time

Section titled “Until the next time”

Well, we’ve given a brief introduction to our aims with moss and associated tooling, and you can get more information by checking the moss README.md.

The takeaway is we want a package-based software update mechanism that is reliable and trusted, and custom-built to handle Serpent OS intricities, with a simple approach to building and maintaining the distribution.

For now, we’re gonna stop talking, and start coding.