Skip to content
aerynOS
Blog

Blog

Unlocking Moss

- 3 min read - 486 words
Ikey
Ikey
Founder: Retired

Wait, what? Another blog post? In the same WEEK? Yeah totally doing that now. So, this is just another devlog update but there have been some interesting updates that we’d like to share.

LDC present in the bootstrap

Section titled “LDC present in the bootstrap”

Thanks to some awesome work from Peter, we now have LDC (The LLVM D Lang Compiler) present in the stage3 bootstrap. To simplify the process we use the official binary release of LDC to bootstrap LDC for Serpent OS.

In turn, this has allowed us to get to a point where we can now build moss and boulder within stage3. This is super important, as we’ll use the stage3 chroot and boulder to produce the binary packages that create stage4.

Some patching has taken place to prevent using ld.gold and instead use lld to integrate with our toolchain decisions.

A wild LDC appears

Reworking the Payload implementation

Section titled “Reworking the Payload implementation”

Originally our prototype moss format only contained a ContentPayload for files, and a MetaPayload for metadata, package information, etc. As a result, we opted for simple structs, basic case handling and an iterable Reader implementation.

As the format expanded, we bolted deduplication in as a core feature. To achieve this, we converted the ContentPayload into a “megablob”, i.e. every unique file in the package, one after the other, all compressed in one operation. We then store the offsets and IDs of these files within an IndexPayload to allow splitting the “megablob” into separate, unique assets. Consequently, we added a LayoutPayload which describes the final file system layout of the package, referencing the unique ID (hash) of the asset to install.

So, while the format grew into something we liked, the code supporting it became very limiting. After many internal debates and discussions, we’re going to approach the format from a different angle on the code front.

It will no longer be necessary/possible to iterate payloads in sequence within a moss archive, instead we’ll preload the data (unparsed) and stick it aside when reading the file, winding it to the ContentPayload segment if found. After initial loading of the file is complete, the Reader API will support retrieval (and lazy unpacking) of a data segment. In turn this will allow code to “grab” the content, index and layout payloads and use advanced APIs to cache assets, and apply them to disk in a single blit operation.

In short, we’ve unlocked the path to installing moss packages while preserving the advanced features of the format. The same new APIs will permit introspection of the archives metadata, and of course, storing these records in a stateful system database.

Thank you for the unnecessary update

Section titled “Thank you for the unnecessary update”

Oh you’re quite welcome 😛 Hopefully now you can see our plan, and that we’re on track to meet our not-28th target. Sure, some code needs throwing away, but all codebases are evolutionary. Our major hurdle has been solved (mentally) - now it’s just time to implement it and let the good times roll.

Cleanups Complete

- 2 min read - 308 words
Ikey
Ikey
Founder: Retired

Well, we’re officially back to working around the clock. After spending some time optimising my workflow, I’ve been busy getting the entire codebase cleaned up, so we can begin working towards an MVP.

Lots and lots of work

Codebase hygiene

Section titled “Codebase hygiene”

Since Friday, I’ve been working extensively on cleaning up the codebase for the following projects:

  • boulder
  • moss
  • moss-core
  • moss-format

As it now stands, 1 lint issue (a non-issue, really) exists across all 4 projects. A plethora of issues have been resolved, ranging from endian correctness in the format to correct idiomatic D-lang integration and module naming.

Bored

Section titled “Bored”

Granted, cleanups aren’t all that sexy. Peter has been updating many parts of the bootstrap project, introducing systemd 247.3, LLVM 11.0.1, etc. We now have all 3 stages building correctly again for the x86_64 target.

And then….

Section titled “And then….”

Yikes, tough audience! So we’ve formed a new working TODO which will make its way online as a public document at some point. The first stage, cleanups, is done. Next is to restore feature development, working specifically on the extraction and install routines. From there, much work and cadence will be unlocked, allowing us to work towards an MVP.

TODO

You keep saying MVP..

Section titled “You keep saying MVP..”

I know, it makes me feel all … cute and professional. At the moment we’re cooking up a high level description of how an MVP demonstration could be deployed. While most of the features are ambiguous, our current ambition is to deploy a preinstalled QCOW2 Serpent OS image as a prealpha to help validate moss/boulder/etc.

It’ll be ugly. Likely slow. Probably insecure as all hell. But it’ll be something. It’ll allow super basic interaction with moss, and a small collection of utilities that can be used in a terminal environment. No display whatsoever. That can come in a future iteration :)

ETA? Definitely not by the 28th of February. When it’s done.

Dr-Evil making air quotes

Build System Functional

- 3 min read - 462 words
Ikey
Ikey
Founder: Retired

Wow, has it been a hectic few weeks, and it definitely shows: last time we blogged it was about re-bootstrapping with glibc. Feels like ancient news already! So, what’s new in the world of Serpent OS? Apart from yours truly now being proud parent to a beautiful baby girl, work has resumed on the development of Moss, our package manager. And it builds stuff. Awesomely. Let’s quickly catch up with those updates and see where we’re headed next.

Look at all the buildiness

It lives!

Section titled “It lives!”

We’re now able to build the majority of a moss package. Notice I’ve made a distinction there. So, we’re able to run the build instructions, and use all the metadata, configuration and macros available. The only thing we’re not actually doing is dumping the built files into a binary package.

We’re able to do the build-system part rather well, now. Right now we support the following features in the build system:

  • Multiple, profile-based architecture support
  • Automatic -m32 profile based cross compilation support for 32-bit libraries on 64-bit OS
  • Basic Profile Guided Optimisation via the workload key. Set a workload, optimise accordingly.
  • LLVM Context Sensitive Profile Guided Optimisation. This is default for LLVM with the workload key, and comes for free, with multiple build stages.
  • Profile based tuning options, such as optimize for speed, size, disabling hardening, etc.
  • Trivially switch between gnu and llvm toolchain in stone.yml, with profiles knowing the right flags to use with tuning options.
  • Recursive macro support in build scripts, defined on per-profile level
  • Architecture specific profiles support in stone.yml, i.e. profiles -> aarch64 -> setup: to avoid if/else spaghetti.

Next on the agenda

Section titled “Next on the agenda”

Now that the huge amount of scaffolding work has been done, we can actually turn the results of builds into installable binary packages using our moss.format.binary module. We’ll add some magic sauce to have automatic subpackages + inter-package dependencies, along with the expected automatic runtime dependencies + such. Cue some linting, et voila, build tool that’s a pleasure to work with.

Once we have all those packages building, we’ll need a way to install them. Luckily some scaffolding is in place for this already, and it won’t take much effort to support moss install somepkg.stone. Then we throw a few dozen packages in the mix, add some dependency handling, repo support, Bob’s your uncle, and your aunt is downloading exclusive early-access images from our Open Collective1 as soon as they’re available. 😮

Footnotes

Section titled “Footnotes”

  1. Since this blog post, we no longer use OpenCollective. For funding, please see our Funding Page and for ISOs, please see our Download Page. ↩

Results Of The Experiment

- 4 min read - 768 words
Ikey
Ikey
Founder: Retired

It seems like only yesterday we announced to the world a Great Experiment. It was in fact 2 months ago, and a whole lot of work has happened since that point. A few take-homes are immediately clear, the primary one being the need to be a community-oriented Linux distribution.

To quote ourselves 2 months ago:

If the experiment is a success, which of course means having tight controls on scope and timescale, then one would assume the primary way to use Serpent OS would be through some downstream repackaging or reconfiguration, i.e. basing upon the project, to meet specific user demand.

It turns out that so far the experiment has been successful, and being forkable is still at the very heart of our goals. Others have joined us on our journey, and expressed the same passion in our goals as we have. A community has formed around the project, with individuals sharing the same ambitions for a reliable, rolling operating system with a powerful package manager.

Toolchain bootstrap

An open community

Section titled “An open community”

Over the past 2 months we’ve transformed from set of ideas into a transparent, community-first organisation with clear leadership and open goals. I’ve stepped into the Benevolent Dictator For Life position, and Peter has taken on daily responsibilities for the project running. Aydemir is now our treasurer on Open Collective1, and many individuals contribute to our project.

Transactional package management

Section titled “Transactional package management”

No need to rehash this, but the defining feature of Serpent OS has clearly become moss, something initially not anticipated when we started. A read-only root filesystem, transactional operations, rollbacks, deduplicating throughout and atomic updates. Combine that with a rolling release model, stateless policy and ease-of-use, the core feature-set is already powerful.

Deferral of musl integration

Section titled “Deferral of musl integration”

In recent weeks we’ve been working on libwildebeest and libc-support, primarily as a stop-gap to provide glibc compatibility without using glibc. While musl has many advantages, it is clear to us now that writing another libc through our support projects isn’t what we originally planned. With that in mind we’re adopting glibc and putting our musl works under community ownership, until such time as reevaluation shows that musl is what is needed in Serpent OS. Note the primary motivator here is investing our efforts where it makes sense, and obtaining the best results in the most manageable fashion for our users.

Our new toolchain configuration will be as follows:

  • LLVM/clang as primary toolchain
  • libc++ as C++ library
  • glibc as C library
  • gcc+binutils (ld.bfd) provided to build glibc, elfutils, etc.
  • Permitting per-package builds using GCC, i.e. kernel to alleviate Clang IAS issues.

Thus, our toolchain will in fact be a hybrid GNU/LLVM one. This will allow both source and binary compatibility with the majority of desktop + server Linux distributions, facilitating choice of function for our users.

It should be noted this decision has been made after much discussion internally, on our IRC1, on our OpenCollective2, etc. Our bootstrap-scripts is being improved to support both glibc and musl, so that the decision can continuously be reviewed. If we reach a position whereby musl inclusion once again makes sense, thanks to atomic updates from moss, it will be possible to switch.

Final thoughts

Section titled “Final thoughts”

Initially Serpent OS emerged as a collective agreement on IRC1 as a set of notions as opinions. Over the past few months those opinions have solidified into tangible ideas, and a sense of community. In keeping with what is right for the community, our messaging has been reworked.

It is fair to say our initial stance appeared quite hostile, as a bullet-point list of exhaustion with past experiences. As we’ve pivoted to being a fully community-oriented distribution, we’ve established our goals of being a reliable, flexible, open, rolling Linux distribution with powerful features imbued by the package manager, and an upstream-first approach.

As such we’ve agreed to not let our own pet-peeves interfere with the direction of the project, and instead enable users to do what they wish on Serpent OS, be it devops, engineering, browsing, gaming, you name it. We’re a general purpose OS with resilience at the core.

Our focus is on the usability and reliability of the OS - thus our efforts will be invested in areas such as the package manager, hardware enabling, the default experience, etc.

So, strap yourself in, as we’re fond of saying. Development of Serpent OS is about to accelerate rapidly.

Footnotes

Section titled “Footnotes”

  1. aerynOS now uses a Zulip server for community discussion. ↩ ↩2 ↩3

  2. Since this blog post, we no longer use OpenCollective. For funding, please see our Funding Page ↩

Source Format Defined

- 3 min read - 365 words
Ikey
Ikey
Founder: Retired

Following quickly on the heels of yesterday’s announcement that the binary format has been defined, we’ve now implemented the initial version of our source format. The source format provides metadata on a package along with instructions on how to build the package.

The next step of course, is to implement the build tool, converting the source specification into a binary package that the end user can install. With our 2 formats defined, we can now go ahead and implement the build routines.

Very trivial package recipe

A YAML based format

Section titled “A YAML based format”

The eagle-eyed among you will already see this is a derivation of the package.yml format I originally created while at Solus. Minor adaptations to the format have been made to support multiple architectures via the profiles key, and package splitting behaviour has now been grouped under a packages key to make the structure more readable.

In package.yml, one would have to redefine subpackage summaries as a key in a list of the primary summary key, such as:

    rundeps:
        - primary-run-dep
        - dev: secondary-run-dep
    summary:
        - Some Summary
        - dev: Some different summary

We’ve opted to group “Package Definition” behaviour into core structs, which are allowed to appear in the root-level package and subpackages:

    summary: Top-level summary
    packages:
        - dev:
            summary: Different summary
            rundeps: Different rundeps

Multiple architecture support

Section titled “Multiple architecture support”

In keeping with the grouping behaviour, we’re baking multiple architecture configurations into the YML file. A common issue encountered with the older format was how to handle emul32:

    setup: |
        if [[ -z "${EMUL32BUILD}" ]]; then
            %configure --some-emul32-option
        else
            %configure
        fi


    build: |
    %make

Our new approach is to group Build Definitions into the root level struct, which may then individually be overridden for each architecture. For example:

    profiles:
        - ia32:
            setup: |
        %configure --some-emul32-option
    setup: |
    %configure

Permutations

Differences

Section titled “Differences”

As you can see it is highly similar to package.yml - which is a great format. However, with our tooling and aims being slightly different, it was time to reevaluate the spec and bolster it where appropriate. We’re happy to share our changes, but in the interest of not causing a conflict between the 2 variants, we’ll be calling ours “stone.yml”.

Our main motivation came from the tooling, which is written in the D language. With D we were able to create a strongly typed parser and explicit schema, and with a struct-based approach it made it more trivial to group similar definitions.

Other than that, we have the same notions with the format, intelligent automatic package splitting, ease of developer experience, etc.