Blog

Stage3 Complete

Sep 7, 2020

Ikey

Founder

Another week, another milestone completed. We’re pleased to announce that we’ve completed the initial Stage3 bootstrap. While we have some parallel works to complete for Stage 4, we can now begin to work on the exciting pieces! We’re now bootstrapped on ARMv8(-a) and x86_64

Build

Announcing Moss

Our immediate focus is now to implement our package manager: moss. Currently it only has a trivial CLI and does absolutely nothing. We will now shift our attention to implement this as a core part of the Stage 4 bootstrap. Moss is so-called as we’re a rolling release, a rolling stone gathers no moss. The package manager is being implemented in the D Programming Language.

Moss does not aim to be a next generation package management solution, it instead inspired by eopkg, RPM and swupd, aiming to provide a reliable modern package management and update solution with stateless design at the core. Core features and automatic dependencies will be managed through capability subscriptions. The binary format will be versioned and deduplicated, with multiple internal lookup tables and checksums. Our chief focus with moss is a reliable system management tool that is accessible even in situations where internet access is limited.

Ongoing Works

In order to complete stage3, we provided linker stubs in libwildebeest to ensure we can build. Obviously this introduces runtime errors in systemd and our stage3 isn’t bootable, just chrootable. This will be resolved when systemd is properly packaged by moss in stage4.

Docker Image

We now have a test container available on Docker Hub. You can install and run the simple bash environment by executing the following command with a Docker-enabled user:

    docker run -it --rm serpentos/staging:latest

Currently we only have an x86_64 image, but may experiment with multiarch builds later in stage4.

IMPORTANT: The staging image is currently only a dump of the stage3 tree with minor cleaning + tweakups. It is not, in any way, shape or form, representative of the final quality of Serpent OS. Additionally zero performance work or security patching has been done, so do not use in a production environment.

The image is provided currently as a means to validate the LLVM/musl toolchain.

Future ISO

We cannot currently say when we’ll definitely have an ISO, however we do know that some VM-specific images will arrive first. After that we’ll focus on an installer (package selection based) and a default developer experience. All we can say is strap in, and enjoy the ride.

Looking Stage4 In The Eye

Aug 28, 2020

Ikey

Founder

Well, we’ve made an awful lot of progress in these last few days. It wasn’t that long ago that we introduced some of the new projects required to get stage3 off the ground.

Building systemd the easy way

libc-support

Our libc-support project has been growing, thanks primarily to the contributions of Jouni Roivas. We now have initial working versions of getent and getconf. The getconf program is considered feature-complete for our current requirements, and the focus is now on cleaning up getent making it more modular and easy to maintain in the long run.

libwildebeest

We began work on libwildebeest to quickly unlock building systemd. Remember, our ambition with this project is to provide a sane, centralised way of maintaining source compatibility with various projects that rely on features currently only available in the GNU toolchain + runtime. This is our alternative vision to patching every single package that fails to build against our LLVM+musl toolchain, ensuring our work scales.

Right now, libwildebeest implements some missing APIs, and indeed, some replacement APIs, for when GNU behaviours are expected. It does so in a way that doesn’t impact the resulting binary’s license, or any of the system ABI. We provide some pkg-config files with explicit cflags and libs fields set, such as:

      -L${libdir} -lwildebeest-ftw -Wl,--wrap=ftw -Wl,--wrap=nftw -I${includedir}/libwildebeest --include=lwb_ftw.h

Our headers take special care to mask the headers provided by musl to avoid redefinitions, and instruct the linker to replace calls to these functions with our own versions, i.e.:

int __wrap_ftw(const char *dir, int (*funcc)(const char *, const struct stat *, int),
               int descriptors)

It then becomes trivial to enable wildebeest for a package build, i.e.:

        export CFLAGS="${CFLAGS} $(pkg-config --cflags --libs libwildebeest)"

Right now - we’ve only provided stubs in libwildebeest to ensure we can build our packages. Our next focus is to actually implement those stubs using MIT licensed code so that applications and libraries can rely on libwildebeest to provide a basic level of GNU compatibility in a reliable fashion.

Until such point as all the APIs are fully and safely implemented, it would be highly ill-advised to use libwildebeest in any project. We’ll announce stability in the coming weeks.

systemd

We’ve made great progress in enabling systemd in Serpent OS. Where libwildebeest is in place, it now enables our currently required level of source compatibility to a point where systemd is building with networkd, resolved and a number of other significant targets enabled.

In a small number of cases, we’ve had to patch systemd, but not in the traditional sense expected to make it work with musl.

The only non-upstreamable patching we’ve done (in combination with libwildebeest enabling) was to the UAPI headers, as the musl provided headers clash with the upstream kernel headers in certain places (if_ether.h, if_arp.h) - but this is a tiny cost to bear.

The other patches, were simply portability fixes, ensuring all headers were included:

It should be noted both of these (very trivial) pull requests were accepted and merged upstream, and will be part of the next systemd release.

Next On The Agenda

Our major ticket items involve fleshing out stage3 with some missing libraries to further enable systemd, rebuilds of util-linux to be systemd-aware, and continue fleshing out dbus, systemd and libwildebeest support to the point we have a bootable disk image.

At that point we’ll move into stage4 with package management, boot management, and a whole host of other goodies. And, if there is enough interest, perhaps some early access ISOs!

After having engaged in discussions with a variety of developers using musl as their primary libc, we’ve catalogued common pain points. We therefore encourage developers to contribute to our libwildebeest and libc-support projects to complete the tooling and experience around musl-based distributions.

Our aim for Serpent OS is a full fat experience, which means we have large ticket items on our horizon, such as NSS/IDN integration, performance improvements, increasing the default stack size, along with source compatibility for major upstream projects.

Until the next blog post, you can keep up to date on our IRC channel. Join #serpentOS on freenode!

Stage 3 Progress

Aug 20, 2020

Ikey

Founder

Well, it’s been a few days since we last spoke, so now it’s time for a quick roundup. Long story short, we’re approaching the end of the stage3 bootstrap.

Fully functional chroot

In an effort to simplify our bootstrap process, we dropped the newly-introduced stage2.5 and came up with a new strategy for stage3. In order to make it all work nicely, we bind-mount the stage2 resulting runtime at /serpent within the stage3 chroot environment, executing the /serpent/usr/bin/bash shell.

In order to make this work, we build an intermediate musl package for libc.so in the very start of stage3, with all subsequent builds being performed in chroots. Part of the build is done on the host, i.e. extraction and patching, minimising the tool requirements for the chroot environment. The configuration, build and install is performed from within the initially empty chroot environment, replacing all the /serpent/usr/bin tools and /serpent/usr/lib libraries.

Mild Blockers

As we move further through stage3, towards a fully usable chroot environment, we’ve encountered a small number of blockers. Now, we could solve them by using existing patchwork and workarounds, but most have not and will not be accepted upstream. Additionally it is incredibly hard to track the origin and history of most of these, making security rather more painful.

libc-support

We’re going to start working on a project to flesh out the musl runtime with some missing utilities, written with a clean-room approach. These will initially include the getconf and getent tools, which will be written only with Linux in mind, and no legacy/BSD support.

These will be maintained over at our GitHub

libwildebeest

As a project we strive for correctness in the most pragmatic way. Some software, such as systemd, is heavily reliant on GNU GCC/GLibc extensions. In some software there are feasible alternatives when using musl, however in a good number of cases, functionality required to build certain software is missing and has no alternative.

Over time we’ll try to work with upstreams to resolve those issues, but we’re working on an interim solution called ‘libwildebeest’. This will provide source compatibility for a limited number of software packages relying on so-called ‘GNUisms’. Binary compatibility is not an aim whatsoever, and will not be implemented. This convenience library will centralise all patchwork on packages that need more work to integrate with musl, until such time as upstreams have resolved the remaining issues.

Additionally it will help us track those packages needing attention in the distribution, as they will have a build-time dependency on libwildebeest. We do not intend to use this project extensively.

This will be maintained over at our GitHub

A Word On Init Systems

Recently we’ve had many queries regarding the init system, as there is an expectation that due to our use of musl/llvm we also dislike systemd or wish to be a small OS, etc. There is a place in the world for those projects already, and we wish them much success. However from our own stance and goals, systemd has already “won the battle” and actually fits in with our design.

If it is possible in future with package manager considerations and packaging design, then we may make it possible to swap systemd for a similar set of packages. However, we only intend at this time to support systemd/udev/dbus directly in Serpent OS and leave alternatives to the community.

Other News

Just a quick heads up, we’ve been talking to the cool folks over at fosshost.org and they’ve agreed to kindly provide us with additional hosting and mirroring. This will allow us to build scale in from the very start, ensuring updates and images are always available. Once the new server is up and running we’ll push another blogpost with the details and links.

While initially we intended to avoid public bug trackers, the rate of growth within the project and community have made it highly apparent that proper communication channels need establishing. Therefore we will be setting up a public Phabricator instance for reporting issues, security flaws, and contributing packaging.

Much of our website is in much need of update, but our current priority is with building the OS. Please be patient with us, we’ll have it all sorted out in no time.

Where We At?

Well, stage3 completes fully, builds the final compiler, which has also been verified. A usable chroot system is produced, built using musl, libc++, libunwind, clang, etc. Some might say that stage3 is complete, however we wish to avoid circular dependency situations. We’ll mark stage3 as complete once we’ve integrated an initial slimmed down build of systemd and appropriately relinked the build.

As soon as this stage is done, we’ll proceed with stage4. This is the final stage where we’ll add package management and define the OS itself, with global flags, policies, etc.

With the speed we’re moving at, that really isn’t too far away.

And finally..

I personally wish to thank the Serpent OS team as a whole for the commitment and work undertaken of late. Additionally I want to thank the growing community around Serpent OS, primarily residing in our IRC channel (#serpentOS on freenode) and our Twitter account. Your input has been amazing, and it’s so refreshing to have so many people on the same page. Stay awesome.

Stage2 Complete

Aug 16, 2020

Ikey

Founder

Just in case you thought we were sleeping behind the wheel, we’ve got another blogpost for your viewing pleasure. In a nutshell, we completed stage2 bootstrap.

Complete build-target for ARMv8

In order to simplify life, we greatly reduced the size of the stage2 build component. This decision was taken to better support cross-compilation in the face of software that is distinctly cross-compilation unfriendly.

A support stage, stage2.5 will be added which will chroot into a copy of stage2, and natively compile a small handful of packages required to complete stage3, also within the chroot environment.

For cross-compilation, we’ll be relying on qemu-static to complete 2.5 and 3. However, at this point in time, we have the following:

Working cross-compilation of the entire bootstrap
Complete LLVM based toolchain: clang, llvm, libc++, lib++abi, libunwind
Entirety of stage2 built with musl libc.
Working, minimal, chroot environment as product of stage2, with working compiler (C & C++)

x86_64

This is a major milestone for the project, as it is an early indication that we’re self hosting.

Multiple Architecture Support

At this point in time, we now have build support for two targets: x86_64 and ARMV8a. Our intent is to support haswell and above, or zen and above, for the x86_64 target.

With our ARMv8 target, we’re currently looking to support the Pine Book Pro, if we can manage to get hold of some testing hardware. It will likely be some time after full x86_64 support that we’d officially support more hardware, however it is very important that our bootstrap-scripts can trivially target multiple platforms.

ARMv8

An interesting change when cross-compiling for other architectures, is the chicken & egg situation with compiler-rt and other LLVM libraries. When we detect cross-compilation, we’ll automatically bootstrap compiler-rt before building musl, and then cross-compile libc++, libc++abi and libunwind to ensure stage1 can produce native binaries for the target with correct linkage.

Next Steps

As we’ve mentioned, we’ll push ahead with 2.5 and 3, which will complete the initial Serpent OS bootstrap, producing a self-hosting, self-reliant rootfs. This is the point at which we can begin to bolt-on package management, boot management, stateless configuration utilities, etc.

Our initial focus is x86_64 hardware with UEFI, and as we gain access to more hardware we can enable support for more targets, such as ARMv8a. Our bootstrap-scripts will always remain open source, as will all processes and tooling within Serpent OS, or anything used to build and deploy Serpent OS.

This will make it much easier in future to create custom spins of Serpent OS for different configurations or targets, without derailing the core project. It should therefore be the simplest thing in the world to fork Serpent OS to one’s liking or needs.

If you want to support our work, you can jump onto our IRC channel (#serpentOS on freenode) or support us via the Team page.

Stage1 Complete

Aug 8, 2020

Ikey

Founder

Short and sweet, stage1 of the bootstrap is complete. As I indicated on the Lispy Snake blog, I’m still in the process of settling into new accommodation. This is going well, but still awaiting proper broadband connectivity. Work has begun, however, and we’re now moving onto stage2 of the bootstrap.

This is handled via our bootstrap-scripts project and can be run by anyone on a relatively modern Linux distribution.

Validating stage1 cross-compiler

Next on the list is completing stage2, which we’ve already started on. This is simply a cross-compiled chroot environment with all the basic bits in place to build stage3, sanitizing and cleansing the toolchain.

Even though it is early days, we can already, automatically produce a working cross-compiler that targets LLVM’s libc++, the musl libc.so, and x86_64-serpent-linux-musl host triplet.

Probably a dull update for most, but an update it is.