After many months and much work, our infrastructure is finally online.
We’ve had a few restarts, but it’s now running fully online with 2 builders, ready to serve builds
around the clock.
Firstly, I’d like to apologise for the delay since our last blog post. We made the decision to move
this website to static content, which took longer than expected. We’re still using our own D codebase,
but prebuilding the site (and fake “API”) so we can lower the load on our web server.
This is the page you can see over at dash.serpentos.com. It contains
the build scheduler. It monitors our git repositories, and as soon as it discovers any missing builds
it creates build tasks for them. It uses a graph to ensure parallel builds happen as much as possible,
and correctly orders (and blocks) builds based on build dependencies.
We have 2 instances of Avalanche, our builder tool, running. This accepts configuration + build requests
from Summit, reporting status and available builds.
Super early days with the infra but we now have builds flowing as part of our continuous delivery solution.
Keeping this blog post short and sweet… we’re about to package GNOME and start racing towards our first
real ISOs!
Enough of this “2 years” nonsense. We’re finally ready for lift off. It is with immense pleasure we can
finally announce that Serpent OS has transitioned from a promise to a deliverable. Bye bye, phantomware!
As mentioned, we spent 2 years working on tooling and process. That’s .. well. Kinda dull, honestly. You’re
not here for the tooling, you’re here for the OS. To that end I made a decision to accelerate development of
the actual Linux distro - and shift development of tooling into a parallel effort.
I deferred final enabling of the infrastructure until January to rectify the chicken/egg scenario whilst allowing
us to grow a base of contributors and an actual distro to work with. We’re in a good position with minimal blockers
so no concern there.
This is our term for the classical “package repository”. We’re using a temporary collection right now to store all
of the builds we produce. In keeping with the Avalanche requirements, this is the volatile software collection. Changes
a lot, hasn’t got a release policy.
It goes without saying, really, that our project isn’t remotely possible without a community. I want to take the time
to personally thank everyone that stepped up to the plate lately and contributed to Serpent OS. Without the work of the
team, in which I include the contributors to our venom recipe repository, an ISO was never possible. Additionally contributions
to tooling has helped us make significant strides.
It should be noted we’ve practically folded our old “team” concept and ensured we operate across the board as a singular community,
with some members having additional responsibilities. Our belief is all in the community have equal share and say. With that said,
to the original “team”, members both past and present, I thank for their (long) support and contributions to the project.
We actually went ahead and created our first ISO. OK that’s a lie, this is probably the 20th revision by now. And let’s be brutally
honest here:
It sucks.
We expected no less. However, the time is definitely here for us to begin our public iteration, transitioning from suckness to a project
worth using. In order to do that, we need to get ourselves to a point whereby we can dogfood our work and build a daily driver. Our focus
right now is building out the core technology and packaging to achieve those aims.
So if you want to try our uninstallable, buggy ISO, chiefly created as a brief introduction to our package manager and toolchain, head to our
newly minted Download page. Set your expectations low, ignore your dreams, and you will not be disappointed!
All jokes aside, it took a long time to get to point where we could even construct our first, KVM-focused, UEFI-only snekvalidator.iso. We now
have a baseline to improve on, a working contribution process, and a booting, self-hosting system.
The ISO is built using 2 layered collections, the protosnek collection containing our toolchain, and the new volatile collection. Much of the
packaging work has been submitted by venom contributors and the core team. Note you can install neofetch which our very own Rune Morling (ermo)
patched to support the Serpent OS logo.
Boot it in Qemu (or certain Intel laptops) and play with moss now! Note, this ISO is not installable, and no upgrade path exists. It is simply
the beginnings of a public iteration process.
In January we’ll launch our infrastructure to scale out contributions as well as to permit the mass-rebuilds that need to happen. We have to
enable our -dbginfo packages and stripping, which were disabled due to a parallelism issue. We need to introduce our boot management based around
systemd-boot, provide more kernels, do hardware enabling, introduce moss-triggers, and much more. However, this is a pivotal moment for our
project as we’ve finally become a real, if not sucky, distro. The future is incredibly bright, and we intend to deliver on every one of our
promises.
As always, if you want to support our development, please consider sponsoring the work, or engaging with the community on Matrix or indeed
our forums.
You can discuss this blog post, or leave feedback on the ISO, over at our forums.
After much deliberation - we’ve decided to pull out of Open Collective. Among other reasons, the fees
are simply too high and severely impact the funds available to us. In our early stages, the team consensus
is that funds generated are used to compensate my time working on Serpent OS.
As such I’m now moving funding to my own GitHub Sponsors page - please do migrate! It ensures your entire
donation makes it and keeps the lights on for longer =) Please remember I’m working full time on Serpent OS
exclusively - I need your help to keep working.
On the surface, moss looks and feels roughly the same as just about any other traditional package manager out there.
Internally, however, its far more modern and has a few tricks up its sleeve. For instance, every time you initiate
an operation in moss, be it installation, removal, upgrade, etc, a new filesystem transaction is generated. In short,
if something is wrong with the new transaction - you can just boot to an older transaction when things worked fine.
Now, it’s not implemented using bundles or filesystem specific functionality, internally its just intelligent use of
hardlinks and deduplication policies, and we have our own container format with zstd based payload compression. Our
strongly typed, deduplicating binary format is what powers moss.
Behind the scenes we also use some other cool technology, such as LMDB for super reliable and speedy database access.
The net result is a next generation package management solution that offers all the benefits of traditional package
managers (i.e. granularity and composition) with new world features, like atomic updates, deduplication, and repository
snapshots.
It’s one thing to manage and install packages, it’s another entirely to build them. boulder builds conceptually
on prior art such as pisi and the package.yml format used in ypkg. It is designed with automation
and ease of integration in mind, i.e. less time spent focusing on packaging and more time on actually
getting the thing building and installing correctly.
Boulder supports “macros” as seen in the RPM and ypkg world, to support consistent builds and integration.
Additionally it automatically splits up packages into the appropriate subpackages, and automatically scans
for binary, pkgconfig, perl and other dependencies during source analysis and build time. The end result
is some .stone binary packages and a build manifest, which we use to flesh out our source package index.
We’ve spent considerable time reworking moss, our package manager. It now features
a fresher (terminal) user interface, progress bars, and is rewritten to use the
moss-db module encapsulating LMDB.
It’s also possible to manipulate the binary collections (software repositories)
used by moss now. Note we’re going to rename “remote” to “collection” for consistency.
Serpent OS is now officially self hosting. Using our own packages, we’re able to
construct a root filesystem, then within that rootfs container we can use our own
build tooling (boulder) to construct new builds of our packages in a nested
container.
The protosnek collection has been updated to include the newest versions of moss
and boulder.
As a fun experiment, we wanted to see how far along things are. Using a throwaway
kernel + initrd build, we were able to get Serpent OS booting using virtualisation (qemu)
Right now everyone is working in the snekpit organisation to
get our base packaging in order. I’m looking to freeze protosnek, our bootstrap collection,
at the latest of tomorrow evening.
We now support layered, priority based collections (repositories) and dependency solving across
collection boundaries, allowing us to build our new main collection with protosnek acting as
a bootstrap seed.
Throughout this week, I’ll focus on getting Avalanche, Summit and Vessel into shape for PoC so
that we can enjoy automated builds of packages landing in the yet-to-be-launched volatile collection.
From there, we’re going to iterate + improve packaging, fixing bugs and landing features as we
discover the issues. Initially we’ll look to integrate triggers in a stateless-friendly
fashion (our packages can only ship /usr by design) - after that will come boot management.
An early target will be Qemu support via a stripped linux-kvm package to accelerate the bring up,
and we encourage everyone to join in the testing. We’re self hosting, we know how to boot, and
now we’re able to bring the awesome.
I cannot stress how important the support to the project is. Without it - I’m unable to work full
time on the project. Please consider supporting my development work via GitHub Sponsors.
I’m so broke they’ve started naming black holes after my IBAN.
Since the last post, I’ve pivoted to full time work on Serpent OS, which is
made all the more possible thanks to everyone supporting us via OpenCollective <3.
We’ve been working towards establishing an online infrastructure to support the automation of package builds, while
revisiting some core components.
During the development of the Serpent OS tooling we’ve been exploring the possibilities of D Lang, picking up new
practices and refining our approach as we go. Naturally, some of our older modules are somewhat … smelly.
Most noticeable is our moss-db module, which was initially intended as a lightweight wrapper around RocksDB.
In practice that required an encapsulation API written in D around the C API, and our own wrapping on top of that. Naturally,
it resulted in a very allocation-heavy implementation that just didn’t sit right with us, and due to the absurd complexity
of RocksDB was still missing quite a few features.
We’re now using the Lightning Memory-Mapped Database as the driver implementation
for moss-db. In short, we get rapid reads, ACID transactions, bulk inserts, you name it. Our implementation takes
advantage of multiple database indexes (MDB_dbi) in LMDB to partition the database into internal components,
so that we can provide “buckets”, or collections. These internal DBs are used for bucket mapping to permit a
key-compaction strategy - iteration of top level buckets and key-value pairs within a bucket.
The majority of the API was designed with the boltdb API in mind. Additionally
it was built with -preview=dip1000 and -preview=in enabled, ensuring safely scoped memory use and no
room for memory lifetime issues. While we prefer the use of generics, the API is built with immutable(ubyte[])
as the internal key and value type.
Custom types can simply implement mossEncode or mossDecode to be instantly serialisable into the database
as keys, values or bucket identifiers.
Moss will be ported to the new DB API and we’ll gather some performance metrics,
while implementing features like expired state garbage collection (disk cleanup),
searching for names/descriptions, etc.
Avalanche is a core component of our upcoming infrastructure, providing the
service for running builds on a local node, and a controller to coordinate
a group of builders.
Summit will be the publicly accessible project dashboard, and will be responsible
for coordinating incoming builds to Avalanche controllers and repositories.
Developers will submit builds to Summit and have them dispatched correctly.
So far we have the core service process in place for the Controller + Node,
and now we’re working on persistence and handshake. TLDR; fancy use of
moss-db and JSON Web tokens over mandated SSL. This means our build infra
will be scalable from day 1 allowing multiple builders to be online very
early on.
Hot damn we’ve been busy lately. No, really.
The latest development cycle saw us focus exclusively on boulder, our build tooling. As of
today it features a proof of concept boulder new subcommand for the automatic generation of
packaging templates from an upstream source (i.e. tarball).
Before we really start this blog post off, I’d like to thank everyone who is supporting the
project! All of the OpenCollective contributions will make it easier for me to work full
time on Serpent OS =) Much love <3
Alright you got me there, certain projects prefer to abstract the configuration, build and
installation of packages and be provided with some kind of hint to the build system, i.e.
manually setting autotools, etc.
Serpent OS packaging is declarative and well structured, and relies on the use of RPM-style
“macros” for distribution integration and common tasks to ensure consistent packaging.
We prefer a self documenting approach that can be machine validated rather than depending
on introspection at the time of build. Our stone.yml format is very flexible and powerful,
with automatic runtime dependencies and package splitting as standard.
..Doesn’t mean we can’t make packaging even easier though.
Pointing boulder at an upstream source will perform a deep analysis of the sources to determine
the build system type, build dependencies, metadata, licensing etc. Right now it’s just getting
ready to leave POC stage so it has a few warts, however it does have support for generating
package skeletons for the following build systems:
cmake
meson
autotools
We’re adding automation for Perl and Python packaging (and Golang, Rust, etc) so we can enforce consistency,
integration and ease without being a burden on developers. This will greatly reduce the friction
of contribution - allowing anyone to package for Serpent OS.
We’re also able to automatically discover build time dependencies during analysis and add those
to the skeleton stone.yml file. We’ll enhance support for other build systems as we go, ensuring
that each new package is as close to done on creation as possible, with review and iteration left
to the developer.
A common pain in the arse when packaging for any Linux distribution is ensuring the package
information is compliant in terms of licensing. As such we must know all of the licensing
information, as well as FSF and OSI compliance for our continuous integration testing.
…Finding all of that information is truly a painful process when conducted manually.
Thankfully boulder can perform analysis of all licensing files within the project to
greatly improve compliance and packaging.
Every license listed in a stone.yml file must use a valid SPDX identifier,
and be accurate. boulder now scans all license files, looking for matches with both SPDX IDs
as well as fuzzy-matching the text of input licenses to make a best-guess at the license ID.
This has so far been highly accurate and correctly identifies many hundreds of licenses,
ensuring a compliant packaging process with less pain for the developers. Over time we’ll
optimise and improve this process to ensure compliance for our developers rather than
blocking them.
As of today we support the REUSE specification for expressing software licenses too!
Work formally begins now on Bootstrap Bill (Turner). Whilst we did successfully bootstrap Serpent OS
and construct the Protosnek repository, the process for that is not reproducible as boulder
has gone through massive changes in this time.
The new project will leverage boulder and a newly
designed bootstrap process to eliminate all host contamination and bootstrap Serpent OS from
stone.yml files, emitting an immutable bootstrap repository.
Layering support will land in moss and boulder to begin the infrastructure projects.
The aim is to complete bill in a very short time so we can bring some initial infrastructure
online to facilitate the automatic build of submitted build jobs. We’ll use this process
to create our live repository, replacing the initial bootstrap repository from bill.
At this point all of the tooling we have will come together to allow us all to very
quickly iterate on packaging, polish up moss and race towards installed systems with
online updates.
