Blog

Packaging Automation, Next Steps

Jun 22, 2022 - 4 min read - 745 words

Hot damn we’ve been busy lately. No, really. The latest development cycle saw us focus exclusively on boulder, our build tooling. As of today it features a proof of concept boulder new subcommand for the automatic generation of packaging templates from an upstream source (i.e. tarball).

Before we really start this blog post off, I’d like to thank everyone who is supporting the project! All of the OpenCollective¹ contributions will make it easier for me to work full time on Serpent OS 😀 Much love ❤️

Look at all the buildiness

But, but build flavours …

Alright you got me there, certain projects prefer to abstract the configuration, build and installation of packages and be provided with some kind of hint to the build system, i.e. manually setting autotools, etc.

Serpent OS packaging is declarative and well structured, and relies on the use of RPM-style “macros” for distribution integration and common tasks to ensure consistent packaging.

We prefer a self documenting approach that can be machine validated rather than depending on introspection at the time of build. Our stone.yml format is very flexible and powerful, with automatic runtime dependencies and package splitting as standard.

..Doesn’t mean we can’t make packaging even easier though.

A Word From The Founder

Jun 6, 2022 - 2 min read - 243 words

Ikey

Founder: Retired

Well well, it’s been a long time since I personally wrote a post.. 😀 So let’s keep this short and sweet, shall we? I’m returning to full time work on Serpent OS.

The 6^th of July will be my last day at my current employment having tendered my 30 day notice today. Despite having enjoyment at my current position, the reality is that my passion and focus is Serpent OS.

I’m now in a transition process and will ramp up my efforts with Serpent OS. Realistically I need to reduce the outgoing costs of the project and with your help I can gain some level of financial support as we move through the next stages of development. Worst case, I will only take on any part-time or contractual gigs, allowing my primary focus to be Serpent OS.

I’ll begin accelerating works and enabling community contribution so we can get the derailed-alpha train back on the tracks.

I have absolute faith in this project, the community and our shared ability to deliver the OS and tooling. To achieve it will require far more of my time and I’m perfectly willing to give it.

Thank you all to everyone who has been supporting the project, it is now time to deliver. Not just another run of the mill distribution but a technically competent and usable distribution that is not only different but better.

Let’s do this in the most grassroots and enjoyable way possible 😊

RELR Brings Smaller Files, More Performance?

Apr 5, 2022 - 7 min read - 1268 words

SunnyFlunk

Distro Engineer: Retired

RELR is an efficient method of storing relative relocations (but is not yet available in glibc upstream). This has a significant reduction on file size often in the vicinity of 5% for libraries and even higher for PIE binaries. We also take a look at the performance impact on enabling RELR and that looks really good too! Smaller files with more performance - you can have your cake and eat it too!

Delicious Size Savings

Everyone enjoys smaller files, especially when it’s for free! RELR provides a very efficient method of storing relative relocations where it only requires a few percent compared to storing them in the .rela.dyn section which is what currently happens. However, it can’t store everything so the .rela.dyn section remains (though much smaller).

Here’s an example of the sections of libLLVM-13.so with and without RELR. Here we see the .rela.dyn section taking up a whole 2MB! When enabling RELR, .rela.dyn shrinks down to just under 100KB while adding a new section .relr.dyn which is just over 20KB! That’s nearly a 1.9MB file size reduction, so you’ll get smaller packages, smaller updates and it will be even faster to create delta packages from our servers. For reference, some of the biggest files have a .rela.dyn section over 10MB!

Section	Without RELR	With RELR
.dynstr	2,285,738	2,285,723
.rela.dyn	2,006,472	97,464
.relr.dyn	-	21,688
.text	28,708,290	28,708,386
.dynamic	592	608

Total	44,853,987	42,966,764

Making Deltas Great Again! (Part 1)

Feb 11, 2022 - 13 min read - 2706 words

SunnyFlunk

Distro Engineer: Retired

In Optimising Package Distribution we discussed some early findings for implementing binary deltas in Serpent OS. While discussing the implementation we have found the requirements to be suboptimal for what we were after. We provide a fresh look at the issue and what we can do to make it useful in almost all situations without the drawbacks.

Deltas Have a Bad Reputation

I remember back in the early 2000s on Gentoo when someone set up a server to produce delta patches from source tarballs to distribute to users with small data caps such as myself. When requesting the delta, the job would be added to the queue (which occasionally could be minutes) and then created a small patch to download. This was so important at the time to reduce the download size that the extra time was well worth it!

Today things are quite different. The case for deltas has reduced for users as internet speeds have increased. Shrinking 20MB off a package size may be a second reduction for some, but 10 seconds for others. The largest issue is that deltas have typically pushed extra computational effort onto the users computer in compensation for the smaller size. With a fast internet connection that cost is a real burden where deltas take longer to install than simply downloading the full package.

Can Hardly Contain Myself, Plus a Bonus

Jan 20, 2022 - 4 min read - 619 words

SunnyFlunk

Distro Engineer: Retired

One of the core steps for building a package is setting up a minimal environment with only the required (and stated) dependencies. Currently we have been building our stones in an systemd-nspawn container, where the root contains every package that’s been built so far. This makes the environment extremely difficult to reproduce!

Today we announce moss-container, a simple but flexible container creator that we can integrate for proper containerized builds.

Versatile For Many Use Cases

Containers have a multitude of uses for a Linux distro, but our immediate use case is for reproducible container builds for boulder. However, we have plans to use moss-container for testing, validation and benchmarking purposes as well. Therefore it’s important to consider all workloads, so features like fakeroot and networking can be toggled on or off depending on what features are needed.

moss-container takes care of everything, the device nodes in the /dev tree, mounting directories as tmpfs so the environment is left in a clean state, and mounting the /sys and /proc special file-systems. These are essential for a fully functioning container where programs like python and even clang won’t work without them. And best of all, it’s very fast so fits in well with the rest of our tooling!

The next step is integrating moss-container into boulder, so that builds become reproducible across machines, and makes it much easier for users to run builds on their host machines.