zerforschen+

Keeping scrapers out

Mon, 15 Sep 2025 20:00:00 +0200

After reading an LWN article on high AI scraper traffic, I also wanted to do something to “protect” the blog. I do not see any continuous load peaks, but I dislike the idea of giant corporations profiting off of my writing without any negotiation.

One solution that stands out currently is anubis. It tries to challenge requests in a way that is hard or resource intensive for scrapers at scale, but easy and resource-light for normal users.

I could not find a good summary on how to proxy an nginx virtual host through anubis, so here we go.

Starting configuration

The following examples are based on the configuration of this blog. Most things not relevant here are removed, but you can take a look at the original.

{ pkgs, ... }:
{
  services.nginx = {
    enable = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;
    recommendedGzipSettings = true;
    recommendedOptimisation = true;

    virtualHosts = {
      "zerforschen.plus" = {
        addSSL = true;
        enableACME = true;
        root = pkgs.zerforschen-plus-content;
      };
    };
  };
}

Hide real host

The first thing to do is hiding the real website. Otherwise, a scraper can just use whatever we rename it to.

I wanted to forward everything through unix domain socket. The kind I’ll use behave like a file, which means we do not have to make sure that e.g. the application only listens on localhost. It also makes permission management trivial, but we’ll see that later.

{ pkgs, ... }:
let
  # /run/nginx already exists
  blog-domain-socket = "/run/nginx/blog.sock";
in
{
  services.nginx = {
    enable = true;
    virtualHosts = {
      "zerforschen.plus" = {
        addSSL = true;
        enableACME = true;
        # we do not have anything hosted here anymore
      };
      # a new virtual host contains the site now
      "blog-in-anubis" = {
        root = pkgs.zerforschen-plus-content;
        # specifying any listen overrides the defaults
        listen = [
          {
            # this makes nginx create the unix domain socket
            addr = "unix:" + blog-domain-socket;
          }
        ];
      };
    };
  };
}

Create anubis instance

Next, we need an anubis service that forwards to the socket created by nginx.

{ pkgs, ... }:
let
  blog-domain-socket = "/run/nginx/blog.sock";
in
{
  services = {
    # previous nginx config

    anubis.instances.main = {
      enable = true;
      settings = {
        TARGET = "unix://" + blog-domain-socket;
      };
    };
  };
}

Forward public host to anubis

Now we have an anubis instance running, but no way to access it from outside.

I did not want to expose anubis to the public internet directly, so I configured another unix domain socket for this purpose.

{ pkgs, ... }:
let
  blog-domain-socket = "/run/nginx/blog.sock";
  anubis-domain-socket = "/run/anubis/anubis-blog.sock";
in
{
  services = {
    nginx = {
      enable = true;
      virtualHosts = {
        "zerforschen.plus" = {
          addSSL = true;
          enableACME = true;
          locations."/" = {
            # HTTP over unix domain socket
            proxyPass = "http://unix:" + anubis-domain-socket;
          };
        };
        # "blog-in-anubis" = { ... };
      };
    };

    anubis.instances.main = {
      enable = true;
      settings = {
        # this makes anubis create and listen to the socket
        BIND = anubis-domain-socket;
        TARGET = "unix://" + blog-domain-socket;
      };
    };
  };
}

Socket permissions

The config is nearly complete now. Applying the configuration now produces permission errors when trying to access the site, because the sockets are owned by the service user and grou with a 660 permission. In my case, I simply added the services to each others groups. If this was a server I was making my living with, I’d probably create a new group and folder and add both services to that one.

{ pkgs, ... }:
{
  systemd.services = {
    nginx.serviceConfig.SupplementaryGroups = [ "anubis" ];
    anubis-main.serviceConfig.SupplementaryGroups = [ "nginx" ];
  };
}

Final configuration

Overall, I think NixOS makes the configuration required easy to read and write. The NixOS options search is a great storage of knowledge for any software configurable through the NixOS configuration. If the default policy does not fit your needs, that would be another step required¹.

Update: Asking ChatGPT about this article indicates everything is working as intended:

My final configuration for nginx and anubis was as follows:

{ pkgs, ... }:
let
  blog-domain-socket = "/run/nginx/blog.sock";
  anubis-domain-socket = "/run/anubis/anubis-blog.sock";
in
{
  systemd.services = {
    nginx.serviceConfig.SupplementaryGroups = [ "anubis" ];
    anubis-main.serviceConfig.SupplementaryGroups = [ "nginx" ];
  };

  services = {
    nginx = {
      enable = true;

      recommendedProxySettings = true;
      recommendedTlsSettings = true;
      recommendedGzipSettings = true;
      recommendedOptimisation = true;

      virtualHosts = {
        "zerforschen.plus" = {
          addSSL = true;
          enableACME = true;
          locations."/" = {
            proxyPass = "http://unix:" + anubis-domain-socket;
          };
        };

        "blog-in-anubis" = {
          root = pkgs.zerforschen-plus-content;
          listen = [
            {
              addr = "unix:" + blog-domain-socket;
            }
          ];
        };
      };
    };

    anubis.instances.main = {
      enable = true;
      settings = {
        BIND = anubis-domain-socket;
        TARGET = "unix://" + blog-domain-socket;
      };
    };
  };
}

Part of the reason I am writing this article is to check if my RSS reader can handle it. ↩︎

WHY 2025 - Day 1

Sat, 16 Aug 2025 23:00:00 +0200

I went to WHY2025, a Dutch hacker camp.

This is the third part of my travel log, covering everything happening on day 1.

Disclaimer WHY2025

I am trying to write more, so I am experimenting with a series of travel-log style post. The resulting text is boring to read, the contained conclusions are obvious for anyone with two or more brain cells and you will just waste your time reading it.

Some posts contain photos, so I should probably tell you about the photo policy of events like this. People are allowed to take pictures, but only after asking every single person in the photo individually. In practice this means you cannot really take good pictures to capture the vibe of the place - you are either photographing an object or a mostly empty space.

As with all other posts, I may update previous ones without telling anyone. You may want to wait until I added all the photos and polished the text a bit.

Waking up

At day 1 I woke up in a warm tent to ABBAs “Gimme Gimme”, making further sleep impossible. The weather service now predicted hotter weather than originally planned for, so I had to ration my cooler clothing for the hottest days, which had yet to come.

I got a pancake in the food court and continued writing. After saturation settled in, the next level in the pyramid of needs was social connections, so I walked around a bit.

Angry Nerds

After a while I came across a village by the “Angry Nerds” podcast, which is sadly in Dutch so I was not familiar with it. One of the creators of it was there and we talked for a while about differences between the German and Dutch hacker events. WHY20205 feels less political than the events by CCC. This may seem weird given that there is a big support for Ukraine tent, but at Congress there is a bunch of talks and assemblies about civil rights and climate action, by e.g. Netzpolitik.org or Bits&Bäume.

He also pointed out the preparations of the firewall (marked by big flames on the map) and showed me a sneak peek. Without being told, I could not have guessed what those grey boxes do.

As we can time travel here, you can also see the firewall at night already:

After a while, I did not want to bother my captive audience anymore so I continued exploring.

Anderstorps

I walked past a bar at one of the villages playing very loud music to something you could describe as art. I had just found Anderstorps, right next to BornHack.

// FIXME: add picture of intact art piece

As it seems like I do not have a picture of the art piece intact, here is a short description. There was a pavillion with a white table in the middle. On the table there was a black iron cage containing an accordeon. In front of the table there was a huge “accordeons forbidden” sign.

When asked about it, the inhabitants told me to come back later as they were too hung over from the night of Day 0, when different music will be playing. As I wanted to sit down and write some more, I used the provided shade here. Not much later, I noticed they were literally playing the same music, as in looping a single track.

🎵 Faxe Kondi by Klumben & Raske Penge

On Day 3 or so, the cage including the instrument were gone, which may or may not have been planned beforehands. They did not provide any further context apart from “have you heard what they sound like?”.

First talk

Next up was a talk that made me realize more stuff that was different from CCCamp. It expected a talk about either cloud security in general, or one about finding vulnerabilities in a deployment. Instead, this talk was about securing proprietary Microsoft Azure services (a company still rightiously considered evil in many parts of the chaos) using other proprietary Microsoft Azure services. Whenever there was a security issue, the solution was to use more Azure services. There was barely enough context to figure out how to generalize what was said for other products, so it was not really helpfull for me. The following questions were also about specific products and services. I am pretty sure this would not have been on the Fahrplan in Germany, and as stated by others this would also have raised eyebrows in the Dutch hacker community one or two decades ago.

Power issues

I wanted to get a new mate, but had to walk to a different bar because of issues with the payment system again. As Heaven announced, the power outage was not on site and affected all of North Holland. This meant that everything that was attached to the regional power grid, which was meant to be more stable than the on-site generators, did not work. Because of this, all talks for the rest of the day were cancelled and mostly rescheduled in the following days. At least bar payments were possible again shortly thereafter.

When the sun started to settle, the soundcheck of a Metallica tribute band fittingly called Sad but Tribute started. From then on, the evening was filled with surprisingly close covers. The castle-shaped stage (“party area”) and the flamethrowers attached to the towers were visible from all over the camping grounds.

What the fog

In the nights, it got really foggy, and I mean really foggy. This made all the lights everywhere stand out even more.

Combine this with a bunch of very strong fog machines and you could maybe see two meters into the distance.

Post by @vinzenz@chaos.social

View on Mastodon

Someone described their shift as customers popping into existance in front of the bar, ordering, and returning into nothingness after receiving their drinks.

After turning of the fog machines at milliways, it cleared up quicky around us. It looked unreal even in person.

The rest of the night I played PolyGen, which will be described in a future blog post.

WHY 2025 - Day 0

Sat, 09 Aug 2025 19:00:00 +0200

I went to WHY2025, a Dutch hacker camp.

This is the second part of my travel log, covering everything happening on day 0 after arriving on the site.

Disclaimer WHY2025

As with all other posts, I may update previous ones without telling anyone. You may want to wait until I added all the photos and polished the text a bit.

Entrance

The shuttle dropped us off on a parking lot next to the event area. To my surprise, there was also a team you could pass your luggage to that were driving everything to the info tent. I wondered what would happen if there were issues with your ticket in that case. Also, I did not want to have to search where I could get my tent back in case time mattered to get a good spot, and I wanted my tent near the border between the cyan and red fields, which meant I did not have to walk as far.

The ticket checking was very fast without any queue. This year’s ribbon is dark blue with star patterns on it - pretty! I also liked the fact you only had to show your ID in case you wanted an additional “18+” ribbon to be able to buy alcohol and so on (and you did not need an ID later to get drinks). I felt kind of flattered by the fact I had to show mine, as I started getting self conscious about my age when someone guessed 10y too high a year ago.

People were handed a paper goodie bag which were supposed to also include stickers, but ended up mostly containing ads for the sponsors of the event. I usually really dislike ads, but in this case I liked the organisations beaing advertised for (e.g. NIST and Support for Ukraine), so I did not mind. It also contained one sticket and a paper map as well as guidance for muggles that did not read the wiki.

Tent

I found a good spot exactly where I wanted one. It ended up right next to a power station, which meant my short extension/splitter was enough to reach it. Being able to charge laptop and phone overnight in your tent is a must on a hacker camp.

At CCCamp, I did bring a router and long ethernet cable, but did not end up using it much because there was great WiFi coverage. This time I did not, and because life is hard and unfair, neither my laptop nor phone were able to keep a connection while in the tent. That meant I sadly did not use a Datenklo (data toilet).

While I got my home base ready, the sound check on the stages started, as stated on signs at the entrance earlier. The next half hour was filled with an instrumental version of “Never gonna give you up” with different equalizer settings and volumes.

Buildup

With events like this, the days are literally numbered, with day one being the day of the opening. A big team consisting of mostly unpaid volunteers had already spent a week on site building up the infrastructure. As with the German camp, the existing fixed infrastructure is not enough for the thousands of people going to the toilet, expecting cooled mate, playing on arcade machines or running pop-up data centers. On Day 0 (also sometimes called Day -0), heavy machinery was still driving around when I started exploring.

😱🍾💸

Being near the food court had the advantage that the first thing I found was a bar, so I wanted to get a mate. I was shocked (shocked!) twice.

The first thing was the mate itself. I was used to choosing between brands, Flora Power being my favorite, but here it was all the different flavors of Club-Mate.

// FIXME: ask at the bar to take a photo of the fridge and put it here

The other thing was that they did not acccept any cash. The German hacker events I went to were very proud about not accepting digital payments (Jede Kartenzahlung wird überwacht), and here it was the other way around! There was a way to pay pseudonymously though, and that was to charge an event payment card with cash at the info desk. According to the sticker on the card, you can reprogram them to do something else afterwards, which is nice.

What this prepaid card did not solve though was that you cannot pay if power is down. Later in the evening one of the bars had to close early due to exactly this issue. Whenever power was cut, the switch that provided PoE and internet to the payment terminals had to reboot, which took a while. Because they were frequent enough, that process repeated mutliple times without them being able to sell anything.

As with the mate, I think there would be a good middle ground somewhere in between. Maybe there could have been one cash register per bar at WHY and one accepting cards at C3. But I am not an organizer of things like this, so what do I know.

I continued exploring properly caffeinated.

Light and sound installation

// FIXME: ask V what it is called

Near the entrance of the event, there was a hill with a huge sign on it. A colorful contruction caught my eye, so I wanted to take a closer look.

I got to know the artist, let’s call them V here, who was still setting it up. They built a tent out of foamy, semi-translucent packing material, with RDB light strips adding movement. It looked like a giant isopod crawling up the hill.

The installation noticed when someone was outside and made an old rotary dial phone inside ring, which prompted the visitor to go inside and listen to the voice on the other end.

Angel badge

As I was already exhausted, I did not feel like volunteering that day, but I still got my angel status set to arrived and got the very pretty badge:

Milliways

I also checked Milliways, the restaurant at the end of the universe. They have a village or assembly at many hacker events and seem to have grown a lot over the years. Hacking area, bar, kitchen and midnight snacks all in one.

As usual, they brought their geodesic dome, this time decorated to feel jungle-like inside. The dome was one of the places playing techno most day and night.

What I did not know was that they usually get challenge coins made to finance the material costs. They were very pretty, but only payable in cash. As I hoped to not have to go to the next ATM in Alkmaar because of the cashless bars and food court, so I made a mental note to come back at the end of the event.

Need a pillow?

I found V at one of the other bars and we talked for a while. As this was their first hacker event, it was a great opportunity to share some great experiences I had at other events while drinking our first Tschunk. When I told him that when someone needs anything, it can usually be found, they asked for advice regarding camping gear. V did not know this was a camp before coming here, which was also why the installation was tent-shaped. I told them to ask at the info desk, which apparently worked great as they just got handed a blanket and pillow there for free.

I was yearning for exploration, so we went our separate ways. Even though the opening was still more than half a day in the future, the camp was already glowing in all colors of the rainbow. Parties and hackers everywhere, I had a great time finding orientation in the big area. My phone said I walked around 13,5k steps before 12 am.

Going to sleep

The sanitary conditions were great for a festival, and would still have been okay for a conference. After such a long day, I was not going to sleep without a proper shower. Without control over the temperature, I did not expect it to be super comfortable, but I really struggled cleaning myself because the water was freezing. Maybe it was my tiredness at 3:30 am after being awake for 21h, but I did not take a shower as cold in a decade. The next day the water was heated and maybe even too warm, so I do not know what happened there.

UPDATE: As it turned out, the gas for the water heater was just empty. They later put up a sign.

WHY 2025 - Preparations

Fri, 08 Aug 2025 02:30:00 +0200

I went to WHY2025, a Dutch hacker camp.

This is the first part of my travel log, covering everything happening before arriving on site.

Disclaimer WHY2025

As with all other posts, I may update previous ones without telling anyone. You may want to wait until I added all the photos and polished the text a bit.

Expectations

What Hackers Yearn 2025 is a so-called hackercamp. From the wiki:

At its core, a hackercamp is a self-organized space where hackers, makers, artists, activists, and technology enthusiasts come together to learn, create, and celebrate technology. The landscape comes alive with colorful LED installations, interactive art, hand-built sculptures, and high-tech experiments happening in every corner.

wiki.why2025.org/Newcomers_Guide

As is tradition with the CCCamp, the event takes place every four years, which means overall there is one “big” hackercamp every two years. It’s one of those events where your do not have to worry about stuff being stolen from your rent as much, but where you should install any available security patches beforehand.

While much of the infrastructure is semi-centrally managed, part of the fun is exploring the assemblies, villages and self organized sessions. As many of the talks are recorded and can be watched on media.ccc.de afterwards, I was mostly interested workshops and the social aspects of the event.

For me, this was the first hacker event in the Netherlands, the second hacker camp, and the sixth hacker event overall.

Tickets

Compared to the Chaos Communication Camp 2023, where it felt like getting a ticket required a good internet connection and solving a captcha quick enough, you could just go to their website and complete your purchase. The tickets did sell out in the end, but according to the public stats that happened over the span of days, not seconds.

By planning ahead, I was able to convince myself that taking a train very early in the morning was a good idea as the arrival time was more manageable. While I did say I “planned ahead”, that did not mean I committed to this early.

I booked my train ticket only a few days in advance, which meant that the FlexPreis (a ticket that is valid for any connection instead of a specific one) seemed like a good idea. I later noticed that when traveling over the border, seat reservations are required, defeating the whole purpose of buying the more expensive ticket for that part of the journey. Booking late also meant no bike tickets were left for the good connections, so I left that at home (I would have had to fix a flat first anyways).

Because I have a Deutschlandticket I can ride any regional trains for free. What that also means though is that your rights regarding connections are limited. Because of that, I usually take a train earlier so I do not miss my inter city train. In that case it meant getting up even earlier just so I could wait at Ostbahnhof for an hour.

Packing

As usual with events like this, I pushed any packing and so on to the last hours of the day before. Of course work also took longer than expected. Luckily just the weekend before, I was at the wonderful Brimborium, a tiny festival near the border to Poland, so I did not bother to store away my camping gear. That meant the basics were still scatered around on the living room floor and just had to be put back into (or attached to) the backpack.

I packed both way too much stuff and not enough. Whenever I thought I was done, I remembered something essential that was missing.

Because I had to carry all of this, I weighed what I wanted to take with me. Overall I had over 35kg of stuff. Though I did know that was too much, it was too late to send anything by mail or re-pack everything. This was now a problem for future me. Do you notice a theme yet?

Getting there

Boy did I regret all my decisions when my alarm rang. Going to bed at the usual time and just waking up really early did not get me started in a good mood.

Apart from some last-minute adjustments, I still had no cash, nothing to eat and nothing to do in the 10 hours of train travel I was going to do.

At the Späti (kiosk that never closes), I had the idea of buying some cheap headphones, as my five year old ones gave up only half a year ago and I did not get around to compare the gazillion options till now. I always wondered who buys stuff like that, now I know. First I was surprised they only cost 10€, but I found out why after getting them out of the package. They weighed nothing, which meant the badly fitting ear pieces were pulled out by the cables all the time. Before even boarding the first train, one of them fell apart because of a manufacturing defect in the molded plastic, revealing one of the two cables inside was not isolated at all. The sound was even worse than expected, producing a loud hiss while audio was playing. That made listening to podcasts especially annoying as after each sentence, the noise would stop for half a second, only to start again when someone continued to talk. The good news is that I did not have to endure that for long, as both earpieces fell apart while boarding my first connection and I was not able to fix them while on the train. 0/10 stars, worse than not having anything. Samsung, you should do better or not sell this crap at all.

As I did have a longer wait at Ostbahnhof, I was finally able to get some food and coffee. You should have seen me juggle my giant backpack, the big duffel bag, the food and a hot cup of black coffee. What had to happen did happen, and so the rest of the day I walked around with a big stain on my shirt and a slightly burned chest.

Update from more than a week later: the burned area is still clearly visible. The strap chafing on my chest all day probably did not help.

The actual train ride was comfortable and had a delay of only 30m, which is fine for a days worth of travel. My biggest complaint is that none of the toilets I checked had soap. If I had not packed as much stuff, it would say it was a better experience than getting there by car. I think it was funny that there Dutch crew stressed the point that you do not have to take a specific train in the Netherlands, even with the SparPreis, eliminating the last possible reason to have bought the expensive ticket.

From the train station, the event provided a free shuttle service to the camping ground.

Optimizing Rust binary size

Wed, 09 Apr 2025 20:29:48 +0200

In CCC Berlin, there is a large pixel matrix hanging on the wall that we call “ServicePoint display”. It receives commands from the local network via UDP, which contain things like very basic text rendering and overwriting parts of the pixel buffer. The commands are sent in an efficient binary data structure. I wrote (most of) the Rust library servicepoint, which implements this protocol including serialisation, deserialisation and a bunch of extras for easily crating these packets. There are also bindings for other languages, including C.

A few weeks ago, the only user of those C bindings I know informed me, with a big grin on their face, that they’d stop using the library and instead wanted to write everything by hand. While I know from experience that writing such a library is great fun (and thus does not need another reason), I was intrigued and wanted to know why. The main reason they cited was binary size, and while there’s probably something wrong with your computer if you do not have 1 MB to spare, I agreed that it was too big for what it does and that I would investigate. They knew what they were doing and it worked, I was immediately nerd-sniped and could not think about anything else in my spare time for a whole week. I had to find out why it was so big, and there would have to be a way to fix it.

This is part one, where I optimize the core library for size for fun and experience. The order in which I tried all the options is changed for a better text structure, but the results are re-created in the order they appear using the stated tools. In a future post, I also want to document how I got the C bindings smaller, as those use all features by default and cannot be reasoned about as much by the Rust compiler.

Most of the techniques I used are descibed in Minimizing Rust Binary Size, though I hope the specific example I provide makes the topic interesting to readers not writing Rust code.

Let’s get hacking!

Starting point

The commit I started on was fe67160974d9fed542eb37e5e9a202eaf6fe00dc, which is not part of main as of the writing of this post.

As I needed some binary to compare, I chose the example announce:

//! An example for how to send text to the display.

/// [1]
use clap::Parser;
use servicepoint::{
    CharGrid, CharGridCommand, ClearCommand, Connection, UdpConnection,
    TILE_WIDTH,
};

/// [2]
#[derive(Parser, Debug)]
struct Cli {
    #[arg(short, long, default_value = "localhost:2342",
        help = "Address of the display")]
    destination: String,
    #[arg(short, long, num_args = 1.., value_delimiter = '\n',
        help = "Text to send - specify multiple times for multiple lines")]
    text: Vec<String>,
    #[arg(short, long, default_value_t = true,
        help = "Clear screen before sending text")]
    clear: bool,
}

/// example: `cargo run -- --text "Hallo" --text "CCCB"`
fn main() {
    /// [3]
    let mut cli = Cli::parse();
    if cli.text.is_empty() {
        cli.text.push("Hello, CCCB!".to_string());
    }

    /// [4]
    let connection = UdpConnection::open(&cli.destination)
        .expect("could not connect to display");

    /// [5]
    if cli.clear {
        connection.send(ClearCommand).expect("sending clear failed");
    }

    let text = cli.text.join("\n"); /// [6]
    let command: CharGridCommand = CharGrid::wrap_str(TILE_WIDTH, &text).into(); /// [7]
    connection.send(command).expect("sending text failed"); /// [8]
}

Let’s just run you through the program quickly.

Some imports of the used libraries.
The structure Cli is defined to hold the command line arguments. clap is used to automatically derive a Parser from the attributes on the fields.
The command line arguments are parsed and a default value for the text to send is set.
A UDP connection is opened¹
Depending on the arguments, the screen is cleared.
All text snippets provided as an argument are concatenated with newlines in between. --text "Hallo" --text "CCCB" turns into Hallo\nCCCB.
The string is wrapped to the width of the display, resulting in a CharGrid, which is then immediately turned into a CharGridCommand. No fields are changed after this, so the text will be rendered in the top left of the screen when executed on the display.
The command is sent to the display.

At some steps, the program panics with an error message in case something went wrong.

I started with rustc 1.82.0 (f6e511eec 2024-10-15) from nixpkgs 0ff09db9d034a04acd4e8908820ba0b410d7a33a. For compiling the example, I just used the usual cargo build --release --example announce and checked the binary size with ll -B target/release/examples.

The resulting size was 1.1 MB, which should be easy enough to beat.

Low hanging fruits

Compiler options

The first thing that came to mind was telling the compiler to optimize for size, like with gcc -Os. The Rust equivalent is opt-level = "s", and for even more optimization, z also disables loop vectorization.

Option	size in isolation (change)	size cumulative (change)
baseline	1.137.384	1.137.384
opt-level = ‘z’	1.186.104 (+48.720)	1.186.104 (+48.720)
opt-level = ’s'	1.120.416 (-16.968)	1.120.416 (-65.688)
lto = true	914.496 (-222.888)	808.528 (-311.888)
codegen-units = 1	982.904 (-154.480)	775.888 (-32.640)
panic = ‘abort’	979.840 (-157.544)	703.096 (-72.792)
strip = true	915.944 (-221.440)	580.056 (-123.040)
switching back to opt-level = ‘z’		555.480 (-24.576)

So it turns out, if you want to halve your binary size, a few flags are enough in stable Rust. The most significant impacts came from link time optimization (LTO) and stripping of symbols from the binary. Interestingly, differnet combinations of these settings didn’t scale the way I would have intuitively thought.

The only compromise apart from compilation time with these settings is the change in panic behavior, as this means no stack traces on crash².

To only compile like this in specific scenarios, you can add a new profile to a crates Cargo.toml like this:

[profile.size-optimized]
inherits = "release"
opt-level = 's'     # Optimize for size
lto = true          # Enable link-time optimization
codegen-units = 1   # Reduce number of codegen units to increase optimizations
panic = 'abort'     # Abort on panic
strip = true        # Strip symbols from binary

The profile can be used by passing --profile=size-optimized instead of --release to cargo build. Because of the different profile, the binary ends up in a different folder (ll -B target/size-optimized/examples to check size).

Features

Rust has a very handy way to represent variability in a library called features. The servicepoint library has the following declaration in it’s Cargo.toml:

[features]
default = ["compression_lzma", "protocol_udp", "cp437"]
compression_zlib = ["dep:flate2"]
compression_bzip2 = ["dep:bzip2"]
compression_lzma = ["dep:rust-lzma"]
compression_zstd = ["dep:zstd"]
all_compressions = ["compression_zlib", "compression_bzip2", "compression_lzma", "compression_zstd"]
rand = ["dep:rand"]
protocol_udp = []
protocol_websocket = ["dep:tungstenite"]
cp437 = ["dep:once_cell"]

Line two means by default, cargo will enable LZMA compression, sending via UDP sockets and conversion between CP-437 and UTF-8. Each of those features pulls in an optional dependency (which is why I made those features toggleable in the first place). In the code, CP-437 and compression are not needed³, but UDP is obviously used.

Features can be toggled on the command line, which means the invocation can be changed to the following: cargo build --example announce --profile=size-optimized --no-default-features --features=protocol_udp⁴. Doing that means less library code and less dependencies are pulled into the compilation process.

The result is a 555.480 Byte binary, which is exactly the same as without those flags. This is not really surprising, as we enabled a bunch of compiler options that help remove whole sections of code that are not needed, especially link time optimization. It is cool to see that the binary is identical, though.

In the rest of this post, I will omit those parameters, probably to the detriment of compilation time.

Digging deeper

While this was a big improvement already, this was still 50 times the size of the C program.

If it was this easy halving it, can I do that a second time?

Everything from here on required unstable features of the rust toolchain, both because tooling depends on it for more information about the program, and because the compiler options from here on are (and maybe never will be) stabilized.

The version I ended up with was rustc 1.88.0-nightly (5e17a2a91 2025-04-05). In my environment, I had to call nightly cargo with rustup run nightly cargo, but that part is not included in the rest of the commands. The executables I got with the unstable version were already a bit smaller again (546.528 bytes).

The first thing I noticed was that I got some new warnings when compiling, all of which I fixed immediately. As it was mostly inside of the documentation, I did not expect this to affect file size.

Next up, I added cargo-bloat to my flake. This tool can show you which functions take up most of the space in your binary. The invocation is similar to building - cargo bloat --example announce --profile=size-optimized resulted in the following output:

File  .text     Size        Crate Name
 1.0%   5.5%  21.0KiB clap_builder clap_builder::parser::parser::Parser::get_matches_with
 0.9%   5.3%  20.5KiB          std std::backtrace_rs::symbolize::gimli::Cache::with_global
 0.6%   3.3%  12.6KiB          std std::backtrace_rs::symbolize::gimli::Context::new
 0.4%   2.4%   9.2KiB          std gimli::read::dwarf::Unit<R>::new
 0.4%   2.1%   7.9KiB          std addr2line::line::LazyLines::borrow
 0.3%   2.0%   7.5KiB     announce announce::main
 0.3%   1.8%   7.1KiB          std miniz_oxide::inflate::core::decompress
 0.3%   1.6%   6.3KiB          std addr2line::unit::ResUnit<R>::find_function_or_location::{{closure}}
 0.3%   1.5%   5.6KiB clap_builder clap_builder::builder::command::Command::_build_self
 0.2%   1.4%   5.3KiB clap_builder clap_builder::output::help_template::HelpTemplate::write_templated_help
 0.2%   1.3%   5.1KiB clap_builder clap_builder::error::Error<F>::print
 0.2%   1.3%   4.9KiB clap_builder clap_builder::parser::parser::Parser::react
 0.2%   1.2%   4.8KiB clap_builder clap_builder::output::help_template::HelpTemplate::write_args
 0.2%   1.2%   4.6KiB          std gimli::read::unit::parse_attribute
 0.2%   1.1%   4.4KiB          std addr2line::function::Function<R>::parse_children
 0.2%   1.0%   3.7KiB clap_builder clap_builder::output::help_template::HelpTemplate::write_subcommands
 0.2%   1.0%   3.7KiB clap_builder clap_builder::output::usage::Usage::write_arg_usage
 0.2%   1.0%   3.7KiB          std gimli::read::rnglists::RngListIter<R>::next
 0.1%   0.8%   3.1KiB          std std::backtrace_rs::symbolize::gimli::elf::<impl std::backtrace_rs::symbolize::gimli::Mapping>::new_debug
 0.1%   0.8%   3.0KiB clap_builder clap_builder::parser::parser::Parser::match_arg_error
10.8%  61.8% 237.3KiB              And 993 smaller methods. Use -n N to show more.
17.5% 100.0% 384.2KiB              .text section size, the file size is 2.1MiB

Starting with the largest, the biggest functions in the program are shown. From the table, we can already see some interesting stuff.

For some reason, the .text section (the machine code) is only a small part of the executable, and the total size increased by a factor of 4.
The biggest function and a bunch of other big ones are from clap_builder, a crate that is part of the command line argument parser.
std thakes up most of the rest.
main is unexpectedly huge?
servicepoint does not even show up in the top list.

Let’s cover those points in order.

1. Unexpected binary size when building via cargo-bloat

Using GNU size, we can check the size per section in the ELF binary. Using -G or -B output formats does not work for this, as it will only show the .text and .data section, which in this case only make up around 500KB. Thus the command I used was size -A --common target/size-optimized/examples/announce, giving the following result:

section                size     addr
.dynsym                1680      856
.dynstr                1198     3500
.rela.dyn             22800     4704
.gcc_except_table      3728    27552
.rodata               36592    31280
.eh_frame_hdr          8116    67872
.eh_frame             52488    75992
.text                393449   132576
.data.rel.ro          18760   530248
.relro_padding         2888   550072
.data                  2400   554168
.debug_abbrev          1810        0
.debug_info          525404        0
.debug_aranges         6256        0
.debug_ranges        157856        0
.debug_str           726991        0
.debug_line          149936        0
Total               2115147

(I filtered out the rows <1KB for brevity)

Turns out cargo-bloat disables symbol stripping, because it needs those to show to the user. It’s not even the symbols that are included in release builds by default - all the debugging information is included. That means, I can ignore that problem and focus on the .text size.

2. Removing clap

While clap is super handy, it looks like the code needed to parse two simple arguments blows up the executable. As the C program I was comparing against had all the parameters hard-coded, I just ripped out the dependency and hard-coded the values I needed.

The result is the first version of tiny_announce, as I did not want to change the existing example.

//! An example for how to send text to the display.

use servicepoint::{
    CharGrid, CharGridCommand, ClearCommand, Connection, UdpConnection,
    TILE_WIDTH,
};

/// example: `cargo run -- --text "Hallo" --text "CCCB"`
fn main() {
    let text = "Hello, CCCB!";

    let connection = UdpConnection::open("127.0.0.1:2342")
        .expect("could not connect to display");

    connection.send(ClearCommand).expect("sending clear failed");

    let command: CharGridCommand = CharGrid::wrap_str(TILE_WIDTH, &text).into();
    connection.send(command).expect("sending text failed");
}

The command to compile changed slightly because of the new name. cargo build --example tiny_announce --profile=size-optimized && ll -B target/size-optimized/examples/tiny_announce gave me the new binary size. Drumroll… 324.624 Bytes! With argument parsing removed, we saved 40% of the remaining binary size. This also makes the main disappear from the top sized functions for now.

While removing a library you do not really need is also available in stable Rust, I was only able to notice that with tooling only available on nightly, so I am putting it into that category.

3. build-std

Looking at the biggest functions again (now cargo bloat --example tiny_announce --profile=size-optimized) showed that all the big functions left were from std. Most of that looked like stack unwinding and debug data parsing, which is odd as we added panic = 'abort' in the first chapter.

As it turns out, as an optimization for the development workflow, by default cargo does not recompile the standard library. Instead, a prebuilt version included in the toolchain is used. The compiler arguments for that are fixed, and to change that and get more control over how stdlib is compiled and linked, we neeed the unstable option -Zbuild-std and have to list which sub-crates we want to build (which is pretty much all of them). Because we also have panic = "abort" set, we need to also pass in -Zbuild-std-features="panic_immediate_abort" so there is no compilation error.

cargo build --example tiny_announce --profile=size-optimized -Zbuild-std="core,std,alloc,proc_macro,panic_abort" -Zbuild-std-features="panic_immediate_abort"

This produces a binary that is now only 30.992 bytes!

In-between find: to_socket_addrs

The remaining top 3 functions were:

 File  .text    Size         Crate Name
 4.4%  11.0%  2.0KiB           std <&T as std::net::socket_addr::ToSocketAddrs>::to_socket_addrs
 3.8%   9.4%  1.7KiB tiny_announce tiny_announce::main
 2.9%   7.3%  1.4KiB     [Unknown] main

Main shows up again! But what is that? 4.4% used by to_socket_addrs? We found the last string parsing code, this time in the standard library, to read the IP and Port from a string. After changing it in the example, it still showed up which brings me to the first and only time I actually changed the servicepoint library as a result from this saga.

- let socket = UdpSocket::bind("0.0.0.0:0")?;
+ let addr = SocketAddr::from(([0, 0, 0, 0], 0));
+ let socket = UdpSocket::bind(addr)?;

This also seemed to remove other functions as well, as the size was down to 17.272 bytes, nearly halving the size again. It is now smaller than this article as plain text markdown.

4. no_main

You’d think that now main is the top function, but Iter::next is now the biggest function for some reason. Still, [Unknown] main and the actual main take up 10% of the remaining size according to cargo bloat.

We surely cannot reduce that, right? Wrong! With #[no_main], you can tell Rust to not add any initialization code. This means the normal fn main() does not get used, and the linker complains about the missing function. To fix this, the function can be converted to a C-style main.

I also removed some more code by initializing the CharGrid directly instead of wrapping a string, which saved 400 bytes.

#![no_main]

use servicepoint::{
    CharGrid, CharGridCommand, ClearCommand, Connection, UdpConnection,
};
use std::net::SocketAddr;

#[unsafe(no_mangle)]
pub extern "C" fn main(_argc: isize, _argv: *const *const u8) -> isize {
    // not parsing the address from str removes 3KB
    let addr = SocketAddr::from(([172, 23, 42, 29], 80));

    let connection = UdpConnection::open(addr).unwrap();
    connection.send(ClearCommand).unwrap(); // <--

    let grid = CharGrid::from_vec(5, vec!['H', 'e', 'l', 'l', 'o', 'W', 'o', 'r', 'l', 'd']).unwrap();
    connection.send(CharGridCommand::from(grid)).unwrap();
    0
}

This resulted in a 8.064 byte executable, finally beating both GCC and LLVM compiling the minimal C program (around 10KB). If we were to remove the marked line and not clear the screen, we could drop it further to 7.696 bytes.

Advanced compiler abuse

There are two things left to reach the absolute bottom without ripping out the standard libary alltogether.

In Rust, a function can tell the compiler to get the calling location as a parameter to the function. With -Zlocation-detail=none, we instruct the Rust compiler to just not bother with that.

-Zfmt-debug=none is similar but worse, because it changes all the default Debug implementations to do nothing at all. The change in behavior is not obvious in this example, but do this in an application that has logging and it will be horribly broken.

As an icing on the cake, those two options cannot be passed via cargo arguments, so we have to use the environment variable RUSTFLAGS to pass this through to when rustc is invoked.

The final command to build the tiniest possible announce in all it’s glory:

RUSTFLAGS="-Zlocation-detail=none -Zfmt-debug=none" \
cargo build \
    --example tiny_announce \
    --profile=size-optimized \
    --no-default-features \
    --features=protocol_udp \
    -Zbuild-std="core,std,alloc,proc_macro,panic_abort" \
    -Zbuild-std-features="panic_immediate_abort"

All of this reduces the final binary size to 8.064 bytes.

Conclusion

Through this journey, I’ve managed to reduce the size of the binary from an unwieldy 1.1 MB to an impressive 8 KB, without sacrificing all of the standard library. For me the most unexpected was the size of clap code, though I learned dozens of things at every step about the intricacies of how cargo build produces your final binary.

There is no single option that in itself is the solution, it’s a matter of experimenting with a combination of compiler flags, feature toggles, and code optimizations. While extreme options can be great if you want to squeeze out the last bytes, it’s probably not worth using those in a “normal” computer scenario.

The key takeaway is that optimizing binary size in Rust, while not always straightforward, is achievable with the right techniques. It is certainly easier to create a big binary than in C, calling Rust bloated is blaming the language a bit too much.

Stay tuned for part two, in which I will try to do something similar with a C version of the example, using the C bindungs of the servicepoint crate.

Yes, I know UDP does not have connections. Internally, this just opens a UDP socket ↩︎
Technically, you can catch a panic while unwinding and there may even be a weird performance argument for doing that, see ↩︎
Some commands can be compressed, but the text ones (both CP-437 and UTF-8) cannot. Clear is a very simple command that does not have any payload, so no compression there either. If a BitmapCommand was used instead, using into() on a Bitmap would have hidden the fact that the default compression is used in that case. The default compression in turn is either LZMA or no compression, depending on whether the LZMA feature is enabled. ↩︎
This works here because announce is an example inside of the library itself. As an actual dpendent, you would specify this in your Cargo.toml. ↩︎

Why I do not use flake-utils

Sun, 06 Apr 2025 14:49:03 +0200

I have been using Nix for a while now. Around a year ago, I switched everything from the servicepoint library to my machine configuration over to flakes.

For me the biggest advantage flakes bring is not additional functionality. Instead, they are an easier and semi-standardized way to do what you could before.

When learning flakes, you often see flake-utils being used. With it, you can shorten your flakes by not having to specify everything per system.

Edit: The patterns described here as well as flake-parts are also compared in Practical Nix flake anatomy.

Without anything

{
  description = "Flake utils demo - without flakes";

  outputs = { self, nixpkgs, flake-utils }:
  {
    packages."x86_64-linux" = rec {
          hello = pkgs.hello;
          default = hello;
    }
    packages."aarch64-linux" = rec {
          hello = pkgs.hello;
          default = hello;
    }

    # more systems ...
  }
}

With flake-utils

{
  description = "Flake utils demo";

  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let pkgs = nixpkgs.legacyPackages.${system}; in
      {
        packages = rec {
          hello = pkgs.hello;
          default = hello;
        };
      }
    );
}

With function in flake

To make a long story short, here is what I usually do instead:

{
  description = "forAllSystems demo";

  outputs = { self, nixpkgs }:
    let
      supported-systems = [ # [1]
        "x86_64-linux"
        "aarch64-linux"
      ];
      forAllSystems =
        f:
        nixpkgs.lib.genAttrs supported-systems /* [2] */ (
          system:
          f rec {
            pkgs = nixpkgs.legacyPackages.${system}; # [3]
            inherit system;
          }
        );
    in
    {
      packages = forAllSystems ({ pkgs, ... }: rec {
        hello = pkgs.hello;
        default = default;
      });
    }
}

That’s definitely more code! - Yes, but it also includes more information in the flake, while getting rid of an external dependency. While more code can be intimidating for beginners, it actually helps remove a barrier to understanding how the flake works in this case. For me, it wasn’t a problem to ignore boilerplate like this at first, slowly learning more language features until I finally understood everything.

At [1], the supported systems are specified. I personally use x86_64-linux and aarch64-linux, but I also usually support x86_64-darwin and aarch64-darwin in public projects. If you want to support any system, you can use nixpkgs.lib.system.flake-exposed at [2] instead of defining your own list.

Example variations

Because the definition is right inside the flake, you can tweak what gets passed to the function. By doing that, you have to explicitly add the new identifier to each part that uses it, instead of having a global let binding that is implicitly used. system being available here is another bonus, as otherwise this would require duplicate lets everywhere.

An example for how to do it is already right there: at [3], pkgs is provided. Some real-world usages I wrote or encountered:

servicepoint-life flake: passes in initialized naersk instance and a shorthand to refer to packages defined in the flake itself
RedoxOS development flake: uses it to pass a custom rust-toolchain
my NixOS flake: uses something similar for per-host config

Another possible tweak: You may want to define separate supported systems for each output. This is useful, for example, if the target environment you’re developing for cannot support a development shell.

Conclusion

For me, the trade-offs are worth it, as they provide greater transparency and control over the flake configuration. For you, this may be different, in which case keep using it! Ultimately, it’s also matter of personal preference.