Hey,

This week I wanted to get a better understanding of firecracker, and it turns out that I’ve actually have never booted my own build of Linux.

It turns out that you can build and boot a system that you created from scratch relying entirely on userspace tools (essentially, QEMU), which emulates a machine down to the processor level, making the barrier way lower than if we needed to deal with details of hardware.

I’ve used QEMU before when I was getting Concourse workers to run on a Raspberry PI device (see A Raspberry PI Concourse Worker) to emulate ARMv7 from my x86_64 macOS, but that was just user-mode emulation (getting the code that was built for a specific instruction set to be run in a system that runs other instruction set), not really knowing about the possibilities that such tool could allow in other modes (like, system emulation, where it provides a full computer system).

For this, I followed two blog posts:

Below you find my notes with some comments on each step.

1. create a “workspace” directory

There’s more than a single thing to be built, so, let’s keep it all nice and clean under a single directory tree.

    export WORKSPACE=$(pwd)
    mkdir -p kdev

2. install build and runtime dependencies

Assuming you’re on Ubuntu Disco (19.04):

    apt update && apt install -y \ 
            bison \
            build-essential \
            flex \
            git \
            libelf-dev \
            libssl-dev \
            ncurses-dev \
            qemu \
            qemu-system-x86

ps.: you can see qemu-system-x86 here as in this build the goal is to target a x86_64 machine.

3. clone and build linux

Be prepared to pull few GBs from GitHub, and spend few minutes of compilation time.

    # retrieve the source code right from github rather than `git.kernel.org` 
    #
    git clone https://github.com/torvalds/linux
    cd $_


    # leverage the most minimum configuration
    #
    # >  defconfig	- New config with default from ARCH supplied defconfig"
    #
    make O=$WORKSPACE/build/linux-x86-basic allnoconfig

There are just three features that we need to activate here though:

This is where the process building Linux starts overlapping with the learnigs from the “Writing an OS in Rust” series: in the Serial Port section (https://os.phil-opp.com/testing/#serial-port), we leverage a Crate called [uart_16550] that implements exactly this functionality.

To activate those, we select those options in their corresponding sections:

    # bring up the configuration using ncurses
    #
    make O=$WORKSPACE/build/linux-x86-basic nconfig

In the configuration menu:

    [*] 64-bit kernel

    -> General setup
            [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support

    -> Executable file formats / Emulations
            [*] Kernel support for ELF binaries
            [*] Kernel support for scripts starting with #!

    -> Device Drivers
      -> Character devices
            [*] Enable TTY
        -> Serial drivers
            [*] 8250/16550 and compatible serial support
            [*]   Console on 8250/16550 and compatible serial port

    -> File systems
      -> Pseudo filesystems

            [*] /proc file system support
            [*] sysfs file system support

ps.: on each menu, you can search using / (like in vim).

That done, under $WORKSPACE/build/linux-x86-basic/.config you can find the generated .config file that specifies the parameters of your system.

With the configuration prepared, we can move on to the compilation.

    make O=$WORKSPACE/build/linux-x86-basic -j$(nproc)

With the kernel built, we can move on to preparing the root filesystem that it’d use when booting up so that we can try doing something while at userspace.

4. setup the rootfs

initramfs:

The only purpose of an initramfs is to mount the root filesystem.

At boot time, the boot loader loads the kernel and the initramfs image into memory and starts the kernel

The kernel checks for the presence of the initramfs and, if found, mounts it as / and runs /init

first, we download and extract busybox so that we can build it from source code:

    BUSYBOX_VERSION=1.31.0

    # download the tar.bz2 file, piping it right to tar so that it extracts
    # it as it goes.
    #
    curl https://busybox.net/downloads/busybox-${BUSYBOX_VERSION}.tar.bz2 | tar xjf -
    cd ./busybox-${BUSYBOX_VERSION}

Then, just like with the kernel, create a compilation configuration being created at an output location under $WORKSPACE/build, then compile the code.

    # create the destination and then make the config
    #
    mkdir -p $WORKSPACE/build/busybox-x86
    make O=$WORKSPACE/build/busybox-x86 defconfig


    # compile the code producing a static binary
    #
    LDFLAGS="--static" make O=$WORKSPACE/build/busybox-x86 -j$(nproc)
    make O=$WORKSPACE/build/busybox-x86 install


    # create the target directory for our simplistic version of initramfs, and then
    # copy the files over to it.
    #
    mkdir -p $WORKSPACE/build/initramfs/busybox-x86/{bin,sbin,etc,proc,sys,usr/{bin,sbin}}
    cp -av $WORKSPACE/build/busybox-x86/_install/* $WORKSPACE/build/initramfs/busybox-x86

Write our init script (note: we can use a plain shell script because we enabled the interpreation of “shebangs” in our kernel).

    cat << EOT >> $WORKSPACE/build/initramfs/busybox-x86/init
    #!/bin/sh

    mount -t proc none /proc
    mount -t sysfs none /sys

    echo -e "\nBoot took $(cut -d' ' -f1 /proc/uptime) seconds\n"

    exec /bin/sh
    EOT

    chmod +x ./init

create a cpio gzipped archive that corresponds to the initramfs:

    find . -print0 \
       | cpio --null -ov --format=newc \
       | gzip -9 > $WORKSPACE/build/initramfs-busybox-x86.cpio.gz

5. boot

then, finally, boot on top of qemu:

    qemu-system-x86_64 \
            -kernel $WORKSPACE/build/linux-x86-basic/arch/x86_64/boot/bzImage \
            -initrd $WORKSPACE/build/initramfs-busybox-x86.cpio.gz \
            -nographic -append "console=ttyS0"