RISC-V: running GNOME OS on SiFive hardware for the first time

We continue to participate in the promising RISC-V ecosystem. Already in 2021, Ben Dooks fixed a challenging kernel bug, Robin Heywood prototyped an Arduino Mini clone, and we completed the initial bootstrap of Freedesktop SDK to the RISC-V architecture.

Since our last post, James Thomas and Valentin David worked on stabilising the remaining parts of Freedesktop SDK. We are delighted that the new 21.08 release of Freedesktop SDK comes with RISC-V support out of the box, as this opens up the whole Flatpak app ecosystem on RISC-V platforms. All this was first demo'd in a talk at Linux App Summit 2021.

GNOME's testing-only OS was our next goal and we recently demo-ed this running on RISC-V, first on virtual hardware and then on a SiFive Unmatched board. This lets us validate the whole GNOME desktop software stack on RISC-V, which is exciting for everyone - many of the same software components are also used in the commercial Linux space.

So how close are we to a full desktop experience? Will desktop Linux be ready when the first RISC-V laptops arrive? Read on for more about what we've done and what challenges remain.

Building for RISC-V

RISC-V support is now available in stable releases of the various toolchains we need (with a little help from Codethink in the case of Rust).

Valentin summarized the status of toolchain support in his GUADEC 2021 talk:

Freedesktop SDK has a clean bootstrap process designed to support cross-compilation and to avoid any circular dependencies, so the initial RISC-V bootstrap was quick. The majority of its components must be native-compiled and this is more tricky. Due to IO and CPU constraints, compiling on today's RISC-V hardware takes a long time - it's faster to emulate a RISC-V system on your x86_64 laptops and build there.

QEMU user mode

QEMU is a powerful, open source machine emulator. In "user mode", it allows a single running Linux kernel to run binaries built for other architectures, without needing a full virtual machine. The binaries can be executed transparently using the kernel's binfmt_misc feature. Following our instructions, the whole BuildStream build process can run on an x86_64 machine and produce working RISC-V binaries and images.

While running our first builds with this method, we encountered a bug that caused some programs to hang. We first noticed the issue in cargo, the build tool for Rust. The hang can be triggered when a program uses fork() and threads, and when we investigated we saw it was inside GLib, a dependency used in a lot of Linux desktop software. This library is used by QEMU itself, but it might also be used in the process being emulated. When a fork() call happens, GLib's memory allocator hits a situation that it isn't designed to handle, and its thread synchronization locks up completely.

We discussed this upstream in a GLib issue and a QEMU issue, and we hope that QEMU can eventually work around the problem.

In the meantime, you can work around it manually by telling GLib to use malloc() instead of its own allocator - malloc() isn't necessarily safe to use in a forked process either, but the GNU C Library guarantees that it is. To enable this workaround you set a special environment variable: G_SLICE=always-malloc.

GNOME on virtual hardware

There are a lot of components involved in today's graphical desktops. GNOME defines all of these in the gnome-build-meta repository, and uses BuildStream and Gitlab CI to automate building them. Over the last 12 months we worked to enable RISC-V builds across the stack, including configuration changes, component upgrades, and lots of testing. This is now all merged to gnome-build-meta (in MR 1112) and its dependency Freedesktop SDK (issue 1165), enabling anyone to build and test mainline GNOME for RISC-V using the instructions in Merge Request 1112. Try it out!

The end result of this process is shown here: a happy Valentin playing a game of SuperTuxCart in a RISC-V virtual machine:

SuperTuxKart RISC-V running as @FlatpakApps on a bootable image of @FreedesktopSdk on qemu-system-riscv64. pic.twitter.com/XHIuYIKPiI

— Valentin David (@valentindavid) February 18, 2021

GNOME on bare metal: SiFive Unmatched

SiFive are a big player in the RISC-V hardware space. In early 2021 we ordered several HiFive Unmatched devboards and waited excitedly for them to arrive. As soon as they did, we set it up in a case and we got to work preparing GNOME OS for the new platform. There is no built-in graphics hardware on the Unmatched board, so we added a Sapphire PCIe graphics card, shown here:

The two main challenges going from virtual to real hardware are getting the initial boot process to work, and ensuring the specific device drivers and other kernel tweaks are all available. Codethink has already done this for some ARM devices - you can already test GNOME OS on Pinebook Pro, Raspberry Pi 4 and more. The SiFive board isn't fully supported in mainline Linux yet, so we apply the necessary patches within Freedesktop SDK until they land upstream, along with the SiFive specific firmware and device tree configuration. You can see most of the changes along with basic instructions in MR 5046 - "Add support for SiFive Unmatched". The end result was demoed during Valentin's talk at GUADEC 2021:

Let's talk about bootloaders

The boot process is often intricate and it was one of the most complex aspects of the Unmatched bringup. I asked Valentin to help me understand OpenBSI, a component specific to the RISC-V boot process:

Sifive Unmatched firmware will load an image of U-Boot with OpenSBI from specific partitions. U-Boot will then find systemd-boot in the EFI boot partition and execute it. Through the EFI API of U-Boot, systemd-boot will load the correct kernel EFI-compatible image file. Note that the disk where U-Boot+OpenSBI is loaded from does not have to be the same as the EFI boot disk since U-Boot will scan for EFI Boot partitions in all detected disks.

The nice thing about this approach is we can use the same installer ISO for QEMU and for the Unmatched board. We prepare an SD card with U-Boot, OpenSBI and a device tree. This loads the installer ISO from a USB stick, and the GNOME OS image is installed to the attached hard drive. That's the idea, but it took some work to make this work on the Unmatched board.

Most of the boot software that we use, like dracut, and systemd-boot, required only minimal patches. But they depend on GNU EFI, a bridge between the usual Linux executable format and the COFF/PE32+ format used by EFI. Mainline GNU EFI provides support for x86 and ARM families of processors. As part of the "Add EFI boot for RISC-V" merge request, we added a large patch which enables support for building EFI binaries on RISC-V, which works well but has some remaining issues to solve before it can be merged upstream.

Next steps

In the short term, we have some work to make sure the various patches are reviewed and merged by the relevant upstream projects.

Long term, we want to see RISC-V hardware with the same capabilities as other platforms. The work we've done here shows what is possible, but we can't yet enable RISC-V builds by default in GNOME and Freedesktop CI pipelines - limited processor and IO speeds mean that it would slow down the developer workflows if we ran a RISC-V build and test cycle for every merge requests, so some of this work depends on further iterations by hardware manufacturers.

Once we are able to source fast RISC-V build machines, we'll look at linking them up to and the GNOME and Freedesktop Gitlab CI instances. This will enable continuous testing to ensure first-class support, and by drawing on our knowledge of OpenQA and LAVA we could even start ongoing hardware testing. We can also add builders to the Flathub build service for apps, and at that point we'll have everything in place to bring the full Flatpak ecosystem of apps to RISC-V.

Keep up with our RISC-V news

Fill out the form to receive our latest updates on the open CPU architecture in your inbox.

Related blog posts:

• Our start with RISC-V: Codethink's first research about the open instruction set >>
• Making a breakout board for a RISC-V microcontroller: RISC-V: A Small Hardware Project >>