Bringing up a new distro for the CVA6 RISC‑V FPGA processor

In this blogpost we will cover how to bring the CVA6 processor up with Linux on it and how we replaced the OS there with a different one, based on Freedesktop SDK, updating the kernel version on the way.

CVA6 is being developed by the OpenHW Group, part of the Eclipse Foundation, of which Codethink is a member. We thought it would be useful to see how we can combine our approach to building software with the processor developed by a fellow Eclipse Foundation member.

CVA6 has its source code available on GitHub, written in SystemVerilog. It implements the RISC‑V instruction set and can target a few FPGA boards. We found the best FPGA board with CVA6 support was the Digilent Genesys2, so we selected that to get going.

We also had a lot of experience with using Freedesktop SDK as a platform for building a Linux OS, so we wanted to try to run it on CVA6. Freedesktop SDK allows you to easily get reproducible builds, and be as close to upstream as possible.

During this project we managed to build and deploy a new OS, and solved a few open issues along the way. We've proposed updates and fixes to the CVA6 SDK. For QEMU, we proposed an initial CVA6 implementation with ethernet support. And we've published the build instructions for our OS on GitHub at CodethinkLabs/cva6-sdk-bst, which you can build yourself using BuildStream – more on that below. Read on to find out how we did it, and what we'd like to happen next!

Generating and flashing the bitstream

The configuration data that's loaded into an FPGA is called bitstream, and all FPGA vendors have their own software suites to generate bitstreams. Despite being an open-source processor, CVA6 requires one of these proprietary tools to generate its bitstream. There's an open-source alternative called Yosys, but it's lacking some functionality of commercial tools.

We are targeting a Xilinx FPGA, so we have to use the Xilinx's software suite called Vivado. CVA6 requires Vivado 2018.2 version to compile, which in turn requires old Ubuntu to run. To work around that we used a container. You can find instructions on how to install Vivado 2018.2 in a container here.

We also need to compile RISC‑V GCC toolchain following instructions in the CVA6 repository.

Now that we have working Vivado and RISC‑V GCC, we can generate the bitstream from the CVA6 repository as follows:

source /path/to/Xilinx/Vivado/2018.2/settings64.sh
LD_PRELOAD=/lib/x86_64-linux-gnu/libudev.so.1 CC="$RISCV/bin/riscv-none-elf-gcc" make fpga

The LD_PRELOAD is to work around this issue. The $RISCV variable comes from toolchain installation.

The next step is flashing the bitstream to the board. Add the udev rule to your system and then follow the flashing instructions.

After flashing you should see zero-stage bootloader output on the board's UART port. It tries to load something from the SD card, but since we haven't inserted the card yet, the bootloader just waits and does nothing.

Building cva6-sdk

Now let's generate an SD card image for our processor. The official way of doing it is to use cva6-sdk, which can generate SD card image with OpenSBI, U‑Boot, and Linux with busybox initrd. To do that, run make images in the cva6-sdk repository. After that insert an SD card into your computer, and run sudo -E make flash-sdcard SDDEVICE=/dev/sdX, where sdX is your SD card.

Now insert the SD card into the Genesys2 board, and power it on. On the UART port you should see booting logs and eventually get to the busybox shell. You can play tetris there if you run /tetris :-)

We now have Linux running on our processor, even though the kernel and userspace versions are somewhat outdated.

Using Freedesktop SDK userspace with old kernel

On our way to full Freedesktop SDK OS, we first aimed at an intermediate step where we wanted to use cva6-sdk's bootloader and kernel but replace busybox initrd with our rootfs.

We modified cva6-sdk to use a separate partition for rootfs and not include initrd in the kernel – this allowed us to easily use Freedesktop SDK based rootfs. To generate it we've written a few BuildStream “elements”, which are .bst files with YAML syntax that describe how to produce various software, in our case – rootfs image. As part of doing that we replaced default Freedesktop SDK kernel headers with the older ones from cva6-sdk, and disabled some profiling functionality in binutils, to fix a compilation error.

Our first attempt to boot the new rootfs failed because systemd was timing out waiting for the serial device to appear. It was a problem with the slow boot process – by the time systemd-udev-trigger.service started, it was too late, the serial device systemd unit had already stopped waiting. We solved this by increasing the default device timeout. We also disabled a few non-critical systemd services as they were taking a very long time to start. This is something to revisit later.

After doing that we were able to log into Freedesktop SDK userspace!

Using new kernel

With Freedesktop SDK userspace working, it was time to update the kernel. cva6-sdk used Linux v5.10, which is an LTS kernel with support until 2026, but we wanted the newest stable release – v6.14 in this case. Using newer version brings us better performance and some new features.

The first problem we encountered was the larger kernel size – U‑Boot's bootcmd wasn't reading enough data from the SD card, so we increased that amount.

The kernel started booting now but was triggering kernel panic at a later stage. We found out that it was happening because of the memory protections set up by OpenSBI. Turned out in newer kernel versions Linux doesn't reserve the first 2MB of memory for OpenSBI as it used to. Adding a reserved-memory node to the device tree fixed this issue.

Then we saw systemd failures. On startup it printed an error message about being unable to set its environment, and frozen execution. Also, the error messages were truncated. Unclear how to debug that, we turned to git bisect, with which we found that the errors were happening when Linux was doing unaligned memory accesses. When we disabled them with CONFIG_RISCV_EMULATED_UNALIGNED_ACCESS, the problem went away. We didn't get to the bottom of it, so it's something for future TODO.

Now we have the new kernel working. \o/

We were lacking some functionality though – cva6-sdk had a patch for v5.10 kernel with the board's ethernet driver, so we wanted to port it to v6.14. The porting itself was straightforward – fixing compilation errors, and one runtime error. We noted that the driver itself was poorly written and ideally needed to be rewritten from scratch, but we decided not to do that at the moment, so this is another thing for future TODO. Despite that, the ported driver worked and allowed us to use ssh to connect to the board.

You can find the BuildStream project with all our .bst files and instructions on how to use it to generate an SD card image there – https://github.com/CodethinkLabs/cva6-sdk-bst.

QEMU model

In parallel, we developed a QEMU model that simulates the CVA6. It was useful during initial kernel bring up, because with QEMU we could iterate quicker. Also having a working QEMU model should help in future developments for CVA6. The patches for QEMU were submitted upstream – 1, 2.

Contributing back to cva6-sdk

We wanted to contribute something back to the cva6-sdk, so we replicated our kernel work there and also updated the software versions in their initrd. This is the merge request with the updates – https://github.com/openhwgroup/cva6-sdk/pull/109. Creating it also required replicating everything we've done with 64 bit CVA6 to 32 bit CVA6, as cva6-sdk is common for both widths – this replication was straightforward though.

An interesting fact is that during testing the updated cva6-sdk version with tetris we noticed that it always generated the same shapes. After some investigation, it was obvious that we had a bug in the floating point unit. We confirmed that it was present before our updates and was not caused by us. And then we discovered that it was just fixed in the master branch of CVA6. Regenerating the bitstream and using the latest CVA6 version fixed this bug with tetris.

Conclusion

We demonstrated how to boot Freedesktop SDK on CVA6, created QEMU model of it, and updated the kernel and initrd in cva6-sdk. This brings the software on CVA6 closer to upstream and makes experiments on it easier.

Potential future work includes updating U‑Boot version, debugging why some systemd services time out, rewriting the board's ethernet driver, and debugging unaligned memory access issue.

Are you looking for help with your RISC‑V projects? Get in touch with Codethink's team of experts by writing to us at connect@codethink.co.uk.