Yes, it’s here!
FreeBSD, and it’s downstream distributions (pfSense, OPNSense, FreeNAS/TrueNAS) have been the number one request we have had since first announcing the Ten64. I’m delighted to tell you it is now possible to run FreeBSD on a Ten64!
Dmitry Salychev has been working on a DPAA2 driver for a while now, and after a LOT of work it is now in a functional state. Given the complexity of DPAA2 this is no small feat!
The following functionality works in the current development branch:
- PCI Express including booting from NVMe drive
- USB3 including booting from a USB drive
- DPAA2 Network Interfaces in unmanaged (legacy) mode
Network interfaces will always appear as ‘UP’ regardless of physical status.
The following hardware features are currently not supported:
- DPAA2 Network interfaces in managed mode
- Full SFP support (e.g managing the SFP state, GPIOs, reading transceiver EEPROM and sensors) is likely to require further work as well
- SD card (clock driver issue, to debug)
- Real time clock (Epson RX-8035) - I have been working on a modified
ds1307(4)driver, but need to troubleshoot issues with FreeBSD’s I2C (vf_i2c) driver first. - Hardware sensors
- Including dynamic fan speed control
- GPIOs driven by the PCA954x expander (including the ADMIN LED/button, SFP and LTE/cellular modem control)
- DPAA2 Crypto/SEC engine (FreeBSD does have an armv8 crypto/AES driver, however)
This is a PREVIEW so there ARE bugs and performance issues
As the DPAA2 driver has not been optimized yet, as well as the FreeBSD kernel being compiled in debug (INVARIANTS) mode, performance isn’t as good as Linux. I am getting around 400mbps downstream from iperf3.
That said, so far the general FreeBSD experience on the Ten64 has proven to be surprisingly stable despite both the FreeBSD branch and driver not being ‘STABLE’. I have spent quite a bit of time over the past few weeks doing ‘shakedown’ testing and will continue to do so.
How do I try this?
Glad you asked! A FreeBSD preview image has been added to the appliance store, so you can do this in the recovery firmware to install it:
baremetal-deploy freebsd-dpaa2-preview /dev/nvme0n1
Alternatively, you can download and write this image (compressed qcow2, ~400MiB expands to 10GiB):
curl https://archive.traverse.com.au/pub/ports/freebsd/freebsd-14.0-CURRENT-ten64-2022-06-01.qcow2 -O
write-image-disk freebsd-14.0-CURRENT-ten64-2022-06-01.qcow2 /dev/nvme0n1
If installing manually, you must change the network mode to legacy:
# In recovery firmware
fw_setenv sfpmode legacy
fw_setenv gbemode legacy
reboot
# In U-Boot
setenv sfpmode legacy
setenv gbemode legacy
saveenv
reboot
Firstboot Instructions
On the first boot, FreeBSD will attempt to start DHCP on all interfaces.
As all network interfaces currently appear UP to FreeBSD (due to legacy mode), it will take a while for it to timeout.
# FreeBSD launching the async/background DHCP client
dpni0: link state changed to UP
# FreeBSD launching the normal (synchronous) DHCP client
Starting dhclient.
Use Ctrl-C to stop these requests and get to the login prompt faster.
Edit /etc/rc.conf to set the ifconfig for your preferred interface:
From:
ifconfig_DEFAULT="DHCP inet6 accept_rtadv"
to
ifconfig_dpni6="DHCP inet6 accept_rtadv"
Reboot or run /etc/netstart to make these changes effective.
DPNI’s are numbered the same as Linux, just with dpni instead of eth
Note: The kernel branch for DPAA2 is slightly behind the latest 14-CURRENT branch, so you may see kernel version warnings when attempting to install packages.
Known issues
- Multicast filters not configured, blocking the acquisition of IPv6 addreses
- Kernel panics under very heavy traffic
- Many warnings to dmesg under load (does not result in any packet loss)
What to do if you find a problem
Please check the DPAA2 driver issues on GitHub first.
When filing a bug report, please mention:
Hardware: Ten64 and/or VFIO
Kernel commit from uname -a:
FreeBSD ten64-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #1 173aa2a1fe: Tue May 17 21:13:05 UTC 2022 matt@freebsd:/usr/obj/usr/src/freebsd-src/arm64.aarch64/sys/TEN64-DEV arm64
In this case, the kernel commit is 173aa2a1fe
As this code is from the development branch of FreeBSD, please do not bother official FreeBSD support channels with any issues, unless you can confirm the issue exists on an official FreeBSD.org image. (Hint: Running FreeBSD as a VM on Linux is one such way to do this)
(Note: From time to time, my tree may diverge from Dmitry’s. My source branch is at GitHub - mcbridematt/freebsd-src at ten64 )
FAQ
Q: What does this mean for downstream projects like pfSense and OPNSense
It is still very early to tell, given this is being developed on the FreeBSD 14-CURRENT branch, so presumably one would need to wait until this driver is merged into FreeBSD and then those projects do a 14.0-STABLE release.
That said, someone (maybe you?) could try to compile a version of those projects with the DPAA2 driver…
Q: Does this work under VFIO (DPAA2 passthrough)?
Yes! (And special thanks to Dmitry for restructuring the driver slightly so it can work under the VFIO/QEMU ‘security model’)
Acknowledgements
Apart from the DPAA2 driver, running FreeBSD on the Ten64 has been made possible by previous work from: