FreeBSD preview for Ten64!

Yes, it’s here!

FreeBSD, and it’s downstream distributions (pfSense, OPNSense, FreeNAS/TrueNAS) have been the number one request we have had since first announcing the Ten64. I’m delighted to tell you it is now possible to run FreeBSD on a Ten64!

Dmitry Salychev has been working on a DPAA2 driver for a while now, and after a LOT of work it is now in a functional state. Given the complexity of DPAA2 this is no small feat!

The following functionality works in the current development branch:

  • PCI Express including booting from NVMe drive
  • USB3 including booting from a USB drive
  • DPAA2 Network Interfaces in unmanaged (legacy) mode
    Network interfaces will always appear as ‘UP’ regardless of physical status.

The following hardware features are currently not supported:

  • DPAA2 Network interfaces in managed mode
    • Full SFP support (e.g managing the SFP state, GPIOs, reading transceiver EEPROM and sensors) is likely to require further work as well
  • SD card (clock driver issue, to debug)
  • Real time clock (Epson RX-8035) - I have been working on a modified ds1307(4) driver, but need to troubleshoot issues with FreeBSD’s I2C (vf_i2c) driver first.
  • Hardware sensors
    • Including dynamic fan speed control
  • GPIOs driven by the PCA954x expander (including the ADMIN LED/button, SFP and LTE/cellular modem control)
  • DPAA2 Crypto/SEC engine (FreeBSD does have an armv8 crypto/AES driver, however)

This is a PREVIEW so there ARE bugs and performance issues

As the DPAA2 driver has not been optimized yet, as well as the FreeBSD kernel being compiled in debug (INVARIANTS) mode, performance isn’t as good as Linux. I am getting around 400mbps downstream from iperf3.

That said, so far the general FreeBSD experience on the Ten64 has proven to be surprisingly stable despite both the FreeBSD branch and driver not being ‘STABLE’. I have spent quite a bit of time over the past few weeks doing ‘shakedown’ testing and will continue to do so.

How do I try this?

Glad you asked! A FreeBSD preview image has been added to the appliance store, so you can do this in the recovery firmware to install it:

baremetal-deploy freebsd-dpaa2-preview /dev/nvme0n1

Alternatively, you can download and write this image (compressed qcow2, ~400MiB expands to 10GiB):

curl https://archive.traverse.com.au/pub/ports/freebsd/freebsd-14.0-CURRENT-ten64-2022-06-01.qcow2 -O
write-image-disk freebsd-14.0-CURRENT-ten64-2022-06-01.qcow2 /dev/nvme0n1

If installing manually, you must change the network mode to legacy:

# In recovery firmware
fw_setenv sfpmode legacy
fw_setenv gbemode legacy
reboot
# In U-Boot
setenv sfpmode legacy
setenv gbemode legacy
saveenv
reboot

Firstboot Instructions
On the first boot, FreeBSD will attempt to start DHCP on all interfaces.
As all network interfaces currently appear UP to FreeBSD (due to legacy mode), it will take a while for it to timeout.

# FreeBSD launching the async/background DHCP client
dpni0: link state changed to UP
# FreeBSD launching the normal (synchronous) DHCP client
Starting dhclient. 

Use Ctrl-C to stop these requests and get to the login prompt faster.

Edit /etc/rc.conf to set the ifconfig for your preferred interface:
From:

ifconfig_DEFAULT="DHCP inet6 accept_rtadv"

to

ifconfig_dpni6="DHCP inet6 accept_rtadv"

Reboot or run /etc/netstart to make these changes effective.

DPNI’s are numbered the same as Linux, just with dpni instead of eth

Note: The kernel branch for DPAA2 is slightly behind the latest 14-CURRENT branch, so you may see kernel version warnings when attempting to install packages.

Known issues

What to do if you find a problem
Please check the DPAA2 driver issues on GitHub first.

When filing a bug report, please mention:
Hardware: Ten64 and/or VFIO
Kernel commit from uname -a:

FreeBSD ten64-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #1 173aa2a1fe: Tue May 17 21:13:05 UTC 2022     matt@freebsd:/usr/obj/usr/src/freebsd-src/arm64.aarch64/sys/TEN64-DEV arm64

In this case, the kernel commit is 173aa2a1fe

As this code is from the development branch of FreeBSD, please do not bother official FreeBSD support channels with any issues, unless you can confirm the issue exists on an official FreeBSD.org image. (Hint: Running FreeBSD as a VM on Linux is one such way to do this)

(Note: From time to time, my tree may diverge from Dmitry’s. My source branch is at GitHub - mcbridematt/freebsd-src at ten64 )

FAQ
Q: What does this mean for downstream projects like pfSense and OPNSense

It is still very early to tell, given this is being developed on the FreeBSD 14-CURRENT branch, so presumably one would need to wait until this driver is merged into FreeBSD and then those projects do a 14.0-STABLE release.

That said, someone (maybe you?) could try to compile a version of those projects with the DPAA2 driver…

Q: Does this work under VFIO (DPAA2 passthrough)?
Yes! (And special thanks to Dmitry for restructuring the driver slightly so it can work under the VFIO/QEMU ‘security model’)

Acknowledgements
Apart from the DPAA2 driver, running FreeBSD on the Ten64 has been made possible by previous work from:

  • Semihalf sponsored by Alstom added basic Layerscape support first for the LS1046 and later the LS1028A
  • Bjoern A. Zeeb who worked on the MAC and PHY aspects of the DPAA2 driver.
  • Various other FreeBSD developers who have written patches for NXP Layerscape hardware
2 Likes

I’ll try to do my best to merge the DPAA2 drivers to the FreeBSD 14-CURRENT within two months. The only obstacle is to free resources correctly in order not to cause kernel panics when driver is compiled as a module and unloaded.

Btw, I’m looking forward to compile pfSense with the DPAA2 support, so any help is welcome.

1 Like

Got also a Ten64 board recently, It’s a great news, thanks for your hard work.

How can I help you (I have little knowledge on Freebsd side).

Just try to install this FreeBSD preview image and check how it’ll work for you and report bugs if you want to be involved a bit deeply, or wait till the new versions of pfSense or FreeNAS/TrueNAS with DPAA2 available.