Proxmox/PiMox installer for Ten64

Should I also be using a particular firmware?

Good question! I’m not aware of anything too different between the two firmware series that would affect VFIO.

I know the IOMMU initial configuration is done by U-Boot, so there could be something in there.
It’s worth trying the newest v0.9 firmware (v0.9.1) just to see if it makes a difference.

Another thing worth trying is to make vfio-pci bind to the card immediately at boot.
You can add vfio_pci.ids=14c3:7906 to the kernel command line (in /etc/default/grub → update-grub) to make the vfio-pci driver bind to it on probe.

dmesg | grep -E '(vfio|0001:03)'
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.1.76-traverse-lts-6-1+109-00021-g9889cb2828ba root=UUID=1bc72a0c-9d13-4a2d-86c3-a929bd2b637f ro earlycon arm-smmu.disable_bypass=0 net.ifnames=0 vfio-pci.disable_idle_d3=1 vfio_pci.ids=14c3:7906
[    1.980544] pci 0001:03:00.0: [14c3:7906] type 00 class 0x028000
[    1.980652] pci 0001:03:00.0: reg 0x10: [mem 0x2840000000-0x28400fffff 64bit pref]
[    1.980727] pci 0001:03:00.0: reg 0x18: [mem 0x2840100000-0x2840107fff 64bit]
[    1.980799] pci 0001:03:00.0: reg 0x20: [mem 0x2840108000-0x2840108fff 64bit pref]
[    1.981175] pci 0001:03:00.0: PME# supported from D0 D3hot D3cold
[    1.981411] pci 0001:03:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0001:00:00.0 (capable of 8.000 Gb/s with 5.0 GT/s PCIe x2 link)
[    1.982190] pci 0001:03:00.0: BAR 0: assigned [mem 0x2840100000-0x28401fffff 64bit pref]
[    1.982243] pci 0001:03:00.0: BAR 2: assigned [mem 0x2840000000-0x2840007fff 64bit]
[    1.982295] pci 0001:03:00.0: BAR 4: assigned [mem 0x2840200000-0x2840200fff 64bit pref]
[    4.463752] vfio-pci 0001:03:00.0: Adding to iommu group 1
[    4.469584] vfio_pci: add [14c3:7906[ffffffff:ffffffff]] class 0x000000/00000000

If neither of those suggestions help, it would be good if you can upload the full contents of dmesg.

I just loaded the new firmware and vfio is working :star_struck:

The IOMMU groups are numbered differnetly in this one.

Just got to get the OpenWRT vm guest configured.

I’ve got the dmesg output before the firmware upgrade, if you want a copy. Just tell me the best way to share it.

1 Like