Should I also be using a particular firmware?
Good question! I’m not aware of anything too different between the two firmware series that would affect VFIO.
I know the IOMMU initial configuration is done by U-Boot, so there could be something in there.
It’s worth trying the newest v0.9 firmware (v0.9.1) just to see if it makes a difference.
Another thing worth trying is to make vfio-pci bind to the card immediately at boot.
You can add vfio_pci.ids=14c3:7906 to the kernel command line (in /etc/default/grub → update-grub) to make the vfio-pci driver bind to it on probe.
dmesg | grep -E '(vfio|0001:03)'
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.1.76-traverse-lts-6-1+109-00021-g9889cb2828ba root=UUID=1bc72a0c-9d13-4a2d-86c3-a929bd2b637f ro earlycon arm-smmu.disable_bypass=0 net.ifnames=0 vfio-pci.disable_idle_d3=1 vfio_pci.ids=14c3:7906
[ 1.980544] pci 0001:03:00.0: [14c3:7906] type 00 class 0x028000
[ 1.980652] pci 0001:03:00.0: reg 0x10: [mem 0x2840000000-0x28400fffff 64bit pref]
[ 1.980727] pci 0001:03:00.0: reg 0x18: [mem 0x2840100000-0x2840107fff 64bit]
[ 1.980799] pci 0001:03:00.0: reg 0x20: [mem 0x2840108000-0x2840108fff 64bit pref]
[ 1.981175] pci 0001:03:00.0: PME# supported from D0 D3hot D3cold
[ 1.981411] pci 0001:03:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0001:00:00.0 (capable of 8.000 Gb/s with 5.0 GT/s PCIe x2 link)
[ 1.982190] pci 0001:03:00.0: BAR 0: assigned [mem 0x2840100000-0x28401fffff 64bit pref]
[ 1.982243] pci 0001:03:00.0: BAR 2: assigned [mem 0x2840000000-0x2840007fff 64bit]
[ 1.982295] pci 0001:03:00.0: BAR 4: assigned [mem 0x2840200000-0x2840200fff 64bit pref]
[ 4.463752] vfio-pci 0001:03:00.0: Adding to iommu group 1
[ 4.469584] vfio_pci: add [14c3:7906[ffffffff:ffffffff]] class 0x000000/00000000
If neither of those suggestions help, it would be good if you can upload the full contents of dmesg.
I just loaded the new firmware and vfio is working 
The IOMMU groups are numbered differnetly in this one.
Just got to get the OpenWRT vm guest configured.
I’ve got the dmesg output before the firmware upgrade, if you want a copy. Just tell me the best way to share it.
1 Like