Restool in muvirt?

Ten64 Forum Ten64 specific software (μVirt, recovery)
1

Hello. I’ve installed muvirt and looking to take advantage of DPAA2 for networking in the virtualized sessions. That being said, this page describes the restool utilities. I found restool booting openWRT, but not muvirt.

Digging around on the Traverse openWRT archive, I did find a restool package.

I did download and install it, but it does not seem to function? Any pointers would be appreciated.

muvirt 21.02-SNAPSHOT, r0+16209-104234e569

root@muvirt:/tmp# wget https://archive.traverse.com.au/pub/traverse/ls1088firmwa                                                         
re/openwrt/branches/rebase_20200519/178511079/packages/base/restool_LSDK-20.04-2                                                        _aarch64_generic.ipk                                                                                                                     
Downloading 'https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/rebase_20200519/178511079/packages/base/restoo'
Connecting to 209.141.40.10:443                                                                                                          
Writing to 'restool_LSDK-20.04-2_aarch64_generic.ipk'                                                                                    
restool_LSDK-20.04-2 100% |*******************************|   104k  0:00:00 ETA                                                          
Download completed (107487 bytes)                                                                                                        

root@muvirt:/tmp# opkg install ./restool_LSDK-20.04-2_aarch64_generic.ipk                                                                
Installing restool (LSDK-20.04-2) to root...                                                                                             
Configuring restool.                                                                                                                     

root@muvirt:/tmp# restool                                                                                                                
error: Did not find a device file                                                                                                        
Segmentation fault
2

It’s not in the current 5.10x based version as that kernel does not have support for it (see this topic for more info).

It probably won’t take much effort to backport the kernel patch and enable restool again - I’ll post an update when that happens.

1 Like
3

Matt, I did have a follow-on question. Running the existing muvirt (muvirt 21.02-SNAPSHOT, r0+16209-104234e569) baremetal, can I still use the DPAA functionality? I’ve noticed that there appears to be some configuration done for this - see dpni.* below.

root@muvirt:/# dmesg |grep eth
[ 0.000000] psci: probing for conduit method from DT.
[ 2.525849] fsl_dpaa2_eth dpni.9: Adding to iommu group 1
[ 2.570661] fsl_dpaa2_eth dpni.8: Adding to iommu group 1
[ 2.615437] fsl_dpaa2_eth dpni.7: Adding to iommu group 1
[ 2.660200] fsl_dpaa2_eth dpni.6: Adding to iommu group 1
[ 2.704949] fsl_dpaa2_eth dpni.5: Adding to iommu group 1
[ 2.749690] fsl_dpaa2_eth dpni.4: Adding to iommu group 1
[ 2.794446] fsl_dpaa2_eth dpni.3: Adding to iommu group 1
[ 2.839201] fsl_dpaa2_eth dpni.2: Adding to iommu group 1
[ 2.883956] fsl_dpaa2_eth dpni.1: Adding to iommu group 1
[ 2.928704] fsl_dpaa2_eth dpni.0: Adding to iommu group 1
[ 3.230881] fsl_dpaa2_eth dpni.9: Probed interface eth0
[ 3.279459] fsl_dpaa2_eth dpni.8: Probed interface eth1
[ 3.324982] fsl_dpaa2_eth dpni.7: Probed interface eth2
[ 3.370265] fsl_dpaa2_eth dpni.6: Probed interface eth3
[ 3.415302] fsl_dpaa2_eth dpni.5: Probed interface eth4
[ 3.460345] fsl_dpaa2_eth dpni.4: Probed interface eth5
[ 3.505406] fsl_dpaa2_eth dpni.3: Probed interface eth6
[ 3.550437] fsl_dpaa2_eth dpni.2: Probed interface eth7
[ 3.595502] fsl_dpaa2_eth dpni.1: Probed interface eth8
[ 3.641578] fsl_dpaa2_eth dpni.0: Probed interface eth9
[ 8.399937] usbcore: registered new interface driver cdc_ether
[ 10.864546] fsl_dpaa2_eth dpni.9 eth0: Link Event: state up
[ 10.874972] br-lan: port 1(eth0) entered blocking state
[ 10.880244] br-lan: port 1(eth0) entered disabled state
[ 10.887676] device eth0 entered promiscuous mode
[ 10.894423] br-lan: port 1(eth0) entered blocking state
[ 10.899671] br-lan: port 1(eth0) entered forwarding state
[ 10.911354] fsl_dpaa2_eth dpni.8 eth1: Link Event: state up

root@muvirt:/# dmesg |grep dprc
[ 0.420439] fsl_mc_dprc dprc.1: DMA mask not set
[ 1.694079] fsl_mc_dprc dprc.1: Adding to iommu group 1
[ 3.127886] fsl_mc_dprc dprc.1: DPRC device bound to driver
[ 8.143771] fsl_mc_dprc dprc.1 (unnamed net_device) (uninitialized): netif_napi_add() called with weight 512

Should I simply be able to plug in dpni.X or dprc.X into the virtual machine DPAA2 DPRC container configuration and have it work? I’d probably need an idiots guide to DPAA. My objective is to create a VM for Rockstor and take advantage of DPAA for the connection into the VM.

4

This looks like you are using the default dpl layout (eth-dpl-all.dts) that allocates resources in the container dprc.1 to use all dpmacs and individually connects them to one dpni each. From this starting point, you won’t have any resources left to define an additional dprc that can be passed into your VM. Thus, you have two options:

  1. Use restool to dynamically free some of the resources and insert them into a new dprc.
  2. create your own dpl that creates custom dprcs at boot time. This involves writing your own dts file, compiling it and writing it to the nand. The config for my use case (eth-dpl-vm-router-app.dts) and some notes (Readme) can be found here.

Once you have your additional dprc with the needed resources for the guest defined, you can pass them to the VM by adding list dpaa2container 'dprc.X'to your VM definition in /etc/config/virt. For reference, I post the definition of my testing VM below. Note how dprc.4 is passed to it (along with a pci and a usb device)

config vm 'testing'
    option type 'aarch64'
    option cpu 'host'
    option memory '1024'
    option numprocs '4'
    option mac '52:XX:XX:XX:XX:XX'
    option kernel '/dev/guests/router-kernel-test'
    option append 'earlycon root=/dev/vda console=ttyAMA0,115200 rootwait' ls1088-ext4-img'
    list disks '/dev/guests/router-test'
    option enable '1'
    option provisioned '1'
    option ballooning '1'
    list pcidevice '0002:01:00.0'
    list usbdevice 'vendor=3277,product=178'
    list dpaa2container 'dprc.4'
2 Likes
5

This is very helpful. I will give it a try later today.

6

^ Listen to @dubito - he got it working before I did :slight_smile:

A muvirt build with restool has just come out of “the oven”, download it from here.
I haven’t tested the DPAA2 passthrough just yet but the creation of DPAA2 objects works.

The “default” configuration is to have all 10 ethernet ports wired up in the “root” container (dprc.1).

To play with custom DPAA2 configurations you’ll need to start with the “blank” DPL (eth-dpl-eth0-only.dtb) as there is no “room” left in the default configuration to add anything.

Every VM (anything that crosses an IOMMU boundary) needs it’s own dprc container and objects.

Objects inside a child container can reference (be “linked” to) a parent container object which means you can do:

  • VM with access to “external” Ethernet port (DPNI->DPMAC)
  • VM with “back-to-back” connection either to host or another VM (DPNI->DPNI)
  • VM with connection to DPDMUX (Ethernet virtual bridge)

There is a bit of a pitfall at the moment in that DPAA2 “passthrough” only works when a device tree is used by the child VM, rather than ACPI (via EDKII) which is the default for muvirt.

Two ways around it are:

  • Supply your own compatible kernel image directly to QEMU so the guest kernel is booted directly, as @dubito has done above
  • Use u-boot rather than EDKII as the VM’s bootloader
    The DPDMUX page on the muvirt wiki and the dynamic DPDMUX tree enable this. It’s been a while since I worked on it, but having restool back removes a blocker to having this merged in.

The dpaa2.lua library I wrote for DPDMUX support might serve as a good reference on how to do certain tasks.

Some commands to get started:

$ restool dprc generate-dpl # Dump the data plane config out to a DTS which you can then compile into a .dtb and flash
# hint: the recovery firmware has restool and dtc (the compiler) so you can experiment there

$ ls-addni dpmac.3 # create an ethernet interface with GE4

# create a child DPRC, note the --options are important
restool dprc create dprc.1 --label="child VM dprc" \
--options=DPRC_CFG_OPT_IRQ_CFG_ALLOWED,DPRC_CFG_OPT_SPAWN_ALLOWED,DPRC_CFG_OPT_ALLOC_ALLOWED,DPRC_CFG_OPT_OBJ_CREATE_ALLOWED 
dprc.2 is created under dprc.1

# create DPNI under child dprc
# note that this by itself isn't enough to get an Ethernet interface working, you need other objects
# such as DPBP's,DPCONs,DPIO etc.
$ restool dpni create --container=dprc.2
dpni.4 is created under dprc.2

# connect child VM DPNI to dpmac.4 (host GE5), must be done from the top container
$ restool dprc connect dprc.1 --endpoint1=dpmac.4 --endpoint2=dpni.4

# "plug in" a DPNI (otherwise it won't be activated by the OS)
restool dprc assign dprc.2 --object=dpni.4 --plugged=1

# show contents of a DPRC
$ restool dprc show dprc.2
dprc.2 contains 1 objects:
object          label           plugged-state
dpni.4                          plugged
1 Like
7

@dubito I’ve looked at this on and off today but still am having some issues. I hope that the information below helps to show the error of my ways. My goal is to create a container (dprc.2) which will allow me to connect GE7 (dpmac.6) directly into a VM that I’ve created with OpenSUSE.

  1. I’ve flashed the system with eth-dpl-eth0-only.dtb

  2. When I boot muvirt (version from here: Index of /pub/traverse/software/muvirt/branches/master/358362925/image/)
    I see the following"

    root@muvirt:~# restool dprc info dprc.1
    container id: 1
    icid: 23
    portal id: 0
    dprc options: 0x4f
    DPRC_CFG_OPT_SPAWN_ALLOWED
    DPRC_CFG_OPT_ALLOC_ALLOWED
    DPRC_CFG_OPT_OBJ_CREATE_ALLOWED
    DPRC_CFG_OPT_TOPOLOGY_CHANGES_ALLOWED
    DPRC_CFG_OPT_IRQ_CFG_ALLOWED

    root@muvirt:~# restool dpni info dpni.0
    dpni version: 7.14
    dpni id: 0
    plugged state: plugged
    endpoint state: 1
    endpoint: dpmac.7, link is up
    link status: 1 - up
    mac address: 00:0a:XX:YY:PP:ZZ
    …
    …

Connectivity works as expected over br-lan (eth0).

  1. Next, I create a new DPAA container which I intend to use for a NAS VM.

root@muvirt:~# restool dprc create dprc.1 --label=“child VM dprc”
> --options=DPRC_CFG_OPT_IRQ_CFG_ALLOWED,DPRC_CFG_OPT_SPAWN_ALLOWED,DPRC_CFG_OPT_ALLOC_ALLOWED,DPRC_CFG_OPT_OBJ_CREATE_ALLOWED
dprc.2 is created under dprc.1

root@muvirt:~# restool dprc info dprc.2
container id: 2
icid: 24
portal id: 35
dprc options: 0x47
DPRC_CFG_OPT_SPAWN_ALLOWED
DPRC_CFG_OPT_ALLOC_ALLOWED
DPRC_CFG_OPT_OBJ_CREATE_ALLOWED
DPRC_CFG_OPT_IRQ_CFG_ALLOWED
object label: child VM dprc

  1. Create a dpni instance which I wish to map to GE7 (dpmac.6).

I create the additional resources here which I understood are required in the container. However, I run into errors when I try to create the dpio resource.

root@muvirt:~# restool dpni create --container=dprc.2
dpni.1 is created under dprc.2

root@muvirt:~# restool dprc connect dprc.1 --endpoint1=dpmac.6 --endpoint2=dpni.1

root@muvirt:~# restool dprc assign dprc.2 --object=dpni.1 --plugged=1

root@muvirt:~# restool dpbp create --container=dprc.2
dpbp.10 is created under dprc.2

root@muvirt:~# restool dpmcp create --container=dprc.2
dpmcp.36 is created under dprc.2

root@muvirt:~# restool dpcon create --container=dprc.2
dpcon.73 is created under dprc.2
root@muvirt:~# restool dpcon create --container=dprc.2
dpcon.74 is created under dprc.2
root@muvirt:~# restool dpcon create --container=dprc.2
dpcon.75 is created under dprc.2
root@muvirt:~# restool dpcon create --container=dprc.2
dpcon.76 is created under dprc.2

root@muvirt:~# restool dpio create --container=dprc.2
MC error: No resources (status 0x8)

This is the state of things at the end of the above sequence. Note that the majority of the resources are unplugged, etc. And I’ve tried to configure my VM to use dprc.2 but to no avail (as expected):

root@muvirt:~# restool dprc show dprc.2
dprc.2 contains 7 objects:
object label plugged-state
dpni.1 plugged
dpbp.10 unplugged
dpmcp.36 unplugged
dpcon.76 unplugged
dpcon.75 unplugged
dpcon.74 unplugged
dpcon.73 unplugged

root@muvirt:~# restool dpni info dpni.1
dpni version: 7.14
dpni id: 1
plugged state: plugged
endpoint state: 0
endpoint: dpmac.6, link is down
link status: 0 - down
mac address: 00:00:00:00:00:00
dpni_attr.options value is: 0
num_queues: 1
num_cgs: 1
num_rx_tcs: 1
num_tx_tcs: 1
mac_entries: 16
vlan_entries: 0
qos_entries: 0
fs_entries: 64
qos_key_size: 0
fs_key_size: 56

root@muvirt:~# restool dpmac info dpmac.6
dpmac version: 4.5
dpmac object id/portal id: 6
plugged state: plugged
endpoint state: 0
endpoint: dpni.1, link is down
DPMAC link type: DPMAC_LINK_TYPE_FIXED
DPMAC ethernet interface: DPMAC_ETH_IF_QSGMII
maximum supported rate 1000 Mbps
Counters:
rx all frames: 0
rx frames ok: 0
rx frame errors: 0
rx frame discards: 0
rx u-cast: 0
rx b-cast: 0
rx m-cast: 0
rx 64 bytes: 0

8 
restool dpio create --container=dprc.2
MC error: No resources (status 0x8)

This is failing because restool uses a default for “–num-priorities” that isn’t supported

(AFAIK it’s something to do with the wq_ch_conversion in the DPC, basically NXP suggests having more smaller channels is better than a couple of big ones)

You need to specify “–num-priorities=2” here

restool dpio create --container=dprc.2 --num-priorities=2

You also will want to have as many DPIOs as you do (v)CPUs/cores.

9

Matt, I’ve made some progress based upon the notes above. Here is where things stand now:

  1. I updated the device tree eth-dpl-eth0-only.dts to cut back on the number of resources being used. So I specifically halved the values for dpio dpmac, dpbp, etc. When I boot now, I see dprc.1 with the following configuration:

root@muvirt:~# restool dprc show dprc.1
dprc.1 contains 75 objects:
object label plugged-state
dpni.0 plugged
dpbp.4 plugged
dpbp.3 plugged
dpbp.2 plugged
dpbp.1 plugged
dpbp.0 plugged
dpmac.10 plugged
dpmac.9 plugged
dpmac.8 plugged
dpmac.7 plugged
dpmac.6 plugged
dpmac.5 plugged
dpmac.4 plugged
dpmac.3 plugged
dpmac.2 plugged
dpmac.1 plugged
dpmcp.17 plugged
dpmcp.16 plugged
dpmcp.15 plugged
dpmcp.14 plugged
dpmcp.13 plugged
dpmcp.12 plugged
dpmcp.11 plugged
dpmcp.10 plugged
dpmcp.9 plugged
dpmcp.8 plugged
dpmcp.7 plugged
dpmcp.6 plugged
dpmcp.5 plugged
dpmcp.4 plugged
dpmcp.3 plugged
dpmcp.2 plugged
dpmcp.1 plugged
dprtc.0 plugged
dpio.4 plugged
dpio.3 plugged
dpio.2 plugged
dpio.1 plugged
dpcon.36 plugged
dpcon.35 plugged
dpcon.34 plugged
dpcon.33 plugged
dpcon.32 plugged
dpcon.31 plugged
dpcon.30 plugged
dpcon.29 plugged
dpcon.28 plugged
dpcon.27 plugged
dpcon.26 plugged
dpcon.25 plugged
dpcon.24 plugged
dpcon.23 plugged
dpcon.22 plugged
dpcon.21 plugged
dpcon.20 plugged
dpcon.19 plugged
dpcon.18 plugged
dpcon.17 plugged
dpcon.16 plugged
dpcon.15 plugged
dpcon.14 plugged
dpcon.13 plugged
dpcon.12 plugged
dpcon.11 plugged
dpcon.10 plugged
dpcon.9 plugged
dpcon.8 plugged
dpcon.7 plugged
dpcon.6 plugged
dpcon.5 plugged
dpcon.4 plugged
dpcon.3 plugged
dpcon.2 plugged
dpcon.1 plugged
dpcon.0 plugged

2.Next, I create dprc.2

root@muvirt:~# restool dprc create dprc.1 --label=“child VM dprc”
–options=DPRC_CFG_OPT_IRQ_CFG_ALLOWED,DPRC_CFG_OPT_SPAWN_ALLOWED,DPRC_CFG_OPT_ALLOC_ALLOWE
D,DPRC_CFG_OPT_OBJ_CREATE_ALLOWED
dprc.2 is created under dprc.1

root@muvirt:~# restool dprc show dprc.1 |grep dprc
dprc.1 contains 76 objects:
dprc.2 child VM dprc unplugged

root@muvirt:~# restool dprc show dprc.2
dprc.2 contains 0 objects.
object label plugged-state

  1. Now I use the ls-addni wrapper to create dpni.1 in container dprc.2 which is mapped to dpmac.6 (GE7).

root@muvirt:~# ls-addni --container=dprc.2 --fs-entries=1 dpmac.6
Created interface: (object:dpni.1, endpoint: dpmac.6)

After the ls-addni command I see:

root@muvirt:~# restool dprc show dprc.2
dprc.2 contains 15 objects:
object label plugged-state
dpni.1 plugged
dpbp.5 plugged
dpmcp.19 plugged
dpio.7 plugged
dpio.6 plugged
dpio.5 plugged
dpio.0 plugged
dpcon.44 plugged
dpcon.43 plugged
dpcon.42 plugged
dpcon.41 plugged
dpcon.40 plugged
dpcon.39 plugged
dpcon.38 plugged
dpcon.37 plugged

  1. I assign a MAC address to dpni.1

root@muvirt:~# restool dpni update dpni.1 --mac-addr=00:XX:YY:ZZ:AA:20

On the surface things seem correct(?). However, when I check the status of dpni.1 it reports taht the link status is down.

root@muvirt:~# restool dpni info dpni.1
dpni version: 7.14
dpni id: 1
plugged state: plugged
endpoint state: 0
endpoint: dpmac.6, link is down
link status: 0 - down
mac address: 00:xx:yy:zz:aa:20
dpni_attr.options value is: 0
num_queues: 8
num_cgs: 1
num_rx_tcs: 1
num_tx_tcs: 1
mac_entries: 16
vlan_entries: 0
qos_entries: 0
fs_entries: 1
qos_key_size: 0
fs_key_size: 56
ingress_all_frames: 0
ingress_all_bytes: 0
ingress_multicast_frames: 0
ingress_multicast_bytes: 0
ingress_broadcast_frames: 0
ingress_broadcast_bytes: 0
egress_all_frames: 0
egress_all_bytes: 0
egress_multicast_frames: 0
egress_multicast_bytes: 0
egress_broadcast_frames: 0
egress_broadcast_bytes: 0
ingress_filtered_frames: 0
ingress_discarded_frames: 0
ingress_nobuffer_discards: 0
egress_discarded_frames: 0
egress_confirmed_frames: 0

So at this point I’m stuck again. I’ve tried to configure my openSUSE container with dprc.2 as the passthrough - although as openSUSE is EFI boot I know it’s a non starter in this config.

Interestingly, I’m no longer able to startup my openSUSE VM either (with LAN configured, dprc.2 unset). But I don’t want to cloud the discussions with that. But I think that the network changes did impact this.

10

Ah, I haven’t turned on DPAA2/MC VFIO (CONFIG_VFIO_FSL_MC) in the kernel, so no wonder it doesn’t go!

Bear with me while I compile a build and check it.

1 Like
11

This discussion triggered me to finally upgrade my system to the new 5.10 kernel (shame on me for still running 4.19). In my case there were two steps necessary to get it working.

  1. enable CONFIG_VFIO_FSL_MC in the kernel, as @mcbridematt mentioned

That left me with a guest system that had the dprc passed through and the NICs set up correctly. However, the NIC did not receive any data (i.e. number of RX packets was always zero). I knew that behavior from the 4.19 kernel and the fix back than required some more patches to be applied. Some of them are still part of the LSDK patches applied on top of linux 5.10. Thus the second step was:

  1. Applying the following LSDK patch: 83fc6196 virt/vgic: Increase number of DeviceIDs to 17

With that patch in place the dpaa2 passthrough is working fine (at least with my guest kernels that have a couple of patches that were required some time ago. I still need to check whether a vanilla mainline kernel is working as guest kernel as well)

1 Like
12

Thanks for that! I was trying to figure out what was missing but wasn’t looking in arch/arm64

Have you encountered anything like this on the host?

kvm [5743]: Unsupported FSC: EC=0x24 xFSC=0x21 ESR_EL2=0x92000061

It looks like an IOMMU fault by the guest.
There must be something missing from mainline kernels that causes it.

If I use a guest with an old LSDK (4.14) the passthrough works. I’m certain I had it working with the layerscape/LSDK based OpenWrt (5.4) as well, but those cause the problem too.
edit: Looks like the LSDK-21.08 (5.10) kernel works as guest

13

This is because the link up/down messages for the PHY aren’t being processed correctly when the controller (DPNI) isn’t attached to the Linux host. There is some combination of MC firmware and driver that will work for this, but not the not version currently distributed with Ten64’s.

You’ll need to switch to legacy network mode for now, or delete the mac@X {link_type = "MAC_LINK_TYPE_PHY";} in the DPC for the MAC you want to pass through.

A new build is available here with the latest patches mentioned above as well as providing a u-boot binary.

Use like so:

config vm 'debiandpaa2'
        option memory '1024'
        option numprocs '1'
        list disks '/dev/vmdata/debiandpaa2'
        option mac '52:54:00:87:18:f9'
        option provisioned '1'
        option dpaa2container 'dprc.2'
        option bios '/lib/muvirt/u-boot.bin'
        option enable '1'

The main problem now is finding a guest kernel that works under passthrough. I’ll package up NXP’s tree next week as that seems to work.

1 Like
14

Matt thanks. Just tinkering with the new muvirt build this AM. Seems that legacy mode was already set as the default on my board. Ultimately my goal is to get Rockstor running under muvirt with DPAA2. But it seems that the openSUSE 15.x builds on the store are UEFI only. Is there a build from openSUSE that you could recommend?

-Gabor

15

Yes, I know that error. It is an alignment error happening when the guest tries to access memory. It is fixed by (c315ddb2) soc: fsl: dpio: aligned access of qbman cacheable region

Indeed, that is one of four patches that I applied to the linux 5.4 guest kernels, I am currently running. The others are:

  1. Revert defered probing (5aa4277d and d7f5a9d8)
  2. bus: fsl-mc: DMA configure to have force-dma as default
  3. bus: fsl-mc: Extend ICID size from 16bit to 32bit

Unfortunately, I can’t find the latter two in the LSDK, but I still have the patches. Number three also appears in a patch series that prepared the mainline VFIO support. So it might be merged already. I will continue investigating as soon as I find the time…

In general, it is a bummer that patching the guest kernel is still necessary. Let’s hope the necessary patches will be upstreamed soon.

1 Like
16

Matt,

I’ve kind of fallen on my sword with DPAA2 at this stage. I’m starting to wonder what I’m doing wrong. I have a very rudimentary understanding of DPAA2 and I’m convinced I’m doing something very silly or wrong. Here is what I’ve tried as of this afternoon:

  1. Flashed eth-dpl-eth0-only.dts

  2. Ensured that the legacy modes are enabled (as per your recommendation)

  3. Boot latest muvirt build

muvirt 21.02-SNAPSHOT, r0+16214-8e81d977ac

  1. After bootup, I have a functioning eth0/br-lan (as expected).

  2. I’ve created a script to free up some of the resources in the default dprc.1. It frees up 4 instances of dpbp, 4 instances of dpmcp, 8 instances of dpcon and 4 instances of dpio.

  3. Create dprc.2

root@muvirt:~# restool dprc create dprc.1 --label=“child VM dprc”
–options=DPRC_CFG_OPT_IRQ_CFG_ALLOWED,DPRC_CFG_OPT_SPAWN_ALLOWED,DPRC_CFG_OPT_ALLOC_ALLOWED,DPRC_CFG_OPT_OBJ_CREATE_ALLOWED
dprc.2 is created under dprc.1

  1. Create dpni.1 linked to dpmac.6 using the wrapper. Note that dpmac.6 is defined in dprc.1.

root@muvirt:~# ls-addni --container=dprc.2 --fs-entries=1 dpmac.6
Created interface: (object:dpni.1, endpoint: dpmac.6)

  1. Check that dprc.2 is populated. Here we see the newly creted dpni.1, as plugged (along with all other resources):

root@muvirt:/etc/config# restool dprc show dprc.2
dprc.2 contains 15 objects:
object label plugged-state
dpni.1 plugged
dpbp.0 plugged
dpmcp.2 plugged
dpio.3 plugged
dpio.2 plugged
dpio.1 plugged
dpio.0 plugged
dpcon.7 plugged
dpcon.6 plugged
dpcon.5 plugged
dpcon.4 plugged
dpcon.3 plugged
dpcon.2 plugged
dpcon.1 plugged
dpcon.0 plugged

  1. I followed some instructions here, is in order to “Bind the newly created DPRC device to the vfio-fsl-mc driver”. I’m not sure if this is needed…

root@muvirt:~# echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
root@muvirt:~# echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

On the console I see:

[ 627.304652] vfio-fsl-mc dprc.2: DMA mask not set
[ 627.309408] vfio-fsl-mc dprc.2: Adding to iommu group 5
[ 627.324294] fsl-mc dpbp.0: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.332571] vfio-fsl-mc dpbp.0: Adding to iommu group 5
[ 627.338364] fsl-mc dpmcp.2: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.346741] vfio-fsl-mc dpmcp.2: Adding to iommu group 5
[ 627.352115] fsl-mc dpcon.7: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.360521] vfio-fsl-mc dpcon.7: Adding to iommu group 5
[ 627.365889] fsl-mc dpcon.6: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.374292] vfio-fsl-mc dpcon.6: Adding to iommu group 5
[ 627.379721] fsl-mc dpcon.5: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.388072] vfio-fsl-mc dpcon.5: Adding to iommu group 5
[ 627.393440] fsl-mc dpcon.4: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.401849] vfio-fsl-mc dpcon.4: Adding to iommu group 5
[ 627.407217] fsl-mc dpcon.3: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.415623] vfio-fsl-mc dpcon.3: Adding to iommu group 5
[ 627.420992] fsl-mc dpcon.2: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.429395] vfio-fsl-mc dpcon.2: Adding to iommu group 5
[ 627.434764] fsl-mc dpcon.1: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.443169] vfio-fsl-mc dpcon.1: Adding to iommu group 5
[ 627.448538] fsl-mc dpcon.0: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.456945] vfio-fsl-mc dpcon.0: Adding to iommu group 5
[ 627.462314] fsl-mc dpni.1: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.470631] vfio-fsl-mc dpni.1: Adding to iommu group 5
[ 627.476917] fsl-mc dpio.3: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.485210] vfio-fsl-mc dpio.3: Adding to iommu group 5
[ 627.491498] fsl-mc dpio.2: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.499792] vfio-fsl-mc dpio.2: Adding to iommu group 5
[ 627.506087] fsl-mc dpio.1: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.514347] vfio-fsl-mc dpio.1: Adding to iommu group 5
[ 627.520632] fsl-mc dpio.0: VFIO_FSL_MC: Setting driver override for device in dprc dprc.2
[ 627.528927] vfio-fsl-mc dpio.0: Adding to iommu group 5

  1. At this point, dpni.1 still shows the link down status (for endpoint dpmac.6 too).

root@muvirt:~# restool dpni info dpni.1
dpni version: 7.14
dpni id: 1
plugged state: plugged
endpoint state: 0
endpoint: dpmac.6, link is down
link status: 0 - down
mac address: 00:00:00:00:00:00
dpni_attr.options value is: 0
num_queues: 8
num_cgs: 1
num_rx_tcs: 1
num_tx_tcs: 1
mac_entries: 16
vlan_entries: 0
qos_entries: 0
fs_entries: 1
qos_key_size: 0
fs_key_size: 56
ingress_all_frames: 0
ingress_all_bytes: 0
ingress_multicast_frames: 0
ingress_multicast_bytes: 0
ingress_broadcast_frames: 0
ingress_broadcast_bytes: 0
egress_all_frames: 0
egress_all_bytes: 0
egress_multicast_frames: 0
egress_multicast_bytes: 0
egress_broadcast_frames: 0
egress_broadcast_bytes: 0
ingress_filtered_frames: 0
ingress_discarded_frames: 0
ingress_nobuffer_discards: 0
egress_discarded_frames: 0
egress_confirmed_frames: 0

  1. I’ve tried some of the ls1088 images from here but didn’t have any luck getting a VM booted. My /etc/config/virt file currently looks like (but I’ve tried other images):

config vm ‘dpaa2’
option memory ‘1024’
option numprocs ‘1’
list disks ‘/dev/vmdata/dpaa2’
option mac ‘52:ZZ:YY:87:XX:f9’
option provisioned ‘1’
option dpaa2container ‘dprc.2’
option bios ‘/mnt/muvirtwork/imglibrary/firmware_ls1088ardb_pb_uboot_sdboot.img’
option enable ‘1’

Any hints that you can provide would be greatly appreciated.

17

To clarify, when u-boot is used as the ‘BIOS’ for a VM, it can also boot operating systems that use EFI.
The capabilities are not as expansive as EDKII but it is good enough for many distributions.
So yes, use the openSUSE EFI images/appliances.

(You will need to build/install a patched kernel, otherwise the openSUSE VM will lockup due to the issues identified above)

Pretty much everything you have done up to the last step is correct. You need to use the /lib/muvirt/u-boot.bin as the “BIOS” binary. This is now standard in the latest builds.
This is compiled for the QEMU “virt” machine not any piece of real hardware

Sorry this isn’t very well documented up to now as there are many pieces that need to be in the correct place.

I have built new OpenWrt images (arm64_2102_rc4 branch) that have the “client” side patches @dubito identified (thank you very much!)

You can test it without permanent changes this way:

curl https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/arm64_2102_rc4/362209514/image/openwrt-arm64-efi-generic-ext4-combined.img.qcow2 -O

muvirt-createvm dpaa2test 512 1 lan $(readlink -f openwrt-arm64-efi-generic-ext4-combined.img.qcow2)

uci delete virt.dpaa2test.network # delete the connection to the OpenWrt lan bridge
uci set virt.dpaa2test.dpaa2container=dprc.2
uci set virt.dpaa2test.bios='/lib/muvirt/u-boot.bin'
uci commit virt
/etc/init.d/muvirt start dpaa2test

When inside the OpenWrt VM, it should have two Ethernet interfaces:

eth0 will be to QEMU’s “user” network that appears in the absence of any others [that QEMU knows about]
OpenWrt will set this up as a ‘LAN’, ignore it for now.
(A future change to muvirt will allow you to disable this)

eth1 should be the DPAA2 ethernet that has been passed through (should have the 00:0A:FA:… mac address assigned from host)
This won’t be setup as WAN in this image, but you can make it like so:

root@OpenWrt:/# uci set network.wan.device=eth1
root@OpenWrt:/# uci set network.wan.proto=dhcp
root@OpenWrt:/# ifup wan
root@OpenWrt:/# [   54.412303] fsl_dpaa2_eth dpni.2 eth1: Link Event: state down
[   54.449555] fsl_dpaa2_eth dpni.2 eth1: Link Event: state up
[   54.453565] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

I’ve posted my DPRC script as a github gist as a reference… will put it in the manual soon.

1 Like
18

Eureka! That worked! I’m very pleased to have made progress. :+1:

I did see eth1 as expected and your DPRC script made things much easier. My goal ultimately is to have Rockstor running under muvirt. I’ve used Rockstor on aarch64 previously (on a different board). If I understand correctly, the openSUSE 15.3 image in the “store” would need to have the appropriate client side patches added?

19

@dubito I’ve been tinkering with OpenSUSE 15.3 as a client running under muvirt. I’ve rebuilt the kernel with

  1. CONFIG_VFIO_FSL_MC enabled (building from kernel 5.3.18 as packaged in OpenSUSE kernel-source package).
  2. 83fc6196 virt/vgic: Increase number of DeviceIDs to 17

With this kernel, I see the Unsppported FSC message (my dprc.2 eth2 device appears in the VM, but can’t be activated). So I tried to apply the other patch(es) -

I notice that for patch c315ddb2 the source file has an old copyright from 2016. The version in the 5.3.18 kernel source has a much newer copyright (2016-2019) and appears to be totally refactored/reworked? Is there something I’ve misunderstood? I’m looking at the LSDK 19.09 guide now to see if there is some more detail there around kernel builds.

20

Have a look in here, these are the same as ones I used for the OpenWrt image.

There is a newer patch written by Diana Craciun (“soc: fsl: dpio: Fixed qbman alignment error in the
virtualization context”) which does the same thing.

It looks like openSUSE has backported changes from more recent kernel versions so despite it being labelled 5.3.x the driver code is closer to 5.10.

1 Like