How to compile DPDK and run dpdk-testpmd on Ten64?

piotr · July 20, 2023, 3:54pm

Hello!

I am working for quite a long time on running dpdk-testpmd on Ten64. As far the “best” results I got after:

→ Installing Ubuntu 22.04 using baremetal-deploy ubuntu-kinetic
→ Compiling restool tag LSDK-20.04
→ Using dynamic_dpl.sh from nxp’s dpdk tag LSDK-20.04
→ Modprobing vfio_iommu_type1 and vfio_fsl_mc
→ Using DPDK from apt as well as compiled one (LSDK-20.04)

After that I still have these results (the result is the same for apt’s DPDK and compiled from LSDK):

root@ten64:~# DPRC=dprc.2 dpdk-testpmd -- -i --portmask=1
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
fslmc: Skipping invalid device (power)
EAL: Selected IOVA mode 'PA'
EAL: No available 2048 kB hugepages reported
EAL: No available 32768 kB hugepages reported
EAL: No available 64 kB hugepages reported
EAL: VFIO support initialized
fslmc: Error mapping region (errno = 22)
fslmc: Unable to map MC Portal
fslmc: Unable to setup devices -1
testpmd: No probed ethernet devices
Interactive-mode selected
testpmd: create a new mbuf pool <mb_pool_0>: n=203456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Done
testpmd>

I think there should not be these lines (errno 22 is EINVAL):

fslmc: Error mapping region (errno = 22)
fslmc: Unable to map MC Portal
fslmc: Unable to setup devices -1
testpmd: No probed ethernet devices

Some more details below.

How did I compile restool:

git clone -bLSDK-20.04 -- https://github.com/nxp-qoriq/restool.git
cd -- restool
git config -- user.name user
git config -- user.email user@host
git cherry-pick -- 802764f8ed76f927dff494558332b0b77de7ac65
make -j8 -- CFLAGS='-std=gnu99 -Icommon'
sudo -- make -- install

How did I compile DPDK:

git clone -bLSDK-20.04 -- https://github.com/nxp-qoriq/dpdk.git
cd -- dpdk
CC=gcc-9 meson -Denable_kmods=false --prefix="$PWD/install" -- arm64-build
ninja -j8 -Carm64-build -- install

What is the /proc/cmdline:

default_hugepagesz=1024m hugepagesz=1024m hugepages=8 isolcpus=1-7 iommu.passthrough=1

What is the /proc/version:

Linux version 5.19.0-46-generic (buildd@bos02-arm64-080) (aarch64-linux-gnu-gcc-12 (Ubuntu 12.2.0-3ubuntu1) 12.2.0, GNU ld (GNU Binutils for Ubuntu) 2.39) #47-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 16 13:33:58 UTC 2023

How did I dynamic_dpl.py:

sudo -- env -- DPIO_COUNT=8 \
  bash -- /home/ubuntu/dpdk/nxp/dpaa2/dynamic_dpl.sh dpmac.1
# It went without errors/warnings.
# Without DPIO_COUNT it complains "No resources".

I strace’d the source of the error to mmap:

[pid  1947] openat(AT_FDCWD, "/dev/vfio/vfio", O_RDWR) = 42
[pid  1947] ioctl(42, VFIO_GET_API_VERSION, 0) = 0
[pid  1947] ioctl(42, VFIO_CHECK_EXTENSION, 0x1) = 1
[pid  1947] ioctl(42, VFIO_CHECK_EXTENSION, 0x7) = 0
[pid  1947] ioctl(42, VFIO_CHECK_EXTENSION, 0x8) = 0
[pid  1947] openat(AT_FDCWD, "/dev/vfio/4", O_RDWR) = 44
[pid  1947] ioctl(44, VFIO_GROUP_GET_STATUS, 0xffffc2504358) = 0
[pid  1947] ioctl(44, VFIO_GROUP_SET_CONTAINER, 0xffffa3490238) = -1 EINVAL (Invalid argument)
[pid  1947] openat(AT_FDCWD, "/dev/vfio/vfio", O_RDWR) = 45
[pid  1947] ioctl(45, VFIO_GET_API_VERSION, 0) = 0
[pid  1947] ioctl(45, VFIO_CHECK_EXTENSION, 0x1) = 1
[pid  1947] ioctl(45, VFIO_CHECK_EXTENSION, 0x7) = 0
[pid  1947] ioctl(45, VFIO_CHECK_EXTENSION, 0x8) = 0
[pid  1947] ioctl(45, VFIO_CHECK_EXTENSION, 0x1) = 1
[pid  1947] ioctl(44, VFIO_GROUP_SET_CONTAINER, 0xffffc2504340) = 0
[pid  1947] ioctl(45, VFIO_SET_IOMMU, 0x1) = 0
[pid  1947] ioctl(44, VFIO_GROUP_GET_DEVICE_FD, 0xaaab24c53310) = 46
[pid  1947] ioctl(45, VFIO_DEVICE_PCI_HOT_RESET or VFIO_IOMMU_MAP_DMA, 0xffffc25041f8) = 0
[pid  1947] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 46, 0x6030000) = -1 EINVAL (Invalid argument)
[pid  1947] readlinkat(AT_FDCWD, "/sys/bus/fsl-mc/devices/dpmcp.30/iommu_group", "../../../../../../../kernel/iomm"..., 4096) = 42
[pid  1947] openat(AT_FDCWD, "/dev/vfio/vfio", O_RDWR) = 47
[pid  1947] ioctl(47, VFIO_GET_API_VERSION, 0) = 0
[pid  1947] ioctl(47, VFIO_CHECK_EXTENSION, 0x1) = 1
[pid  1947] ioctl(47, VFIO_CHECK_EXTENSION, 0x7) = 0
[pid  1947] ioctl(47, VFIO_CHECK_EXTENSION, 0x8) = 0
[pid  1947] ioctl(44, VFIO_GROUP_GET_STATUS, 0xffffc25042a0) = 0
[pid  1947] ioctl(44, VFIO_GROUP_GET_DEVICE_FD, 0xaaab24b075e0) = 48
[pid  1947] ioctl(48, VFIO_DEVICE_GET_INFO, 0xffffc2504340) = 0
[pid  1947] ioctl(48, VFIO_DEVICE_GET_REGION_INFO, 0xffffc2504358) = 0
[pid  1947] mmap(NULL, 64, PROT_READ|PROT_WRITE, MAP_SHARED, 48, 0) = -1 EINVAL (Invalid argument)
[pid  1947] close(48)                   = 0
[pid  1947] write(2, "fslmc: Error mapping region (err"..., 41fslmc: Error mapping region (errno = 22)

I am a little bit tired of doing this, please support me

Regards

mcbridematt · July 21, 2023, 5:59am

Is this a Ubuntu provided kernel?
I think VFIO for DPAA2 is one of the few things still not complete in the mainline kernel.

I’ll spend some time next week checking, I have not done DPDK in a while.

NXP has now moved to a ‘Linux Factory’ versioning scheme (common with the i.MX family), if you look under tags for each repository, you will find a lf-6.X.X-y.y.y tag

LSDK 20.04 was for kernel 5.4 so it’s unlikely it’s DPDK will work on newer kernels

All the other setup looks correct.

piotr · July 21, 2023, 11:45am

Thanks @mcbridematt for your reply!

Is this a Ubuntu provided kernel?

Yes, it is the kernel installed by default after booting baremetal-deploy’ed image.

I’ve checked also other versions of DPDK including those lf-* tags - there are the same results, just different compiler warnings (I compile older versions using gcc-9)

I cannot find any instructions/blog that describes how to compile (and which versions) the DPDK on Ten64. The LSDK User Guide is quite helpful but it wants me to use flex-builder or flex-installer to build an image for LS1088ARDB (Reference Board) which won’t boot on Ten64.

Regards

mcbridematt · July 25, 2023, 7:09am

@piotr I was able to get dpdk-testpmd with DPDK 23.03 working today

See the first version of the DPDK HOWTO here:

https://ten64doc.traverse.com.au/software/dpdk/

I have only tested on Debian 12 so far, I will try Debian 11 and Ubuntu when I have some time.

You will need to use the kernel branch from NXP or 6.1 + patches at the bottom.

I am going to try OVS-DPDK and VPP as well.

piotr · July 25, 2023, 11:49am

@mcbridematt Fantastic!

It seems that my main problem was related to invalid kernel and dtb. I thought that mainline Linux 5.16+ is OK as well as I didn’t notice that I should change the dtb.

I will make some progress now, thanks!

I have few notes to the “DPDK on Ten64 Setup Guide (DRAFT)”:

→ Add build-essentials to the apt-get list
→ Add make modules_install before make install in kernel compilation (also it should be sudo’ed)
→ You also need to add ‘iommu.passthrough=1’ to GRUB_CMDLINE_LINUX_DEFAULT

I used dpdk from GitHub - nxp-qoriq/dpdk: Data Plane Development Kit - it has dynamic_dpl script

Thanks again!

piotr · August 1, 2023, 7:43am

Hi,

I finished the test I wanted to do. I can share some snippets, maybe someone will find it helpful.

It compiles and configures Linux, restool, DPDK and Open vSwitch.

Enjoy!

sudo -- eatmydata -- apt -y -- install \
  build-essential git bison flex bc libssl-dev pandoc meson pkg-config \
  python3-pyelftools libtool

$tag=lf-6.1.22-2.0.0

cd --
git clone -b"$tag" -- https://github.com/nxp-qoriq/linux
cd -- linux
make -j8 -- defconfig
make -j8 -- EXTRAVERSION=-altrax
sudo -- make -j8 -- modules_install
sudo -- make -j8 -- install

sudo -- patch -d/etc/default <<EOF
--- grub~
+++ grub
@@ -0,1 +0,1 @@
-GRUB_CMDLINE_LINUX_DEFAULT="earlycon arm-smmu.disable_bypass=0 net.ifnames=0"
+GRUB_CMDLINE_LINUX_DEFAULT="earlycon default_hugepagesz=1024m hugepagesz=1024m hugepages=8 isolcpus=1-7 iommu.passthrough=1 arm-smmu.disable_bypass=0 net.ifnames=0"
EOF
sudo -- update-grub
printf -- '%s\n' vfio-fsl-mc \
  | sudo -- tee -- /etc/modules-load.d/vfio-fsl-mc.conf

sudo -- reboot --

DPDK_INSTALL=/home/debian/dpdk/install
RESTOOL_INSTALL=/home/debian/restool/install
OVS_INSTALL=/home/debian/ovs-dpdk/install
declare -x -- \
  PATH="$DPDK_INSTALL/bin:$RESTOOL_INSTALL/bin:$OVS_INSTALL/bin:$OVS_INSTALL/sbin:$PATH" \
  LD_LIBRARY_PATH="$DPDK_INSTALL/lib/aarch64-linux-gnu" \
  PKG_CONFIG_PATH="$DPDK_INSTALL/lib/aarch64-linux-gnu/pkgconfig"

cd --
git clone -b"$tag" -- https://github.com/nxp-qoriq/restool.git
cd -- restool
make -j8 -- prefix="$PWD/install" \
  bindir_completion="$PWD/install/share/bash-completion/completions" \
  install

cd --
git clone -b"$tag" -- https://github.com/nxp-qoriq/dpdk.git
cd -- dpdk
meson --prefix="$PWD/install" -- arm64-build
ninja -j8 -Carm64-build -- install

cd --
git clone -b"$tag" -- https://github.com/nxp-qoriq/ovs-dpdk.git
cd -- ovs-dpdk
./boot.sh
mkdir -- build
cd -- build
../configure --prefix="$PWD/../install" --with-dpdk=shared
make -j8 -- install

ovsdb-tool create \
  "$OVS_INSTALL/etc/openvswitch/conf.db" \
  "$OVS_INSTALL/share/openvswitch/vswitch.ovsschema"
mkdir -p -- \
  "$OVS_INSTALL/var/run/openvswitch" \
  "$OVS_INSTALL/var/log/openvswitch"
ovsdb-server --detach --log-file --pidfile \
  --remote=punix:"$OVS_INSTALL/var/run/openvswitch/db.sock" \
  --remote=db:Open_vSwitch,Open_vSwitch,manager_options

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x1
ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0x7E
ovs-vsctl --no-wait set Open_vSwitch . other_config:emc-insert-inv-prob=1

bash -- "$DPDK_INSTALL/../nxp/dpaa2/dynamic_dpl.sh" \
  dpmac.1 dpmac.2 \
  dpmac.3 dpmac.4 \
  dpmac.5 dpmac.6
DPRC=dprc.2 ovs-vswitchd --detach --log-file --pidfile

ovs-vsctl -- del-br br0
ovs-vsctl \
  -- add-br br0 \
  -- set bridge br0 datapath_type=netdev
ovs-vsctl \
  -- add-port br0 xg0 \
  -- set Interface xg0 type=dpdk options:dpdk-devargs=dpni.1
ovs-vsctl \
  -- add-port br0 xg1 \
  -- set Interface xg1 type=dpdk options:dpdk-devargs=dpni.2
ovs-vsctl \
  -- add-port br0 ge4 \
  -- set Interface ge4 type=dpdk options:dpdk-devargs=dpni.3
ovs-vsctl \
  -- add-port br0 ge5 \
  -- set Interface ge5 type=dpdk options:dpdk-devargs=dpni.4
ovs-vsctl \
  -- add-port br0 ge6 \
  -- set Interface ge6 type=dpdk options:dpdk-devargs=dpni.5
ovs-vsctl \
  -- add-port br0 ge7 \
  -- set Interface ge7 type=dpdk options:dpdk-devargs=dpni.6
ip link set br0 up
gpioset 4 1=0 5=0

for p in xg0 xg1 ge4 ge5 ge6 ge7 ; do
  ovs-vsctl set Interface "$p" options:n_rxq_desc=1024
  ovs-vsctl set Interface "$p" options:n_rxq=1
  ovs-vsctl set Interface "$p" options:n_txq=1
  ovs-vsctl set Interface "$p" mtu_request=9600
done

# Here you can ovs-ofctl ...