Bringing up a 10G link between 2 Ten64s

Ten64 Forum
#1

I have 2 Ten64 boxes that I have updated to the latest firmware (v0.8.10 /2022-03-15) and I am trying to experiment with a direct 10G link between them. I have a 1 meter FS copper DAC cable that I have verified works on other 10G interfaces that I have here. I can see the cable being detected when I plug it in:

[ 1898.758155] sfp dpmac2_sfp: module FS               SFPP-PC01        rev P    sn F2032555161-1    dc 211228

I have configured the interface in /etc/config/network with:

config interface 'flan8'
	option device 'eth8'
	option proto 'static'
	option ipaddr '100.84.88.1'
	option netmask '255.255.255.0'

When I bring the interface up with ifup flan8 I see:

[ 4065.180405] fsl_dpaa2_eth dpni.1 eth8: configuring for inband/10gbase-r link mode

On the other Ten64 box, I have configured it the same way but with 100.84.88.2 as the IP address. I see the same messages about detecting the cable and configuring the link.

However, ethtool shows:

root@OpenWrt:/# ethtool eth8
Settings for eth8:
	Supported ports: [ ]
	Supported link modes:   1000baseX/Full 
	                        10000baseCR/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10000baseCR/Full 
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: Unknown!
	Duplex: Unknown! (255)
	Port: Direct Attach Copper
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	Link detected: no

I have tried bringing the interface up with ifconfig eth8 up and ifup flan8 to no avail.

Other commands show:

root@OpenWrt:/# ifconfig eth8
eth8      Link encap:Ethernet  HWaddr 00:0A:FA:24:25:2E  
          inet addr:100.84.88.1  Bcast:100.84.88.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@OpenWrt:/# ip link show eth8
10: eth8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:0a:fa:24:25:2e brd ff:ff:ff:ff:ff:ff

I have tried connecting one of the Ten64’s to a 10G switch and directly to a 10G Intel interface on one of my other machines and the carrier never comes up.

I figure I have forgotten something obvious.

#2

Can you see if changing to legacy SFP mode makes the link work? This bypasses the Linux SFP management and makes the link status based purely on the physical connection.

The other thing to look (in the normal mode) is the SFP status in debugfs:

# use dpmac2_sfp for the lower (XG0) port
cat /sys/kernel/debug/dpmac1-sfp/state
Module state: present
Module probe attempts: 0 0
Device state: up
Main state: link_up
Fault recovery remaining retries: 5
PHY probe remaining retries: 12
moddef0: 1
rx_los: 0
tx_fault: 0
tx_disable: 0
#3

I am still in normal mode. There is no /sys/kernel/debug/dpmac1-sfp directory. This is what I do see.

root@OpenWrt:/# ls -l /sys/kernel/debug
drwxr-xr-x    3 root     root             0 May 12 23:48 8000000.crypto
drwxr-xr-x   16 root     root             0 Jan  1  1970 bdi
drwxr-xr-x   14 root     root             0 Jan  1  1970 block
--w-------    1 root     root             0 Jan  1  1970 clear_warn_once
drwxr-xr-x   48 root     root             0 Jan  1  1970 clk
-rw-r--r--    1 root     root             0 Jan  1  1970 debug_enabled
drwxr-xr-x    2 root     root             0 Jan  1  1970 device_component
-r--r--r--    1 root     root             0 Jan  1  1970 devices_deferred
drwxr-xr-x    2 root     root             0 Jan  1  1970 dma_pools
drwxr-xr-x    2 root     root             0 Jan  1  1970 dmaengine
drwxr-xr-x   12 root     root             0 Jan  1  1970 dpaa2-eth
drwxr-xr-x    2 root     root             0 May 12 23:48 dpseci.0
drwxr-xr-x    2 root     root             0 Jan  1  1970 dynamic_debug
drwxr-xr-x    2 root     root             0 Jan  1  1970 extfrag
drwxr-xr-x    2 root     root             0 Jan  1  1970 f2fs
-rw-r--r--    1 root     root             0 Jan  1  1970 fault_around_bytes
-r--r--r--    1 root     root             0 Jan  1  1970 gpio
drwxr-xr-x    2 root     root             0 May 12 23:48 ieee80211
drwxr-xr-x    2 root     root             0 May 12 23:48 iwlwifi
drwxr-xr-x    2 root     root             0 Jan  1  1970 kvm
drwxr-xr-x    2 root     root             0 Jan  1  1970 memblock
drwxr-xr-x    2 root     root             0 Jan  1  1970 mmc0
drwxr-xr-x   13 root     root             0 Jan  1  1970 mtd
drwxr-xr-x    2 root     root             0 Jan  1  1970 pinctrl
drwxr-xr-x    2 root     root             0 Jan  1  1970 pm_genpd
drwxr-xr-x    3 root     root             0 Jan  1  1970 regmap
-r--r--r--    1 root     root             0 Jan  1  1970 sleep_time
drwxr-xr-x    2 root     root             0 Jan  1  1970 swiotlb
drwxr-xr-x    3 root     root             0 Jan  1  1970 ubi
drwxr-xr-x    3 root     root             0 Jan  1  1970 ubifs
drwxr-xr-x    5 root     root             0 Jan  1  1970 usb

root@OpenWrt:/# ls -l /sys/kernel/debug/dpaa2-eth
drwxr-xr-x    2 root     root             0 Jan  1  1970 eth0
drwxr-xr-x    2 root     root             0 Jan  1  1970 eth1
drwxr-xr-x    2 root     root             0 Jan  1  1970 eth2
drwxr-xr-x    2 root     root             0 Jan  1  1970 eth3
drwxr-xr-x    2 root     root             0 Jan  1  1970 eth4
drwxr-xr-x    2 root     root             0 Jan  1  1970 eth5
drwxr-xr-x    2 root     root             0 Jan  1  1970 eth6
drwxr-xr-x    2 root     root             0 Jan  1  1970 eth7
drwxr-xr-x    2 root     root             0 Jan  1  1970 eth8
drwxr-xr-x    2 root     root             0 Jan  1  1970 eth9

root@OpenWrt:/# ls -l /sys/kernel/debug/dpaa2-eth/eth8
-r--r--r--    1 root     root             0 Jan  1  1970 ch_stats
-r--r--r--    1 root     root             0 Jan  1  1970 cpu_stats
-r--r--r--    1 root     root             0 Jan  1  1970 fq_stats

root@OpenWrt:/# cat /sys/kernel/debug/dpaa2-eth/eth8/ch_stats
Channel stats for eth8:
CHID             CPU        Deq busy          Frames           CDANs    Avg Frm/CDAN       Buf count
   6               0               0               0               0               0            1281
   5               1               0               0               0               0            1281
   4               2               0               0               0               0            1281
   3               3               0               0               0               0            1281
   2               4               0               0               0               0            1281
   1               5               0               0               0               0            1281
  80               6               0               0               0               0            1281
  79               7               0               0               0               0            1281

root@OpenWrt:/# cat /sys/kernel/debug/dpaa2-eth/eth8/cpu_stats
Per-CPU stats for eth8
CPU              Rx          Rx Err           Rx SG              Tx          Tx Err         Tx conf           Tx SGTx converted to SG        Enq busy
  0               0               0               0               0               0               0               0               0               0
  1               0               0               0               0               0               0               0               0               0
  2               0               0               0               0               0               0               0               0               0
  3               0               0               0               0               0               0               0               0               0
  4               0               0               0               0               0               0               0               0               0
  5               0               0               0               0               0               0               0               0               0
  6               0               0               0               0               0               0               0               0               0
  7               0               0               0               0               0               0               0               0               0

root@OpenWrt:/# cat /sys/kernel/debug/dpaa2-eth/eth8/fq_stats
FQ stats for eth8:
VFQID             CPU              TC            Type          Frames  Pending frames

I’ll see what happens when I get some time to try out legacy mode.

#4

Apologies, the OpenWrt images didn’t have the debugfs feature enabled

These have it:
Index of /pub/traverse/ls1088firmware/openwrt/branches/arm64_2102/540621433/image/ (21.02)
Index of /pub/traverse/ls1088firmware/openwrt/branches/22_03/540640949/image/ (22.03)

Other things to check:

  1. Do you see Retimer: DS110DF410 found in the U-Boot console on startup?

Retimer: DS110DF410 found (cold boot)

or

Retimer: (retimer on, resetting...) DS110DF410 found (restart)

  1. If you insert the SFP at runtime, does it work?

We do have a microcontroller update (not published yet) which fixes startup issues for certain SFPs when present at power on or reboot. But we haven’t seen any of those issues with passive cables.

#5

I do see the Retimer line on a cold boot with the DAC plugged in:

Using SERDES1 Protocol: 29 (0x1d)
Using SERDES2 Protocol: 20 (0x14)
PCIe0: pcie@3400000 Root Complex: no link
PCIe1: pcie@3500000 Root Complex: x1 gen2
PCIe2: pcie@3600000 Root Complex: x2 gen3
Retimer: DS110DF410 found
OK

I see the exact same lines on a cold boot with the DAC removed.

When I insert the DAC after the boot has completed, I see on the console:

[  100.225389] sfp dpmac2_sfp: module FS               SFPP-PC01        rev P    sn Y2105169778-2    dc 210621

Checking the link with ip and ifconfig shows:

root@OpenWrt:/# ip link show eth8
10: eth8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:0a:fa:24:25:1a brd ff:ff:ff:ff:ff:ff

root@OpenWrt:/# ifconfig eth8
eth8      Link encap:Ethernet  HWaddr 00:0A:FA:24:25:1A  
          inet addr:100.84.88.2  Bcast:100.84.88.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Attempting to bring the link up:

root@OpenWrt:/# ifup flan8
[  355.843466] fsl_dpaa2_eth dpni.1 eth8: configuring for inband/10gbase-r link mode

root@OpenWrt:/# ip link show eth8
10: eth8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:0a:fa:24:25:1a brd ff:ff:ff:ff:ff:ff

root@OpenWrt:/# ethtool eth8
Settings for eth8:
	Supported ports: [ ]
	Supported link modes:   1000baseX/Full 
	                        10000baseCR/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10000baseCR/Full 
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: Unknown!
	Duplex: Unknown! (255)
	Port: Direct Attach Copper
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	Link detected: no

I’ll experiment with the firmware images you mentioned above and let you know how it goes.