Adding more SFP/SFP+ modules (optical)

Ten64 Forum
1

Hello.

As this is my first post, I’d like to thank Traverse for developing an interesting open source device, and I hope this tradition continues; for me, this is the main selling point of the device.

I want to use the SFP+ for both my local network and WAN connection. Sadly, I did not manage to use any of the SFP devices recommended to me by other user of the same ISP and the ISP representative; none of them work at all.

I’d love to know how to add them myself, if at all possible.

signal-2024-08-05-235635_002
signal-2024-08-05-235635_0021131×952 223 KB

For now, this is the log:

[ 779.345975] sfp dpmac2-sfp: module OEM SFP-10G-T-RJ45 rev sn 2024050200002 dc 240502
[ 779.385722] hwmon hwmon0: temp1_input not attached to any thermal zone
[ 839.913435] sfp dpmac2-sfp: module removed
[ 860.313433] sfp dpmac2-sfp: module TP-LINK TL-SM510U rev 2.0 sn 1237072000408 dc 230731
[ 885.151757] sfp dpmac2-sfp: module removed
[11450.412082] sfp dpmac1-sfp: module CTS INC. SFP-31W2BahSM-10 rev A sn 4EA9207F2274070 dc 231218
[11450.422035] fsl_dpaa2_eth dpni.0 eth9: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
[11450.451375] hwmon hwmon0: temp1_input not attached to any thermal zone
[11460.981580] sfp dpmac1-sfp: module OEM SFP-GE-LX-SM1550 rev A sn 20240530526 dc 240530
[11460.991534] fsl_dpaa2_eth dpni.0 eth9: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
[11461.021108] hwmon hwmon3: temp1_input not attached to any thermal zone

2

These look like Gigabit SFPs. You will need to change one or both of the SFP slots to 1G mode, with the method described in SFP#Using 1G SFP’s.

Due to limitations with how the Ethernet MACs and clocks work inside the Layerscape SoC’s, it is not possible to switch between 1<->10G mode at runtime :frowning:

3

I’m sorry for not clarifying this part previously, but I’m pretty sure I did follow that part of the documentation.

Specifically, I start with re-flashing from firmware-v0.8.10-492260694, using the command on the page.

I did not check the boot message, though, and I will do so later today; whether it appears or not changes little if I only follow the current docs here.

For the context, the installed OS, according to the admin, is: “LuCI openwrt-23.05 branch (git-24.154.44192-c70c82e) / OpenWrt 23.05-SNAPSHOT (r0-3fb23be)”.

root@OpenWrt:~# mtd erase bl2 && mtd write bl2_qspi_xg1_1g.pbl bl2
Unlocking bl2 ...
Erasing bl2 ...
Unlocking bl2 ...

Writing from bl2_qspi_xg1_1g.pbl to bl2 ...

Then I restart Ten 64 and add the fibre module one after another, to check “One SFP (top/XG1) in 1G mode”. I also look at logs in “Web admin” at the time.

Injecting the yellow module:

Tue Aug  6 06:00:30 2024 kern.info kernel: [  339.664840] sfp dpmac1-sfp: module OEM              SFP-GE-LX-SM1550 rev A    sn 20240530526      dc 240530
Tue Aug  6 06:00:30 2024 kern.err kernel: [  339.674792] fsl_dpaa2_eth dpni.0 eth9: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:00:30 2024 kern.info kernel: [  339.704053] hwmon hwmon2: temp1_input not attached to any thermal zone

Removing it, replacing with the blue one:

Tue Aug  6 06:04:33 2024 kern.info kernel: [  582.858410] sfp dpmac1-sfp: module OEM              SFP-GE-LX-SM1310 rev A    sn 202405300017     dc 240530
Tue Aug  6 06:04:33 2024 kern.err kernel: [  582.868361] fsl_dpaa2_eth dpni.0 eth9: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:04:33 2024 kern.info kernel: [  582.898286] hwmon hwmon3: temp1_input not attached to any thermal zone

And then, I am removing it and replacing with one of the two violets I have here:

Tue Aug  6 06:07:22 2024 kern.info kernel: [  751.264005] sfp dpmac1-sfp: module CTS INC.         SFP-31W2BahSM-10 rev A    sn 4EA9207F2274070  dc 231218
Tue Aug  6 06:07:22 2024 kern.err kernel: [  751.273961] fsl_dpaa2_eth dpni.0 eth9: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:07:22 2024 kern.info kernel: [  751.303779] hwmon hwmon4: temp1_input not attached to any thermal zone

Let’s try with another module of the same type for a good measure:

Tue Aug  6 06:09:39 2024 kern.info kernel: [  888.699822] sfp dpmac1-sfp: module CTS INC.         SFP-31W2BahSM-10 rev A    sn 4EA9207F2274073  dc 231218
Tue Aug  6 06:09:39 2024 kern.err kernel: [  888.709776] fsl_dpaa2_eth dpni.0 eth9: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:09:39 2024 kern.info kernel: [  888.749550] hwmon hwmon5: temp1_input not attached to any thermal zone

Now, let’s assume that for some reason, the SFP ports are reversed. What if I had to use the lower port, not the top port all the time? I will simply inject the lower port with all the same modules in the same order, then!

Tue Aug  6 06:12:30 2024 kern.info kernel: [ 1059.315712] sfp dpmac2-sfp: module OEM              SFP-GE-LX-SM1550 rev A    sn 20240530526      dc 240530
Tue Aug  6 06:12:30 2024 kern.err kernel: [ 1059.325664] fsl_dpaa2_eth dpni.1 eth8: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:12:30 2024 kern.info kernel: [ 1059.355072] hwmon hwmon6: temp1_input not attached to any thermal zone

Tue Aug  6 06:12:46 2024 kern.info kernel: [ 1075.504787] sfp dpmac2-sfp: module OEM              SFP-GE-LX-SM1310 rev A    sn 202405300017     dc 240530
Tue Aug  6 06:12:46 2024 kern.err kernel: [ 1075.514747] fsl_dpaa2_eth dpni.1 eth8: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:12:46 2024 kern.info kernel: [ 1075.544573] hwmon hwmon7: temp1_input not attached to any thermal zone

Tue Aug  6 06:13:05 2024 kern.info kernel: [ 1094.544625] sfp dpmac2-sfp: module CTS INC.         SFP-31W2BahSM-10 rev A    sn 4EA9207F2274070  dc 231218
Tue Aug  6 06:13:05 2024 kern.err kernel: [ 1094.554577] fsl_dpaa2_eth dpni.1 eth8: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:13:05 2024 kern.info kernel: [ 1094.584337] hwmon hwmon8: temp1_input not attached to any thermal zone

Tue Aug  6 06:13:20 2024 kern.info kernel: [ 1109.844194] sfp dpmac2-sfp: module CTS INC.         SFP-31W2BahSM-10 rev A    sn 4EA9207F2274073  dc 231218
Tue Aug  6 06:13:20 2024 kern.err kernel: [ 1109.854146] fsl_dpaa2_eth dpni.1 eth8: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:13:20 2024 kern.info kernel: [ 1109.884105] hwmon hwmon9: temp1_input not attached to any thermal zone

I’m not sure this provides enough information either, so I’ll also try ethtool.

I’ll take a violet module, both are the same, and inject it to test:

root@OpenWrt:~# ethtool -m eth9
	Identifier                                : 0x03 (SFP)
	Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
	Connector                                 : 0x07 (LC)
	Transceiver codes                         : 0x00 0x00 0x00 0x40 0x10 0x10 0x01 0x01 0x00
	Transceiver type                          : Ethernet: BASE-BX10
	Transceiver type                          : FC: long distance (L)
	Transceiver type                          : FC: Longwave laser (LL)
	Transceiver type                          : FC: Single Mode (SM)
	Transceiver type                          : FC: 100 MBytes/sec
	Encoding                                  : 0x01 (8B/10B)
	BR, Nominal                               : 1300MBd
	Rate identifier                           : 0x00 (unspecified)
	Length (SMF,km)                           : 10km
	Length (SMF)                              : 10000m
	Length (50um)                             : 0m
	Length (62.5um)                           : 0m
	Length (Copper)                           : 0m
	Length (OM3)                              : 0m
	Laser wavelength                          : 1490nm
	Vendor name                               : CTS INC.
	Vendor OUI                                : 00:90:65
	Vendor PN                                 : SFP-31W2BahSM-10
	Vendor rev                                : A
	Option values                             : 0x00 0x1a
	Option                                    : RX_LOS implemented
	Option                                    : TX_FAULT implemented
	Option                                    : TX_DISABLE implemented
	BR margin, max                            : 0%
	BR margin, min                            : 0%
	Vendor SN                                 : 4EA9207F2274073
	Date code                                 : 231218
	Optical diagnostics support               : Yes
	Laser bias current                        : 0.000 mA
	Laser output power                        : 0.0000 mW / -inf dBm
	Receiver signal average optical power     : 0.0001 mW / -40.00 dBm
	Module temperature                        : 23.18 degrees C / 73.72 degrees F
	Module voltage                            : 3.3000 V
	Alarm/warning flags implemented           : Yes
	Laser bias current high alarm             : Off
	Laser bias current low alarm              : Off
	Laser bias current high warning           : Off
	Laser bias current low warning            : On
	Laser output power high alarm             : Off
	Laser output power low alarm              : On
	Laser output power high warning           : Off
	Laser output power low warning            : On
	Module temperature high alarm             : Off
	Module temperature low alarm              : Off
	Module temperature high warning           : Off
	Module temperature low warning            : Off
	Module voltage high alarm                 : Off
	Module voltage low alarm                  : Off
	Module voltage high warning               : Off
	Module voltage low warning                : Off
	Laser rx power high alarm                 : Off
	Laser rx power low alarm                  : On
	Laser rx power high warning               : Off
	Laser rx power low warning                : On
	Laser bias current high alarm threshold   : 100.000 mA
	Laser bias current low alarm threshold    : 0.000 mA
	Laser bias current high warning threshold : 90.000 mA
	Laser bias current low warning threshold  : 0.100 mA
	Laser output power high alarm threshold   : 0.6310 mW / -2.00 dBm
	Laser output power low alarm threshold    : 0.1000 mW / -10.00 dBm
	Laser output power high warning threshold : 0.5012 mW / -3.00 dBm
	Laser output power low warning threshold  : 0.1259 mW / -9.00 dBm
	Module temperature high alarm threshold   : 90.00 degrees C / 194.00 degrees F
	Module temperature low alarm threshold    : -45.00 degrees C / -49.00 degrees F
	Module temperature high warning threshold : 85.00 degrees C / 185.00 degrees F
	Module temperature low warning threshold  : -40.00 degrees C / -40.00 degrees F
	Module voltage high alarm threshold       : 3.6000 V
	Module voltage low alarm threshold        : 3.0000 V
	Module voltage high warning threshold     : 3.5000 V
	Module voltage low warning threshold      : 3.1000 V
	Laser rx power high alarm threshold       : 1.0000 mW / 0.00 dBm
	Laser rx power low alarm threshold        : 0.0025 mW / -26.02 dBm
	Laser rx power high warning threshold     : 0.7943 mW / -1.00 dBm
	Laser rx power low warning threshold      : 0.0040 mW / -23.98 dBm

For the context, the logs are the same, too:

Tue Aug  6 06:25:11 2024 kern.info kernel: [ 1820.456126] sfp dpmac1-sfp: module CTS INC.         SFP-31W2BahSM-10 rev A    sn 4EA9207F2274073  dc 231218
Tue Aug  6 06:25:11 2024 kern.err kernel: [ 1820.466079] fsl_dpaa2_eth dpni.0 eth9: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:25:11 2024 kern.info kernel: [ 1820.505873] hwmon hwmon10: temp1_input not attached to any thermal zone

Now, let’s try the yellow one, too:

root@OpenWrt:~# ethtool -m eth9
	Identifier                                : 0x03 (SFP)
	Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
	Connector                                 : 0x07 (LC)
	Transceiver codes                         : 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x00 0x00
	Transceiver type                          : Ethernet: 1000BASE-LX
	Encoding                                  : 0x01 (8B/10B)
	BR, Nominal                               : 1300MBd
	Rate identifier                           : 0x00 (unspecified)
	Length (SMF,km)                           : 20km
	Length (SMF)                              : 20000m
	Length (50um)                             : 0m
	Length (62.5um)                           : 0m
	Length (Copper)                           : 0m
	Length (OM3)                              : 0m
	Laser wavelength                          : 1550nm
	Vendor name                               : OEM
	Vendor OUI                                : 00:90:65
	Vendor PN                                 : SFP-GE-LX-SM1550
	Vendor rev                                : A
	Option values                             : 0x00 0x1a
	Option                                    : RX_LOS implemented
	Option                                    : TX_FAULT implemented
	Option                                    : TX_DISABLE implemented
	BR margin, max                            : 0%
	BR margin, min                            : 0%
	Vendor SN                                 : 20240530526
	Date code                                 : 240530
	Optical diagnostics support               : Yes
	Laser bias current                        : 0.000 mA
	Laser output power                        : 0.0000 mW / -inf dBm
	Receiver signal average optical power     : 0.0000 mW / -inf dBm
	Module temperature                        : 25.69 degrees C / 78.24 degrees F
	Module voltage                            : 3.2976 V
	Alarm/warning flags implemented           : Yes
	Laser bias current high alarm             : Off
	Laser bias current low alarm              : Off
	Laser bias current high warning           : Off
	Laser bias current low warning            : Off
	Laser output power high alarm             : Off
	Laser output power low alarm              : Off
	Laser output power high warning           : Off
	Laser output power low warning            : Off
	Module temperature high alarm             : Off
	Module temperature low alarm              : Off
	Module temperature high warning           : Off
	Module temperature low warning            : Off
	Module voltage high alarm                 : Off
	Module voltage low alarm                  : Off
	Module voltage high warning               : Off
	Module voltage low warning                : Off
	Laser rx power high alarm                 : Off
	Laser rx power low alarm                  : On
	Laser rx power high warning               : Off
	Laser rx power low warning                : On
	Laser bias current high alarm threshold   : 75.000 mA
	Laser bias current low alarm threshold    : 3.000 mA
	Laser bias current high warning threshold : 70.000 mA
	Laser bias current low warning threshold  : 4.000 mA
	Laser output power high alarm threshold   : 0.6309 mW / -2.00 dBm
	Laser output power low alarm threshold    : 0.1001 mW / -10.00 dBm
	Laser output power high warning threshold : 0.5012 mW / -3.00 dBm
	Laser output power low warning threshold  : 0.1259 mW / -9.00 dBm
	Module temperature high alarm threshold   : 90.00 degrees C / 194.00 degrees F
	Module temperature low alarm threshold    : -45.00 degrees C / -49.00 degrees F
	Module temperature high warning threshold : 85.00 degrees C / 185.00 degrees F
	Module temperature low warning threshold  : -40.00 degrees C / -40.00 degrees F
	Module voltage high alarm threshold       : 3.6000 V
	Module voltage low alarm threshold        : 3.0000 V
	Module voltage high warning threshold     : 3.5000 V
	Module voltage low warning threshold      : 3.0999 V
	Laser rx power high alarm threshold       : 0.6309 mW / -2.00 dBm
	Laser rx power low alarm threshold        : 0.0032 mW / -24.95 dBm
	Laser rx power high warning threshold     : 0.5011 mW / -3.00 dBm
	Laser rx power low warning threshold      : 0.0040 mW / -23.98 dBm

Logs for the yellow module:

Tue Aug  6 06:28:46 2024 kern.info kernel: [ 2035.779098] sfp dpmac1-sfp: module OEM              SFP-GE-LX-SM1550 rev A    sn 20240530526      dc 240530
Tue Aug  6 06:28:46 2024 kern.err kernel: [ 2035.789054] fsl_dpaa2_eth dpni.0 eth9: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:28:46 2024 kern.info kernel: [ 2035.828846] hwmon hwmon11: temp1_input not attached to any thermal zone

Finally, I’ll check the blue module as well:

root@OpenWrt:~# ethtool -m eth9
	Identifier                                : 0x03 (SFP)
	Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
	Connector                                 : 0x07 (LC)
	Transceiver codes                         : 0x00 0x00 0x00 0x02 0x00 0x00 0x00 0x00 0x00
	Transceiver type                          : Ethernet: 1000BASE-LX
	Encoding                                  : 0x01 (8B/10B)
	BR, Nominal                               : 1300MBd
	Rate identifier                           : 0x00 (unspecified)
	Length (SMF,km)                           : 20km
	Length (SMF)                              : 20000m
	Length (50um)                             : 0m
	Length (62.5um)                           : 0m
	Length (Copper)                           : 0m
	Length (OM3)                              : 0m
	Laser wavelength                          : 1310nm
	Vendor name                               : OEM
	Vendor OUI                                : 00:90:65
	Vendor PN                                 : SFP-GE-LX-SM1310
	Vendor rev                                : A
	Option values                             : 0x00 0x1a
	Option                                    : RX_LOS implemented
	Option                                    : TX_FAULT implemented
	Option                                    : TX_DISABLE implemented
	BR margin, max                            : 0%
	BR margin, min                            : 0%
	Vendor SN                                 : 202405300017
	Date code                                 : 240530
	Optical diagnostics support               : Yes
	Laser bias current                        : 0.000 mA
	Laser output power                        : 0.0000 mW / -inf dBm
	Receiver signal average optical power     : 0.0000 mW / -inf dBm
	Module temperature                        : 24.65 degrees C / 76.37 degrees F
	Module voltage                            : 3.3103 V
	Alarm/warning flags implemented           : Yes
	Laser bias current high alarm             : Off
	Laser bias current low alarm              : Off
	Laser bias current high warning           : Off
	Laser bias current low warning            : Off
	Laser output power high alarm             : Off
	Laser output power low alarm              : Off
	Laser output power high warning           : Off
	Laser output power low warning            : Off
	Module temperature high alarm             : Off
	Module temperature low alarm              : Off
	Module temperature high warning           : Off
	Module temperature low warning            : Off
	Module voltage high alarm                 : Off
	Module voltage low alarm                  : Off
	Module voltage high warning               : Off
	Module voltage low warning                : Off
	Laser rx power high alarm                 : Off
	Laser rx power low alarm                  : On
	Laser rx power high warning               : Off
	Laser rx power low warning                : On
	Laser bias current high alarm threshold   : 75.000 mA
	Laser bias current low alarm threshold    : 3.000 mA
	Laser bias current high warning threshold : 70.000 mA
	Laser bias current low warning threshold  : 4.000 mA
	Laser output power high alarm threshold   : 0.6309 mW / -2.00 dBm
	Laser output power low alarm threshold    : 0.1001 mW / -10.00 dBm
	Laser output power high warning threshold : 0.5012 mW / -3.00 dBm
	Laser output power low warning threshold  : 0.1259 mW / -9.00 dBm
	Module temperature high alarm threshold   : 90.00 degrees C / 194.00 degrees F
	Module temperature low alarm threshold    : -45.00 degrees C / -49.00 degrees F
	Module temperature high warning threshold : 85.00 degrees C / 185.00 degrees F
	Module temperature low warning threshold  : -40.00 degrees C / -40.00 degrees F
	Module voltage high alarm threshold       : 3.6000 V
	Module voltage low alarm threshold        : 3.0000 V
	Module voltage high warning threshold     : 3.5000 V
	Module voltage low warning threshold      : 3.0999 V
	Laser rx power high alarm threshold       : 0.6309 mW / -2.00 dBm
	Laser rx power low alarm threshold        : 0.0032 mW / -24.95 dBm
	Laser rx power high warning threshold     : 0.5011 mW / -3.00 dBm
	Laser rx power low warning threshold      : 0.0040 mW / -23.98 dBm

Its logs:

Tue Aug  6 06:33:28 2024 kern.info kernel: [ 2317.171741] sfp dpmac1-sfp: module OEM              SFP-GE-LX-SM1310 rev A    sn 202405300017     dc 240530
Tue Aug  6 06:33:28 2024 kern.err kernel: [ 2317.181695] fsl_dpaa2_eth dpni.0 eth9: validation of inband/1000base-x with support 0000000,00000200,00006440 failed: -22
Tue Aug  6 06:33:28 2024 kern.info kernel: [ 2317.211738] hwmon hwmon12: temp1_input not attached to any thermal zone
4

It sounds like you’ve done everything correctly.

The only thing I can think of is that if you ran the firmware flasher again, you will need rewrite the bl2_qspi_xg1_1g.pbl to flash afterwards. The flash script will replace the bl2 binary to the default (10G/10G).

The validation of inband/1000base-x message means the speed/mode the SFP wants (1000base-x) does not match the bitmask of modes that the interface is set up for (on the right).

This is what happens in the reverse situation (10G SFP into a 1G slot):

validation of inband/10gbase-r with support 0000000,00001000,00006440 failed: -22

From the output above, it looks like the interface is still set to 10G.

5

So, the complete re-flash from the recovery mode and rewriting the bl2_qspi_xg1_1g.pbl as you told me helped on the kernel side. There’s no error anymore, yet there seem to be quirks, probably on the software side.

According to the docs, the modules should be enabled on OpenWRT, but they aren’t sending data, even if they’re now recognized by Ten64.

I’ve tried to enable them manually according to the doc, and that fails.

root@OpenWrt:~# echo 369 > /sys/class/gpio/export
ash: write error: Resource busy
root@OpenWrt:~# echo out > /sys/class/gpio/gpio369/direction
-ash: can't create /sys/class/gpio/gpio369/direction: nonexistent directory
root@OpenWrt:~# 
root@OpenWrt:~# echo 0 > /sys/class/gpio/gpio369/value
-ash: can't create /sys/class/gpio/gpio369/value: nonexistent directory

Things in the log seem to be going correctly for the modules the provider recommends.

Tue Aug  6 08:21:56 2024 kern.info kernel: [ 1917.178783] sfp dpmac1_sfp: module CTS INC.         SFP-31W2BahSM-10 rev A    sn 4EA9207F2274070  dc 231218
Tue Aug  6 08:21:56 2024 kern.info kernel: [ 1917.228338] hwmon hwmon2: temp1_input not attached to any thermal zone
...
Tue Aug  6 08:22:05 2024 kern.info kernel: [ 1926.468614] sfp dpmac2_sfp: module CTS INC.         SFP-31W2BahSM-10 rev A    sn 4EA9207F2274073  dc 231218
Tue Aug  6 08:22:05 2024 kern.info kernel: [ 1926.507924] hwmon hwmon3: temp1_input not attached to any thermal zone

Now I am conducting tests on laptop Ethernet with a simple media converter and a couple module pair types.

First, if I connect the 2nd media converter to the Ethernet, things are good: 923.64 Mbps download and 102.08 Mbps upload.

Next, I move the SFP module to a Ten64 SFP module port and leave one media converter at the laptop. Sadly, there’s no connection without WiFi.

Tue Aug  6 08:57:22 2024 kern.info kernel: [ 4043.394321] sfp dpmac1_sfp: module OEM              SFP-BX-D20D      rev      sn 202312220201     dc 231222
Tue Aug  6 08:57:22 2024 kern.info kernel: [ 4043.433674] hwmon hwmon2: temp1_input not attached to any thermal zone

root@OpenWrt:~# ethtool -m eth9
	Identifier                                : 0x03 (SFP)
	Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
	Connector                                 : 0x01 (SC)
	Transceiver codes                         : 0x00 0x00 0x00 0x00 0x12 0x00 0x01 0x01 0x00
	Transceiver type                          : FC: long distance (L)
	Transceiver type                          : FC: Longwave laser (LC)
	Transceiver type                          : FC: Single Mode (SM)
	Transceiver type                          : FC: 100 MBytes/sec
	Encoding                                  : 0x01 (8B/10B)
	BR, Nominal                               : 1300MBd
	Rate identifier                           : 0x00 (unspecified)
	Length (SMF,km)                           : 20km
	Length (SMF)                              : 20000m
	Length (50um)                             : 0m
	Length (62.5um)                           : 0m
	Length (Copper)                           : 0m
	Length (OM3)                              : 0m
	Laser wavelength                          : 1550nm
	Vendor name                               : OEM
	Vendor OUI                                : 00:00:00
	Vendor PN                                 : SFP-BX-D20D
	Vendor rev                                : 
	Option values                             : 0x00 0x1a
	Option                                    : RX_LOS implemented
	Option                                    : TX_FAULT implemented
	Option                                    : TX_DISABLE implemented
	BR margin, max                            : 0%
	BR margin, min                            : 0%
	Vendor SN                                 : 202312220201
	Date code                                 : 231222
	Optical diagnostics support               : Yes
	Laser bias current                        : 0.000 mA
	Laser output power                        : 0.0073 mW / -21.37 dBm
	Receiver signal average optical power     : 0.1980 mW / -7.03 dBm
	Module temperature                        : 27.82 degrees C / 82.08 degrees F
	Module voltage                            : 3.2863 V
	Alarm/warning flags implemented           : Yes
	Laser bias current high alarm             : Off
	Laser bias current low alarm              : On
	Laser bias current high warning           : Off
	Laser bias current low warning            : On
	Laser output power high alarm             : Off
	Laser output power low alarm              : On
	Laser output power high warning           : Off
	Laser output power low warning            : On
	Module temperature high alarm             : Off
	Module temperature low alarm              : Off
	Module temperature high warning           : Off
	Module temperature low warning            : Off
	Module voltage high alarm                 : Off
	Module voltage low alarm                  : Off
	Module voltage high warning               : Off
	Module voltage low warning                : Off
	Laser rx power high alarm                 : Off
	Laser rx power low alarm                  : Off
	Laser rx power high warning               : Off
	Laser rx power low warning                : Off
	Laser bias current high alarm threshold   : 100.000 mA
	Laser bias current low alarm threshold    : 1.000 mA
	Laser bias current high warning threshold : 80.000 mA
	Laser bias current low warning threshold  : 2.000 mA
	Laser output power high alarm threshold   : 0.6310 mW / -2.00 dBm
	Laser output power low alarm threshold    : 0.1259 mW / -9.00 dBm
	Laser output power high warning threshold : 0.5012 mW / -3.00 dBm
	Laser output power low warning threshold  : 0.1585 mW / -8.00 dBm
	Module temperature high alarm threshold   : 80.00 degrees C / 176.00 degrees F
	Module temperature low alarm threshold    : -10.00 degrees C / 14.00 degrees F
	Module temperature high warning threshold : 70.00 degrees C / 158.00 degrees F
	Module temperature low warning threshold  : 0.00 degrees C / 32.00 degrees F
	Module voltage high alarm threshold       : 3.6000 V
	Module voltage low alarm threshold        : 3.0000 V
	Module voltage high warning threshold     : 3.4500 V
	Module voltage low warning threshold      : 3.1500 V
	Laser rx power high alarm threshold       : 0.6310 mW / -2.00 dBm
	Laser rx power low alarm threshold        : 0.0016 mW / -27.96 dBm
	Laser rx power high warning threshold     : 0.5012 mW / -3.00 dBm
	Laser rx power low warning threshold      : 0.0025 mW / -26.02 dBm

I switch the module.

Tue Aug  6 09:11:03 2024 kern.info kernel: [ 4864.412197] sfp dpmac2_sfp: module CISCO-JT         GLC-BX-U         rev V1.0 sn GLC2407051233    dc 20240705
Tue Aug  6 09:11:03 2024 kern.info kernel: [ 4864.451553] hwmon hwmon3: temp1_input not attached to any thermal zone

It is not Cisco, it’s JT-COM ones from the tray. There’s no connection on the port. Putting the SFP into the media converter, I see 915.87 Mbps download and 102.12 Mbps upload on speedtest.net.

I’m trying another one:

Tue Aug  6 09:30:24 2024 kern.info kernel: [ 6024.814531] sfp dpmac1_sfp: module OEM              SFP-OC24-20B-SC  rev A    sn XC2501240162     dc 240125
Tue Aug  6 09:30:24 2024 kern.info kernel: [ 6024.853905] hwmon hwmon2: temp1_input not attached to any thermal zone

No luck with Ten 64, but with an ONTi media converter it is 891.64 Mbps DL, 102.16 Mbps UP.

I’ll continue trying different modules in general, but for now, I’d like to know whether I can expect one of the FS.com modules (#75336, #37925, #11795) to be stable as I need one of them (or the CTS INC. SFP-31W2BahSM-10 rev A) to connect to the ISP.

As a small note, I think this is starting to remind me of another topic I’ve opened symptom-wise: 10G Ethernet modules: recognized, but not connecting. The module is recognized, but no Ethernet connection is being made.

(Knowing whether I can somehow enable the modules shown previously would be nice, too! They may have some incorrect marking, yet they are compatible with hardware I have, so there probably is an override.)

Again, thanks for a quick reply! :slight_smile:

6

To clarify, when the Linux kernel is managing the SFP state, it will have control of the SFP’s GPIO/control lines, so you cannot control them from userspace.

What you can try looking at the information in debugfs:

# dpmac2_sfp = eth8/XG0, dpmac1_sfp=eth9/XG1
$ cat /sys/kernel/debug/dpmac2_sfp/state 
Module state: present
Module probe attempts: 0 0
Device state: up
Main state: link_up
Fault recovery remaining retries: 5
PHY probe remaining retries: 12
Signalling rate: 10313 kBd
Rate select threshold: 0 kBd
moddef0: 1
rx_los: 0
tx_fault: 0
tx_disable: 0
rs0: 0
rs1: 0

You could also try changing to “legacy” mode and controlling the SFP TX_ENABLE (gpio 369) directly. If you do that, the ‘up’ status of the interface (eth8/eth9) is directly linked to the electrical status of the connection. It could be that a ‘qurik’ needs to be implemented in the Linux SFP code for some of your modules.

I have not tried those FS modules, but I do have both 1 and 10G BiDis from 10Gtek which I have tested successfully:

7

After a quick test, the OpenWRT settings mentioned here in the documentation do not matter here.

lower_sfp_txidsable and lower_sfp_txidsable both on zero and one, give one result.

root@OpenWrt:~# cat /sys/kernel/debug/dpmac1_sfp/state
Module state: present
Module probe attempts: 0 0
Device state: down
Main state: down
Fault recovery remaining retries: 0
PHY probe remaining retries: 0
moddef0: 1
rx_los: 1
tx_fault: 0
tx_disable: 1

And with 1:

cat /sys/kernel/debug/dpmac1_sfp/state
Module state: present
Module probe attempts: 0 0
Device state: down
Main state: down
Fault recovery remaining retries: 0
PHY probe remaining retries: 0
moddef0: 1
rx_los: 1
tx_fault: 0
tx_disable: 1

Context:

root@OpenWrt:~# ethtool -m eth9
...
	Laser wavelength                          : 1490nm
	Vendor name                               : CTS INC.
	Vendor OUI                                : 00:90:65
	Vendor PN                                 : SFP-31W2BahSM-10

I have not tried those FS modules, but I do have both 1 and 10G BiDis from 10Gtek which I have tested successfully:

Thank you, these look nice; sadly, I’m dealing with a single-mode on my ISP’s side, not sure if these modules will be applicable.

It could be that a ‘qurik’ needs to be implemented in the Linux SFP code for some of your modules.

With some probably meaning “most” here, given how much Chinese devices I have. :slight_smile:

The docs mention the extender cable. Is there any chance to add a pull-down resistor on TX_DISABLE at such a cable by replacing one side with my custom board and test whether the issue is only with one GPIO pin? I think it’s better to try someday, given the kernel may change regularly…

8

I decided to test two generic FS.com SFP+ modules I’ve previously mentioned: 75335, 75336.

Tue Aug 13 11:00:23 2024 kern.info kernel: [479015.933265] sfp dpmac2_sfp: module FS               SFP-GE-BX        rev      sn G2430256308      dc 240706

Tue Aug 13 11:01:01 2024 kern.info kernel: [479054.492242] sfp dpmac2_sfp: module FS               SFP-GE-BX        rev      sn G2410406595      dc 240430

Seems like those are recognized, yet I can not connect.

I’m not sure about these “thermal zone” messages either, but it shouldn’t be too terrible as the optical modules do not heat up too much so far.

Tue Aug 13 11:01:01 2024 kern.info kernel: [479054.531950] hwmon hwmon0: temp1_input not attached to any thermal zone

I’ve also bought a bunch of DAC cables and the Horaco’s ZX-SWTG3C12F switch arrived. I’ve tested the switch, it does its job well. We’ll see how this and these generic DAC cables show themselves soon, connected to the switch.

(Also, the comments on the previous message would help.)

9

Sorry for the stupid question, but is the attached interface (eth8 or eth9) set to UP when these captures were taken?
The Linux SFP code will not ‘activate’ the SFP (driving TX_DISABLE), or the host side even, until the corresponding interface has been brought up.
Here is an example:

$ cat /sys/kernel/debug/dpmac2_sfp/state
Module state: present
Module probe attempts: 0 0
Device state: down
Main state: down
Fault recovery remaining retries: 0
PHY probe remaining retries: 0
moddef0: 1
rx_los: 0
tx_fault: 0
tx_disable: 1
$ ifconfig eth8 up
[   39.032017] fsl_dpaa2_eth dpni.1 eth8: configuring for inband/10gbase-r link mode
[   39.108046] fsl_dpaa2_eth dpni.1 eth8: Link is Up - 10Gbps/Full - flow control off
[   39.115687] IPv6: ADDRCONF(NETDEV_CHANGE): eth8: link becomes ready
$ cat /sys/kernel/debug/dpmac2_sfp/state
Module state: present
Module probe attempts: 0 0
Device state: up
Main state: link_up
Fault recovery remaining retries: 5
PHY probe remaining retries: 12
moddef0: 1
rx_los: 0
tx_fault: 0
tx_disable: 0

A DAC is a good sanity check, as that is just a simple electrical connection (which also ignores the control/state signals from the host).

10

Thank you, it’s actually a very good question to ask, as I expected OpenWRT to bring up those ports by itself due to them being mentioned in the UI! Plus, I haven’t noticed, but those interfaces were down in the checks previously.

root@OpenWrt:~# cat /sys/kernel/debug/dpmac1-sfp/state 
Module state: present
Module probe attempts: 0 0
Device state: down
Main state: down
Fault recovery remaining retries: 0
PHY probe remaining retries: 0
moddef0: 1
rx_los: 0
tx_fault: 0
tx_disable: 1

root@OpenWrt:~# ifconfig eth9 up

root@OpenWrt:~# cat /sys/kernel/debug/dpmac1-sfp/state 
Module state: present
Module probe attempts: 0 0
Device state: up
Main state: link_up
Fault recovery remaining retries: 5
PHY probe remaining retries: 12
moddef0: 1
rx_los: 0
tx_fault: 0
tx_disable: 0

I was able to add an Ampcom module and partially connect. One (right, but that depends on the SFP cage) led for the SFPs was blinking, but not the other one. I’m less sure about the TP-link module, but I’ll poke both more. Moreover, the laptop doesn’t get DHCP from this connection, yet indicates it.

Maybe it’s some of the OpenWRT configuration quirks, maybe a small update is needed to enable them by default.

UPD: a reboot brings the SFP slots down again.

For the context, it is the interface list after a reboot and an attempt to enable a 10G Ampcom module / connect to it.

BusyBox v1.36.1 (2024-06-07 03:52:12 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 23.05-SNAPSHOT, 1322022692
 -----------------------------------------------------
root@OpenWrt:~# ifconfig eth9 up
root@OpenWrt:~# ifconfig eth8 up
root@OpenWrt:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd :: permaddr 610:d513:6e36::
3: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-lan state DOWN group default qlen 1000
    link/ether 00:0a:fa:24:35:5d brd ff:ff:ff:ff:ff:ff
4: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-lan state DOWN group default qlen 1000
    link/ether 00:0a:fa:24:35:5e brd ff:ff:ff:ff:ff:ff
5: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-lan state DOWN group default qlen 1000
    link/ether 00:0a:fa:24:35:5f brd ff:ff:ff:ff:ff:ff
6: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-lan state DOWN group default qlen 1000
    link/ether 00:0a:fa:24:35:60 brd ff:ff:ff:ff:ff:ff
7: eth4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0a:fa:24:35:61 brd ff:ff:ff:ff:ff:ff
8: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0a:fa:24:35:62 brd ff:ff:ff:ff:ff:ff
9: eth6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0a:fa:24:35:63 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.108/24 brd 192.168.1.255 scope global eth6
       valid_lft forever preferred_lft forever
    inet6 fe80::20a:faff:fe24:3563/64 scope link 
       valid_lft forever preferred_lft forever
10: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:0a:fa:24:35:64 brd ff:ff:ff:ff:ff:ff
11: eth8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0a:fa:24:35:65 brd ff:ff:ff:ff:ff:ff
12: eth9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0a:fa:24:35:66 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20a:faff:fe24:3566/64 scope link tentative 
       valid_lft forever preferred_lft forever
17: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0a:fa:24:35:5d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fd47:440:1290::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::20a:faff:fe24:355d/64 scope link 
       valid_lft forever preferred_lft forever
20: phy0-ap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP group default qlen 1000
    link/ether 00:0a:52:08:b3:a0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20a:52ff:fe08:b3a0/64 scope link 
       valid_lft forever preferred_lft forever
22: phy1-ap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP group default qlen 1000
    link/ether 00:0a:52:08:b3:a1 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20a:52ff:fe08:b3a1/64 scope link 
       valid_lft forever preferred_lft forever
11

So, while I’d still like some reply to the previous comment and I’m very confused about the whole part with disabled interfaces, I’ve got a DAC cable.

Yes, it connects, even if the interfaces are “disabled”, but it seems like no data is being carried, almost as if all the SFP devices need an additional manual configuration I’m missing.

That reminds me of the docs:

OpenWrt on Ten64 - Traverse Ten64 Documentation
As of September 2023, there are a few features missing from the upstream OpenWrt.org images:

  • SFP status screen in LuCI

I don’t think I saw this screen. Is there something with my OpenWRT? I believe I’ve installed it according to the documentation.

Anyway, I’ve configured the switch to have 192.168.1.3 address and try to ping it with connected DAC. I get results on the local machine and router only if the switch is connected to fiber, that, itself, is connected to the Ethernet.

No traffic through a switch here:

signal-2024-08-15-133503_002
signal-2024-08-15-133503_0021600×900 132 KB

Stable traffic through a switch here, because the fiber is connected to a media converter. The only working Ethernet SFP I have is the Ampcom one currently in use.

signal-2024-08-15-133938_002
signal-2024-08-15-133938_0021600×900 117 KB

UPD: I’ll add some more data for the context.

root@OpenWrt:~# cat /sys/kernel/debug/dpmac2-sfp/state
Module state: present
Module probe attempts: 0 0
Device state: up
Main state: link_up
Fault recovery remaining retries: 5
PHY probe remaining retries: 12
moddef0: 1
rx_los: 0
tx_fault: 0
tx_disable: 0

Screenshot_2024-08-15_14-01-30
Screenshot_2024-08-15_14-01-30622×576 47.5 KB

UPD2: now I feel silly for not realising it earlier: the SFP devices weren’t specifically configured for the bridge network. Instead, they just were there. Everything works after a reboot.

For the context, I visited the network configuration, edited a br-lan device, added the eth8 (for dpmac2) and eth9 (for dpmac1), now everything is OK.

Screenshot_2024-08-15_14-23-36
Screenshot_2024-08-15_14-23-36898×262 18.9 KB

12

I’ve been busy the last few days, apologies for the delayed response.

Yes, that was what I was going to add.
The default OpenWrt setup only adds the first 4 ports as LAN, plus eth6 as WAN. You need to setup the SFP ports as needed (adding to a bridge or it’s their own OpenWrt interface)

Are you using the mainline OpenWrt from OpenWrt.org? The SFP screen is only present in the Traverse builds. It requires a patch to ethtool which was not accepted. It is on my wishlist to rework entirely without the ethtool changes.

1 Like
13

Are you using the mainline OpenWrt from OpenWrt.org?

Besides the idea of having the SFP screen, are there any specific additional markers to check whether I have a Ten64 build?
I’ve used the stock installer and read about the installation method in the Ten64 documentation.

You need to setup the SFP ports as needed (adding to a bridge or it’s their own OpenWrt interface)

Understood, thanks. I believe that, with some additional testing, I should start sending documentation merge requests here and there to point out what parts confused me. :slight_smile:

14

Oops. The package for the SFP status screen has fallen out of the build configuration. (I had to reduce it a few months ago due to the build getting too big, and some other packages breaking)
I am working on refreshing the OpenWrt 23.05.4 build with it (SFP status screen) included.

1 Like
15

Please try this new build (23.05.4), the SFP screen is back:
https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/openwrt-23-05/1417921455/image/

SFP_optical_1
SFP_optical_1939×550 44.7 KB

new_build_version
new_build_version969×483 38 KB

The most reliable way is to see which server opkg pulls package lists from.
Our builds always pull from a build specific folder on archive.traverse.com.au:

root@OpenWrt:/# cat /etc/opkg/distfeeds.conf
src/gz openwrt_core https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/openwrt-23-05/1417921455/image/packages
src/gz openwrt_base https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/openwrt-23-05/1417921455/packages/base
src/gz openwrt_luci https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/openwrt-23-05/1417921455/packages/luci
src/gz openwrt_muvirt https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/openwrt-23-05/1417921455/packages/muvirt
src/gz openwrt_packages https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/openwrt-23-05/1417921455/packages/packages
src/gz openwrt_routing https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/openwrt-23-05/1417921455/packages/routing
src/gz openwrt_telephony https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/openwrt-23-05/1417921455/packages/telephony
src/gz openwrt_ten64 https://archive.traverse.com.au/pub/traverse/ls1088firmware/openwrt/branches/openwrt-23-05/1417921455/packages/ten64
16

Thanks for the update!
I’ll give it a try soon; there should be no issues, as there’s a lot of space on my m.2 drive. :slight_smile: