Getting closer to 10 gigabit

I got my first 10 gigabit client device, I’ve been able to get reasonable speeds out of it but I’m wondering if a) iperf3 is not the best test for this and b) whether and how to set something like jumbo frames on both sides without causing issues with the br-lan device.

Note that I’m able to max out my 2.5 gig clients (through a Mikrotik SFP+ switch) but I’ve only got one 10 gig device I can test with at the moment.

iperf3 server on the Ten64
$ iperf3 -c 10.0.0.1
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.180 port 55338 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   153 MBytes  1.28 Gbits/sec   36    871 KBytes
[  5]   1.00-2.00   sec   150 MBytes  1.26 Gbits/sec    0    996 KBytes
[  5]   2.00-3.00   sec   149 MBytes  1.25 Gbits/sec    2    836 KBytes
[  5]   3.00-4.00   sec   150 MBytes  1.26 Gbits/sec    0    966 KBytes
[  5]   4.00-5.00   sec   149 MBytes  1.25 Gbits/sec    2    805 KBytes
[  5]   5.00-6.00   sec   150 MBytes  1.26 Gbits/sec    0    935 KBytes
[  5]   6.00-7.00   sec   150 MBytes  1.26 Gbits/sec    1    766 KBytes
[  5]   7.00-8.00   sec   149 MBytes  1.25 Gbits/sec    0    906 KBytes
[  5]   8.00-9.00   sec   149 MBytes  1.25 Gbits/sec    2    723 KBytes
[  5]   9.00-10.00  sec   149 MBytes  1.25 Gbits/sec    0    870 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.46 GBytes  1.26 Gbits/sec   43             sender
[  5]   0.00-10.00  sec  1.46 GBytes  1.25 Gbits/sec                  receiver


Accepted connection from 10.0.0.180, port 39418
[  5] local 10.0.0.1 port 5201 connected to 10.0.0.180 port 39424
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   344 MBytes  2.87 Gbits/sec    0    272 KBytes
[  5]   1.00-2.01   sec   342 MBytes  2.87 Gbits/sec    0    272 KBytes
[  5]   2.01-3.01   sec   342 MBytes  2.87 Gbits/sec   12    272 KBytes
[  5]   3.01-4.00   sec   341 MBytes  2.87 Gbits/sec    0    272 KBytes
[  5]   4.00-5.01   sec   342 MBytes  2.87 Gbits/sec    0    272 KBytes
[  5]   5.01-6.00   sec   341 MBytes  2.87 Gbits/sec    0    272 KBytes
[  5]   6.00-7.01   sec   342 MBytes  2.87 Gbits/sec    0    272 KBytes
[  5]   7.01-8.01   sec   342 MBytes  2.87 Gbits/sec    0    272 KBytes
[  5]   8.01-9.01   sec   342 MBytes  2.87 Gbits/sec    0    272 KBytes
[  5]   9.01-10.00  sec   341 MBytes  2.87 Gbits/sec    0    272 KBytes

iperf3 server on the client
# iperf3 -c 10.0.0.180
Connecting to host 10.0.0.180, port 5201
[  5] local 10.0.0.1 port 33074 connected to 10.0.0.180 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   341 MBytes  2.86 Gbits/sec    0    177 KBytes
[  5]   1.00-2.00   sec   341 MBytes  2.86 Gbits/sec    0    185 KBytes
[  5]   2.00-3.00   sec   340 MBytes  2.86 Gbits/sec    0    195 KBytes
[  5]   3.00-4.00   sec   340 MBytes  2.85 Gbits/sec    0    195 KBytes
[  5]   4.00-5.00   sec   341 MBytes  2.86 Gbits/sec    0    195 KBytes
[  5]   5.00-6.00   sec   341 MBytes  2.86 Gbits/sec    0    195 KBytes
[  5]   6.00-7.00   sec   340 MBytes  2.86 Gbits/sec    0    195 KBytes
[  5]   7.00-8.00   sec   340 MBytes  2.85 Gbits/sec    0    317 KBytes
[  5]   8.00-9.00   sec   340 MBytes  2.85 Gbits/sec    0    317 KBytes
[  5]   9.00-10.00  sec   340 MBytes  2.85 Gbits/sec    0    317 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.33 GBytes  2.86 Gbits/sec    0             sender
[  5]   0.00-10.04  sec  3.33 GBytes  2.84 Gbits/sec                  receiver

# iperf3 -c 10.0.0.180 -R
Connecting to host 10.0.0.180, port 5201
Reverse mode, remote host 10.0.0.180 is sending
[  5] local 10.0.0.1 port 37540 connected to 10.0.0.180 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   172 MBytes  1.44 Gbits/sec
[  5]   1.00-2.00   sec   172 MBytes  1.44 Gbits/sec
[  5]   2.00-3.00   sec   172 MBytes  1.44 Gbits/sec
[  5]   3.00-4.00   sec   171 MBytes  1.44 Gbits/sec
[  5]   4.00-5.00   sec   176 MBytes  1.47 Gbits/sec
[  5]   5.00-6.00   sec   159 MBytes  1.33 Gbits/sec
[  5]   6.00-7.00   sec   155 MBytes  1.30 Gbits/sec
[  5]   7.00-8.00   sec   156 MBytes  1.30 Gbits/sec
[  5]   8.00-9.00   sec   182 MBytes  1.53 Gbits/sec
[  5]   9.00-10.00  sec   182 MBytes  1.53 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  1.66 GBytes  1.42 Gbits/sec   54             sender
[  5]   0.00-10.00  sec  1.66 GBytes  1.42 Gbits/sec                  receiver


Hi,

a) iperf3 is monothread and the CPU will be the bottleneck for 10Gbps links.

The former version (iperf) supports multithreading and bitrate will be much better by using the option -P on client side. I got 9.15 Gbits/sec on a 10G_KR link.