Future products - suggestions and feedback

[mcbridematt]

Hi all,

Many thanks for your support on the Ten64. This was the first direct to consumer project we have ever done and it has been a delight interacting with you all.

We would like to start designing our next generation of products soon. While there have been a few possibilities evaluated since the Ten64 launch, we are now finally seeing some new silicon that would be worthy of a successor product.

I cannot say what we are considering just yet as some of the chipsets we are considering have not been publicly announced yet, and it may be a long time before they are available in production quantities.

Ten64 updates

We are still “spinning” new revisions of the Ten64 hardware for customer projects.

As I mentioned in another thread, the most important being a change of the RAM slot to right-angle to work with a new fanless enclosure.
We do have to make other changes to replace parts that are going out of production as well.

On the software side, after the OpenWrt upstreaming, the next focus is on the firmware refresh and then working on some way to unlock full 10G routing performance in OpenWrt, either via XDP or AIOP.

Are there any improvements to the Ten64 that we could do right now to improve it?

What do we want to build next?

I can see room for (at least) two different products:

• Low-to-mid-range:
  • 2.5G Ethernet (at least “LAN”+“WAN”)
  • may have a SFP
  • limited PCIe lanes (typical config: 1x5G modem, 1xWiFi)
  • likely to have onboard eMMC instead of a M.2/SSD slot
  • CPU/SoC similar to Ten64 in total performance, but with much better power/thermals
  • More “industrial” IoT and 5G focus (with external I/O like RS-232 and RS-485)
• High end (Ten64 successor):
  • CPU/SoC significantly better performance than Ten64, slightly better power/thermals (expected to be fanless)
  • At least 1 x 10G+ SFP WAN plus 1 x 10G SFP+ LAN side
  • multiple 2.5GBASE-T LAN
  • more PCIe lanes
    • likely similar setup, M.2/M SSD + M.2/B cellular + M.2/E for WiFi. PCIe switch if needed.

We are open to other ideas, but as a niche player, we try to avoid duplicating things others are doing. (In other words, we aren’t going to build <$100 SBCs). If you have an idea, feel free to mention it!

Both products will aim for Arm SystemReady compliance and be compatible with many distributions out of the box (Debian/Fedora/openSUSE/OpenWrt/Ubuntu).

Areas we would like feedback on
These are not the only things being considered, feel free to ask about anything not listed here.

• System size and form factor: we have had a couple of customers tell us they love the Ten64, but that it’s simply too (physically) big.
  • Is being a standard form factor (like Mini-ITX) useful?
  • One form factor we have been evaluating is Mini-STX (~147x140mm), which is around 30% less area than Mini-ITX. (It’s similar in size to the APU boards from pcengines)
  • We would try to keep such boards “low profile” (no features taller than 1xRJ45/SFP), which makes it easier to design a fanless enclosure.
  • DIN-rail mounts is something we have been looking at.
  • We could also design ways to allow a smaller board to “scale out” (with additional ports or slots) when installed into a larger enclosure
  • Rackmount is something we missed on the Ten64 initially, we will try to have a method from the start next time
• RAM: 32GB was overwhelmingly popular with the Ten64 crowdfunding audience.
  • Does anyone see a need for higher in an edge device?
  • If lower sizes are offered, how low is comfortable? 8GB?
• GPU / display: Desktop and multimedia is not our target market, but if we had a “reasonable” GPU (including video codecs), would this be useful?
  • “reasonable” = relative to other small Arm SBCs on the market (Mali/Vivante level)
    (Assume there will always be a serial console available)
• Number of Ethernet ports: If we shrink the form factor down, then it’s harder to do lots of ports.
  • One potential high end config is 4x2.5G + 2x10G SFP+
• Ethernet switch vs independent controllers
  • We haven’t seen many “Nx2.5G” NICs around, and new designs appear to be favouring built-in switches.
  • Assume any switch we integrate (“internal” to the SoC or through an external chip) will be fully controllable from Linux (including DSA, VLANs, port mirroring etc.)
  • With that said, does anyone have strong opinions on this?
• WiFI: It is getting very hard to compete with the major vendors own SoC’s in both WiFi software (drivers) and hardware, but we will keep some method of using high end WiFi cards (including upcoming WiFi7) going forward.
  The lower end system would be designed for a lower end WiFi solution (like 2x2 dual band) only.
• Power over Ethernet: Instead of WiFi, would multiple PoE ports be useful? How many would be useful? Or even being powered from PoE?
• SATA and NAS applications: would having SATA ports be useful? Or multiple PCIe’s for more NVMe devices?
• Expansion: I don’t want to clone another boards GPIO header, but would something like a mikroBUS expansion header be useful? We have also seen some demand for external RS-232 and I/O ports.
• BMC/IPMI/Redfish: This is something we are looking into in the context of large fleet deployments. Would a “developer” friendly BMC add-on be of interest?

Looking forward to hearing your thoughts.

[dsl]

@mcbridematt Have you ever considered designing a “SoC PCB module”? It’d look like a separate small PCB with SoC, RAM, PMIC, NAND/NOR flash memory, etc. routed and castellated holes around its perimeter to be soldered on top of the other PCBs.

If you had one, it’d be possible to re-use it for different “carrier boards” with peripherals like Ethernet switches, SFP/SFP+ cages, NVMs, WiFi, DSI-to-FPD-Link serializers, industrial IoT, etc.

Such SoC PCB modules themselves can be built upon pin-compatible SoCs like TI’s TDA4VE, TDA4VL or TDA4AL (those are automotive-grade, but good enough as an example).

[mcbridematt]

@dsl

Have you ever considered designing a “SoC PCB module”? It’d look like a separate small PCB with SoC, RAM, PMIC, NAND/NOR flash memory, etc. routed and castellated holes around its perimeter to be soldered on top of the other PCBs.

It’s a good suggestion. We haven’t done it in the past for cost reasons, if there was a customer wanting to buy lots of boards right away, then having a SoM/SoC module would add quite a bit of cost to the final product.

The other problem is that good board to board connectors rated for 10Gbps differential pairs (10G Ethernet, PCIe 3) cost a bit of money, but there are cheaper options becoming available now.

We do have an i.MX6ULL SoM module, it works better there as that SoC is much more compact than the LS10XX and that SoM was designed to be embedded into bigger systems (e.g solar power inverters).

It’s an idea that could work better with the new SoCs we are considering, so I’ll keep it in mind.

[voltagex]

[Attempting to edit ramblings here]

• the Ten64 has been able to take over some of the functions I was previously using a SuperMicro NAS for. When the armsr builds are ready, if KVM works I’ll also be able to move Home Assistant over into a virtual machine - hence the need for RAM.

@mcbridematt

Number of Ethernet ports: If we shrink the form factor down, then it’s harder to do lots of ports.

• One potential high end config is 4x2.5G + 2x10G SFP+

• I’d be sad to lose more ports. I don’t think I’ve quite outgrown the Ten64, but I’ve definitely run out of gigabit ports. I could definitely use a 3rd SFP+ port for a link to a 2.5 gigabit switch or to wifi (as a standalone WAP gave me a lot less trouble than OpenWRT’s wifi support)

• Boards for experimentation - this is extremely niche, but there are few ARM boards with working PCIe and it’d be nice to be able to mess around with something smaller than a full-size x86 board but more powerful than a Pi

@mcbridematt

BMC/IPMI/Redfish: This is something we are looking into in the context of large fleet deployments. Would a “developer” friendly BMC add-on be of interest?

This has been absolute hell for me every time I’ve interacted with it - the $1000 AsrockRack board has barely functional IPMI, SuperMicro’s only worked after a couple of years of updates, the AsrockRack PAUL seems like barely functional vapourware.

It’d definitely be good to play with something I could get a shell on.

@mcbridematt

GPU / display: Desktop and multimedia is not our target market, but if we had a “reasonable” GPU (including video codecs), would this be useful?

• “reasonable” = relative to other small Arm SBCs on the market (Mali/Vivante level)
  (Assume there will always be a serial console available)

Not something I’d be interested in, and I’d be worried about it consuming power / memory.

@mcbridematt

System size and form factor: we have had a couple of customers tell us they love the Ten64, but that it’s simply too (physically) big.

Aw, I think it’s a good size! I think the top lid & rear screws are a little unwieldy, but I think that going smaller would mean compromising on features

@mcbridematt

WiFI: It is getting very hard to compete with the major vendors own SoC’s in both WiFi software (drivers) and hardware, but we will keep some method of using high end WiFi cards (including upcoming WiFi7) going forward.

If it were my choice, I’d say don’t focus on wifi at all, but I had a bumpy ride with this as you know

@mcbridematt

Power over Ethernet: Instead of WiFi, would multiple PoE ports be useful? How many would be useful? Or even being powered from PoE?

I’d be interested to hear how difficult this is from a hardware design point of view. For my uses, it’d have to be a minimum of 802.3at - seems like most Wifi 6(e) WAPs will need this anyway. I’d rather have some kind of addon box that sits on top of my router to do PoE. Software controlled PoE injector anyone?

@mcbridematt

SATA and NAS applications: would having SATA ports be useful? Or multiple PCIe’s for more NVMe devices?

As a separate SKU maybe, although I am an outlier in the amount of storage I use.

@mcbridematt

Expansion: I don’t want to clone another boards GPIO header, but would something like a mikroBUS expansion header be useful? We have also seen some demand for external RS-232 and I/O ports.

This looks cool - a Zigbee board would be excellent for Home Assistant use

[wrobell]

A bit of context. My Ten64 acts as 4G modem, router (WiFi + Ethernet),
NAS, and IOT server. I have custom enclosure, which hides it in plain
sight in the house. I am running slowly two fans, but I am overheating
it to keep it quiet.

I would keep the current form factor, and I would prefer to have 32 GB
of RAM. Optional DIN rail mount would be great.

I would love to see better power supply, and power management

• built-in USB-C power supply (GAN?)
• POE
• CPU frequency scaling (mine is quite idle, until some database query kicks in)
• fanless on top of that would be great

I use NVME to serve VMs. This interferes with my NAS, which uses two USB disks.
My plan is to upgrade this setup to 3 SATA disks (one for VMs, two others for NAS).
I need to remove NVME to put SATA card. It would be great to keep NVME, and
add just two SATA disks. Maybe future version could accomodate that, so I can
keep 2 WiFi cards and 4G/5G modem? (Any suggestions for current setup more
than welcome!)

No need for GPU, but if there is one, please allow to disable it with
DIP switch or a jumper.

I remember a discussion about 10 Gps ports supporting negotiation down
to 5 Gps or 2.5 Gps. That would be nice to have instead of separate
dedicated ports for these speeds.

Finally, screw terminal block for GPIO or something equivalent in
terms of secure connection.

[seanyoung]

The ten64 is great because it has many ethernet ports, nvme and two mini pcie. There are very few arm machines on the market which have this, so it’s filling a niche. Losing ports would not be good.
If anything, I’d like:

• more usb ports - 2 is not enough. Even bog standard rpi has 4 ports.
  • a currentcost monitor for my home’s electricity usage
  • a usb relay which switches the heating on/off
  • usb bt device for connecting to tempod temperature/humidity monitor
• 2 nvme ports - good for backups
• riscv cpu - if possible

[kpfleming]

I’m using a pair of ten64 boxes as routers, DHCP servers, NTP servers, and DNS resolvers. Some notes from my end:

• I’m not using the 1GbaseT ports at all, only the SFP+ slots.
• I’m using the internal USB A port, but it’s a pain since the connector is vertical and quite tall; I was able to find right-angle USB A cables on AliExpress so I made it work, but some other form of connector would be appreciated.
• I’d be happy with USB-PD for powering the box; the fewer ‘custom’ power supplies in my rack the better.
• I’d also be happy with POE+ for powering the box, although I’d use that Ethernet port only for power, not for communications.
• Bonus points if there are two USB-PD ports or two POE+ input ports, for redundant power supply.
• The existing control/GPIO header is quite fragile and connections to it are not mechanically sound unless made via ribbon cable, but that’s more difficult to use inside the ten64 enclosure. If there’s room to use .1" headers instead that would be welcome.
• Unused signals on the mPCIe slots could be made available on test pads, or even better on headers. In my case I was using a GPS receiver which output PPS on one of the LED signals but there was no practical way to get access to that on the ten64 board.

[DanaG]

I was looking at the Ten64 to replace my current router machine, which is a Dell thin client running OpenWRT, with a PCIe 2-port 10-gigabit NIC. My internet service is 1 gigabit, but the cable modem has a 2.5-gigabit ethernet port, so I went with a 10-gigabit NIC to avoid any buggy 2.5-gigabit NICs.

A Ten64 might be just the thing I need to replace that with lower power usage, so long as it’s not loud enough to hear from more than a foot or two away. Is the current Ten64 quiet enough that I shouldn’t need to wait for the passive one or an entirely new product?

In terms of devices I’d like for the router role, two 10-gig and two or more 2.5-gig ports sounds just right, because I have external switches and access points. I tried OpenWRT with the RT3200 router, but I’ve found proprietary to be more reliable, and more convenient about handling multiple APs.

As far as devices for other roles go, I don’t even know what other roles I’d want a device for!

One issue you may run into if you go the SystemReady route is trouble with getting upstream maintainers to accept patches. Just ask https://twitter.com/linux4kix.

[mcbridematt]

Thanks for all the feedback everyone, and keep it coming!

@DanaG

Is the current Ten64 quiet enough that I shouldn’t need to wait for the passive one or an entirely new product?

To be honest, it’s at “idle laptop fan” levels. You will hear it at 1-2 feet if there is nothing else in the room.

@DanaG

Just ask https://twitter.com/linux4kix .

Well aware of it, the LS1088 SoC is in the same family as the LX2160, though the latter (which came out later) had a few design tweaks to better match the Arm standard profiles. The whole saga was a bit disappointing TBH.

We are targeting the embedded profile (IR) which is better suited to our market, where most deployments run some form of customized/compiled OS and there is a desire for the OS to control more of the hardware.

I would not rule out certifying for a higher profile with a new SoC, but there needs to be a clear cost/benefit to justify it.

[6r1d]

that would be worthy of a successor product
High end (Ten64 successor)

While this is a bit off-topic, I’m curious to know whether Traverse has plans to keep the new device that’ll eventually replace Ten64 as open source hardware. This was the key reason for me to try the router and it wasn’t discussed yet, so if possible, please comment on that. Thank you.

---

I’m not sure how useful my feedback is now, but let me comment as the new user.

System size and form factor

• • Rackmount is something we missed on the Ten64 initially, we will try to have a method from the start next time

Personally, I’d be happy with something rack-mounted, possibly even 19 inch, which, IIRC, is similar to music racks. I think a mini-itx could be mounted into such a case, so whatever works is good enough.

RAM: 32GB was overwhelmingly popular with the Ten64 crowdfunding audience.

I’ve bought 32 Gb stick here, because it was easily available. ECC is preferrable to non-ECC.

My friend is curious about soldered RAM to start using the device from the get-go.

GPU / display: Desktop and multimedia is not our target market, but if we had a “reasonable” GPU (including video codecs), would this be useful?

I am not sure; honestly, I’m just looking at some way to access the console remotely, and probably RasPi would be enough for that.

Number of Ethernet ports : If we shrink the form factor down, then it’s harder to do lots of ports.

• • One potential high end config is 4x2.5G + 2x10G SFP+

I am already using the three ports now. I understand the size requirements, but if those apply, a lot more testing and improved SFP support is needed, as right now, I had lots of issues with different SFP modules. (These are partially resolved in the recent discussions, but I don’t yet know how people make OpenWRT work with those in a stable way.)

Ethernet switch vs independent controllers

• Assume any switch we integrate (“internal” to the SoC or through an external chip) will be fully controllable from Linux (including DSA, VLANs, port mirroring etc.)

That sounds curious, although I don’t fully understand the idea.

WiFI: It is getting very hard to compete with the major vendors own SoC’s in both WiFi software (drivers) and hardware, but we will keep some method of using high end WiFi cards (including upcoming WiFi7) going forward.

What are the main issues? I understand the bad driver availability, but besides that, do the WiFi cards impose any limits for routing?

• Power over Ethernet: Instead of WiFi, would multiple PoE ports be useful? How many would be useful? Or even being powered from PoE?

For me, only for a video surveilance device, as a part of home security. This sounds curious, but probably for another device entirely, with a GPU and oriented on recording videos. I’m yet to see a good Linux distro for such devices, though…

• We have also seen some demand for external RS-232 and I/O ports.

UART would be nice to have, but I can have some capabilities with what’s already there…

• BMC/IPMI/Redfish: This is something we are looking into in the context of large fleet deployments. Would a “developer” friendly BMC add-on be of interest?

I did not use those, but I am curious.

[battlez]

Hi Matt,

for the Ten64 successor I think I would really love to have a networking box ( contrary to a WiFi/DSL/5G/… addon version):

• 4x SFP+ ; still 4-8 1G copper
• undecided if an extra 40G QSFP+? Highly depends on what you want to build on.
• two of them fitting in a 1U rack format; ideally connectors for power supply and external buttons again
• ideally fanless with fan header in case needed
• maybe still NXP/DPAA2?
• I don’t think 2.5G is really a big thing to do; feels too “consumer” but not “enterprise” – I could be wrong.
• not too keen on many M.2 for expansion cards but at least one NVME on-baord
• possibly an internal adapter to connect two boards for communications?
• GPIO/I2C for small display etc.
• controllable blinkenlights again
• one external USB on a header would be nice
• supercap rtc; no battery is soooo good
• accessible power monitor
• FTDI still on USB-C header for serial access
• SD card still in addition to internal flash is so much of a winner for (off premises) recovery!
• UEFI/TianoCore EDKII support would be a nice plus

[voltagex]

@mcbridematt

BMC/IPMI/Redfish: This is something we are looking into in the context of large fleet deployments. Would a “developer” friendly BMC add-on be of interest?

Just coming back to this after having used Redfish and a BMC based on MegaRAC - maybe it’s me, but the MegaRAC-based BMCs are unwieldy and slow and the underlying Redfish server seems to be unreliable.

It would be interesting to see what a Ten64 with an OpenBMC board would be like, but you’re basically putting another ARM core into the system “just” to do power on/off and remote serial.

Remote-serial itself on a chip might be worth it IMO.

[battlez]

I’d love to add:

• access to the SFF RS0 (and RS1) pins — some SFPs use them along with TX_FAULT for serial console access which would be extremely useful (assuming 2/4 GPIO ports can be found)