Trying to decide between MuVirt and OpenWRT

DanaG · March 26, 2025, 12:02am

I’m planning to migrate my main router from an x86 machine running OpenWRT to the Traverse Ten64.

I’m still debating whether to run muVirt on the thing, or run plain OpenWRT. My primary use case for guest VMs is just Home Assistant OS, which I currently run as a VM on an N97 running Proxmox. I have other VMs that will stay on the N97 machine anyway.

I noticed that a couple of the packages I use, and their dependencies and related packages, aren’t available on muvirt like they are on OpenWRT.

luci-app-nlbwmon: for per-machine traffic monitoring
luci-app-vnstat2: for overall bandwidth monitoring
luci-app-lxc: for running PiHole on the router cleanly (doesn’t need a full VM)
apcupsd: for talking to the UPS, because NUT doesn’t fully support it
luci-app-nut: for talking to apcupsd and exposing it over the network

luci-app-nut - git-24.346.66847-1bb28ba
nut - 2.8.0-3
nut-avahi-service - 2.8.0-3
nut-common - 2.8.0-3
nut-driver-apcupsd-ups - 2.8.0-3
nut-server - 2.8.0-3
nut-upsc - 2.8.0-3
nut-upscmd - 2.8.0-3
nut-web-cgi - 2.8.0-3

How hard would it be to add those?

Incidentally, is it supported to run Docker directly on MuVirt?

mcbridematt · March 26, 2025, 10:46am

muvirt is mostly a package of scripts to manage VM’s on OpenWrt. There are only some minor differences between in the base OS these days (it is mostly ensuring all the correct options are set) and the hope is that eventually the muvirt functions will be usable on standard OpenWrt.

There are some divergences from OpenWrt to enable all the features to work, such as using nginx as a proxy infront of uhttpd to allow the VM web consoles to work.

The difference in package list is not intentional, it’s just that muvirt is configured separately from OpenWrt. muvirt used to be much thinner as the intention was that all the “real” apps were run under VMs.

As long as the packages aren’t tied to the kernel version, the easiest way is just to add the official OpenWrt.org repositories (for the corresponding OpenWrt release) to your opkg/apk repo list and install them from there.

Otherwise you can use the OpenWrt or muvirt SDK packages (like muvirt-sdk-24.10-SNAPSHOT-armsr-armv8_gcc-13.3.0_musl.Linux-x86_64.tar.zst) to build the packages and install them.

I’m happy to take requests to add them to the package list for (our) OpenWrt or muvirt.

We used to have some constraints on our CI/CD system which limited the number of packages we could build, but it’s much better now.

Yes, and normal OpenWrt as well, but the Docker networking does not integrate with OpenWrt’s firewall system.

You need to do some configuration to allow the containers to access the internet, and basically treat them like separate devices on your network (e.g add firewall rules to port forward instead of using docker’s port forward function).

If you have things that are highly dependent on docker networking behaviours like the port forward or inter-container links, it might be better to put all your Docker/container workloads in a separate VM where it won’t conflict with OpenWrt’s functions.

(Also, make sure you mount a separate partition for Docker’s /var/lib/docker!)

FYI, You can run a port of proxmox on Arm machines like the Ten64

The best way to do it on a Ten64 is by installing Debian and adding repos for the community port, effectively converting from a Debian to Proxmox install.

See Install Proxmox VE on Debian bookworm · jiangcuo/Proxmox-Port Wiki · GitHub

I wrote an install script for an earlier port (called PiMox) and you might find some hints here: GitHub - mcbridematt/pimox-ten64: Proxmox/PiMox V7 installer script for Ten64, but the above install guide should cover everything.

DanaG · March 26, 2025, 6:37pm

Looks like I was able to install those packages just fine on muvirt. Does opkg know to prioritize distfeeds over customfeeds? If I go with that route, how would I deal with sysupgrade?

Additionally, are there any recommendations for how you’d make backups of VMs on muVirt?

If I wanted to include openwrt_nand in place of dhcp/pxe in the default boot order, how would I do that? If I try as the instructions say, I get: Unknown uclass 'openwrt_nand' in label

My u-boot version:

U-Boot 2023.10-rc3 (Sep 23 2023 - 05:51:23 +0000)

aarch64-linux-gnu-gcc (Linaro GCC 7.5-2019.12) 7.5.0
GNU ld (Linaro_Binutils-2019.12) 2.28.2.20170706

My primary reason for wanting the host OS to be the router is so there’s never a case where a VM failing to start brings down the whole network.

DanaG · March 27, 2025, 8:24pm

Here’s what I get with boot_targets set to openwrt_nand nvme0 usb0 mmc0 scsi0:

INFO:    PSCI Power Domain Map:
INFO:      Domain Node : Level 2, parent_node -1, State ON (0x0)
INFO:      Domain Node : Level 1, parent_node 0, State ON (0x0)
INFO:      Domain Node : Level 1, parent_node 0, State OFF (0x2)
INFO:      CPU Node : MPID 0x0, parent_node 1, State ON (0x0)
INFO:      CPU Node : MPID 0xffffffffffffffff, parent_node 1, State OFF (0x2)
INFO:      CPU Node : MPID 0xffffffffffffffff, parent_node 1, State OFF (0x2)
INFO:      CPU Node : MPID 0xffffffffffffffff, parent_node 1, State OFF (0x2)
INFO:      CPU Node : MPID 0xffffffffffffffff, parent_node 2, State OFF (0x2)
INFO:      CPU Node : MPID 0xffffffffffffffff, parent_node 2, State OFF (0x2)
INFO:      CPU Node : MPID 0xffffffffffffffff, parent_node 2, State OFF (0x2)
INFO:      CPU Node : MPID 0xffffffffffffffff, parent_node 2, State OFF (0x2)
INFO:    RCW BOOT SRC is QSPI
INFO:    RCW BOOT SRC is QSPI
WARNING: Using I2C3 for DDR SPD
INFO:    platform clock 700000000
INFO:    DDR PLL 2100000000
INFO:    time base 11 ms
INFO:    Parse DIMM SPD(s)
INFO:    Controller 0
INFO:    DIMM 0
INFO:    addr 0x51
WARNING: Using I2C3 for DDR SPD
WARNING: Using I2C3 for DDR SPD
WARNING: Using I2C3 for DDR SPD
WARNING: Using I2C3 for DDR SPD
INFO:    checksum 0xc06c86cb
INFO:    n_ranks 2
INFO:    rank_density 0x400000000
INFO:    capacity 0x800000000
INFO:    die density 0x6
INFO:    primary_sdram_width 64
INFO:    ec_sdram_width 8
INFO:    device_width 8
INFO:    package_3ds 0
INFO:    rc 0x26 (Raw card G1)
INFO:    rdimm 0
INFO:    mirrored_dimm 1
INFO:    n_row_addr 17
INFO:    n_col_addr 10
INFO:    bank_addr_bits 0
INFO:    bank_group_bits 2
INFO:    edc_config 2
INFO:    burst_lengths_bitmask 0xc
INFO:    tckmin_x_ps 625
INFO:    tckmax_ps 1600
INFO:    caslat_x 0x17ffc00
INFO:    taa_ps 13750
INFO:    trcd_ps 13750
INFO:    trp_ps 13750
INFO:    tras_ps 32000
INFO:    trc_ps 45750
INFO:    trfc1_ps 350000
INFO:    trfc2_ps 260000
INFO:    trfc4_ps 160000
INFO:    tfaw_ps 21000
INFO:    trrds_ps 2500
INFO:    trrdl_ps 4900
INFO:    tccdl_ps 5000
INFO:    trfc_slr_ps 0
INFO:    twr_ps 15000
INFO:    refresh_rate_ps 7800000
INFO:    dq_mapping 0xb
INFO:    dq_mapping 0x2b
INFO:    dq_mapping 0xc
INFO:    dq_mapping 0x2b
INFO:    dq_mapping 0x2b
INFO:    dq_mapping 0xb
INFO:    dq_mapping 0x16
INFO:    dq_mapping 0x36
INFO:    dq_mapping 0xc
INFO:    dq_mapping 0x2b
INFO:    dq_mapping 0x15
INFO:    dq_mapping 0x2c
INFO:    dq_mapping 0xb
INFO:    dq_mapping 0x35
INFO:    dq_mapping 0x16
INFO:    dq_mapping 0x36
INFO:    dq_mapping 0x16
INFO:    dq_mapping 0x36
INFO:    dq_mapping_ors 1
INFO:    DIMM 1
INFO:    done with controller 0
INFO:    cal cs
INFO:    cs_in_use = 3
INFO:    cs_on_dimm[0] = 3
NOTICE:  UDIMM 9965657-052.A00G  
INFO:    Time after parsing SPD 562 ms
INFO:    Synthesize configurations
INFO:    cs 0
INFO:         odt_rd_cfg 0x0
INFO:         odt_wr_cfg 0x5
INFO:         odt_rtt_norm 0x3
INFO:         odt_rtt_wr 0x0
INFO:         auto_precharge 0
INFO:    cs 1
INFO:         odt_rd_cfg 0x0
INFO:         odt_wr_cfg 0x0
INFO:         odt_rtt_norm 0x0
INFO:         odt_rtt_wr 0x0
INFO:         auto_precharge 0
INFO:    cs 2
INFO:         odt_rd_cfg 0x0
INFO:         odt_wr_cfg 0x0
INFO:         odt_rtt_norm 0x0
INFO:         odt_rtt_wr 0x0
INFO:         auto_precharge 0
INFO:    cs 3
INFO:         odt_rd_cfg 0x0
INFO:         odt_wr_cfg 0x0
INFO:         odt_rtt_norm 0x0
INFO:         odt_rtt_wr 0x0
INFO:         auto_precharge 0
INFO:    ctlr_init_ecc 1
INFO:    x4_en 0
INFO:    ap_en 0
INFO:    ctlr_intlv 0
INFO:    ctlr_intlv_mode 0
INFO:    ba_intlv 0x40
INFO:    data_bus_used 0
INFO:    otf_burst_chop_en 1
INFO:    burst_length 0x6
INFO:    dbw_cap_shift 0
INFO:    Assign binding addresses
INFO:    ctlr_intlv 0
INFO:    rank density 0x400000000
INFO:    CS 0
INFO:        base_addr 0x0
INFO:        size 0x800000000
INFO:    CS 1
INFO:        base_addr 0x0
INFO:        size 0x800000000
INFO:    base 0x0
INFO:    Total mem by assignment is 0x800000000
INFO:    Calculate controller registers
INFO:    Skip CL mask for this speed 0x4000
INFO:    Skip caslat 0x4000
INFO:    cs_in_use = 0x3
INFO:    cs0
INFO:       _config = 0x80050522
INFO:    cs[0].bnds = 0x7ff
INFO:    cs_in_use = 0x3
INFO:    cs1
INFO:       _config = 0x80000522
INFO:    cs[1].bnds = 0x7ff
INFO:    sdram_cfg[0] = 0xe5004000
INFO:    sdram_cfg[1] = 0x401151
INFO:    sdram_cfg[2] = 0x0
INFO:    timing_cfg[0] = 0xd1770018
INFO:    timing_cfg[1] = 0xf2fc8245
INFO:    timing_cfg[2] = 0x594197
INFO:    timing_cfg[3] = 0x2161100
INFO:    timing_cfg[4] = 0x220002
INFO:    timing_cfg[5] = 0x5401400
INFO:    timing_cfg[6] = 0x0
INFO:    timing_cfg[7] = 0x26600000
INFO:    timing_cfg[8] = 0x5446a00
INFO:    timing_cfg[9] = 0x0
INFO:    dq_map[0] = 0x2eb32bac
INFO:    dq_map[1] = 0x2d6d95b0
INFO:    dq_map[2] = 0x2f55b658
INFO:    dq_map[3] = 0xd8cac001
INFO:    sdram_mode[0] = 0x3010631
INFO:    sdram_mode[1] = 0x100200
INFO:    sdram_mode[9] = 0x8400000
INFO:    sdram_mode[8] = 0x500
INFO:    sdram_mode[2] = 0x10631
INFO:    sdram_mode[3] = 0x100200
INFO:    sdram_mode[10] = 0x400
INFO:    sdram_mode[11] = 0x8400000
INFO:    sdram_mode[4] = 0x10631
INFO:    sdram_mode[5] = 0x100200
INFO:    sdram_mode[12] = 0x400
INFO:    sdram_mode[13] = 0x8400000
INFO:    sdram_mode[6] = 0x10631
INFO:    sdram_mode[7] = 0x100200
INFO:    sdram_mode[14] = 0x400
INFO:    sdram_mode[15] = 0x8400000
INFO:    eor = 0x40000000
INFO:    interval = 0x1ffe07ff
INFO:    zq_cntl = 0x8a090705
INFO:    ddr_sr_cntr = 0x0
INFO:    clk_cntl = 0x2400000
INFO:    cdr[0] = 0x80080000
INFO:    cdr[1] = 0xc0
INFO:    wrlvl_cntl[0] = 0x8675f605
INFO:    wrlvl_cntl[1] = 0x704080b
INFO:    wrlvl_cntl[2] = 0xf0c0d0a
INFO:    debug[28] = 0x7b
INFO:    Time before programming controller 830 ms
INFO:    Program controller registers
INFO:    total size 32 GB
INFO:    Need to wait up to 8160 ms
INFO:    Reading debug[9] as 0x2d003100
INFO:    Reading debug[10] as 0x2a003200
INFO:    Reading debug[11] as 0x39004100
INFO:    Reading debug[12] as 0x3c003f00
INFO:    cpo_min 0x2a
INFO:    cpo_max 0x41
INFO:    debug[28] 0x6a007b
WARNING: Warning: A009942 requires setting cpo_sample to 0x5c
INFO:    *0x1080000 = 0x7ff
INFO:    *0x1080008 = 0x7ff
INFO:    *0x1080080 = 0x80050522
INFO:    *0x1080084 = 0x80000522
INFO:    *0x1080100 = 0x2161100
INFO:    *0x1080104 = 0xd1770018
INFO:    *0x1080108 = 0xf2fc8245
INFO:    *0x108010c = 0x594197
INFO:    *0x1080110 = 0xe5004000
INFO:    *0x1080114 = 0x401141
INFO:    *0x1080118 = 0x3010631
INFO:    *0x108011c = 0x100200
INFO:    *0x1080120 = 0x1600086b
INFO:    *0x1080124 = 0x1ffe07ff
INFO:    *0x1080128 = 0xdeadbeef
INFO:    *0x1080130 = 0x2400000
INFO:    *0x1080160 = 0x220002
INFO:    *0x1080164 = 0x5401400
INFO:    *0x108016c = 0x26600000
INFO:    *0x1080170 = 0x8a090705
INFO:    *0x1080174 = 0xc675f605
INFO:    *0x1080190 = 0x704080b
INFO:    *0x1080194 = 0xf0c0d0a
INFO:    *0x1080200 = 0x10631
INFO:    *0x1080204 = 0x100200
INFO:    *0x1080208 = 0x10631
INFO:    *0x108020c = 0x100200
INFO:    *0x1080210 = 0x10631
INFO:    *0x1080214 = 0x100200
INFO:    *0x1080220 = 0x500
INFO:    *0x1080224 = 0x8400000
INFO:    *0x1080228 = 0x400
INFO:    *0x108022c = 0x8400000
INFO:    *0x1080230 = 0x400
INFO:    *0x1080234 = 0x8400000
INFO:    *0x1080238 = 0x400
INFO:    *0x108023c = 0x8400000
INFO:    *0x1080250 = 0x5446a00
INFO:    *0x1080270 = 0xffff
INFO:    *0x1080280 = 0xeedeeedd
INFO:    *0x1080284 = 0xedeedddd
INFO:    *0x1080288 = 0x11211122
INFO:    *0x108028c = 0x12112222
INFO:    *0x1080290 = 0xb3de0001
INFO:    *0x1080400 = 0x2eb32bac
INFO:    *0x1080404 = 0x2d6d95b0
INFO:    *0x1080408 = 0x2f55b658
INFO:    *0x108040c = 0xd8cac001
INFO:    *0x1080b20 = 0x8080
INFO:    *0x1080b24 = 0x80000000
INFO:    *0x1080b28 = 0x80080000
INFO:    *0x1080b2c = 0xc0
INFO:    *0x1080bf8 = 0x20502
INFO:    *0x1080bfc = 0x100
INFO:    *0x1080c00 = 0x40000000
INFO:    *0x1080f04 = 0x2
INFO:    *0x1080f08 = 0x9
INFO:    *0x1080f0c = 0x14000c20
INFO:    *0x1080f24 = 0x2d003100
INFO:    *0x1080f28 = 0x2a003200
INFO:    *0x1080f2c = 0x39004100
INFO:    *0x1080f30 = 0x3c003f00
INFO:    *0x1080f34 = 0x35007000
INFO:    *0x1080f48 = 0x1
INFO:    *0x1080f4c = 0x94000000
INFO:    *0x1080f50 = 0xc000f00
INFO:    *0x1080f54 = 0xa001300
INFO:    *0x1080f58 = 0x18002000
INFO:    *0x1080f5c = 0x1b001d00
INFO:    *0x1080f60 = 0x16000000
INFO:    *0x1080f64 = 0x9000
INFO:    *0x1080f68 = 0x20
INFO:    *0x1080f70 = 0x6a007b
INFO:    *0x1080f94 = 0x80000000
INFO:    *0x1080f9c = 0x29002800
INFO:    *0x1080fa0 = 0x28002800
INFO:    *0x1080fa4 = 0x28002600
INFO:    *0x1080fa8 = 0x2b002900
INFO:    *0x1080fac = 0x27000000
INFO:    *0x1080fb0 = 0x3
INFO:    *0x1080fb4 = 0x1f1d1c1e
INFO:    *0x1080fb8 = 0x1e1d1d1d
INFO:    *0x1080fbc = 0x1e1d1d1f
INFO:    *0x1080fc0 = 0x1d1e1c1b
INFO:    *0x1080fc4 = 0x1e1d1b1f
INFO:    *0x1080fc8 = 0x1c1d1f1d
INFO:    *0x1080fcc = 0x1e1c1c1d
INFO:    *0x1080fd0 = 0x1b1c1d1c
INFO:    *0x1080fd4 = 0x1e1e1b1e
INFO:    *0x1080fd8 = 0x1e1e1d1c
INFO:    *0x1080fdc = 0x1e1a1a1b
INFO:    *0x1080fe0 = 0x1c1c1d1a
INFO:    *0x1080fe4 = 0x1f1f1d20
INFO:    *0x1080fe8 = 0x1e201d1e
INFO:    *0x1080fec = 0x1e1e1d1e
INFO:    *0x1080ff0 = 0x1d1f1f1c
INFO:    *0x1080ff4 = 0x1e1d1a1d
INFO:    *0x1080ff8 = 0x1c1d1d1b
INFO:    *0x1080ffc = 0x1f000000

NOTICE:  32 GB DDR4, 64-bit, CL=15, ECC on, CS0+CS1
INFO:    Time used by DDR driver 4396 ms
NOTICE:  BL2: v1.5(debug):
NOTICE:  BL2: Built : 05:52:13, Sep 23 2023
INFO:    Configuring TrustZone Controller
INFO:    Value of region base = ffe00000
INFO:    Value of region base = 1ffe00000
INFO:    Value of region base = fbe00000
INFO:    Value of region base = 8880000000
INFO:    BL2: Doing platform setup
INFO:    BL2: Loading image id 3
INFO:    Loading image id=3 at address 0xfbe00000
INFO:    Image id=3 loaded: 0xfbe00000 - 0xfbe0c16c
INFO:    BL2: Loading image id 5
INFO:    Loading image id=5 at address 0x82000000
INFO:    Image id=5 loaded: 0x82000000 - 0x82153900
NOTICE:  BL2: Booting BL31
INFO:    Entry point address = 0xfbe00000
INFO:    SPSR = 0x3cd
NOTICE:  BL31: v1.5(debug):
NOTICE:  BL31: Built : 05:52:18, Sep 23 2023
NOTICE:  Welcome to LS1088 BL31 Phase
INFO:    GICv3 without legacy support detected. ARM GICV3 driver initialized in EL3
INFO:    BL31: Initializing runtime services
WARNING: BL31: cortex_a53: CPU workaround for 835769 was missing!
WARNING: BL31: cortex_a53: CPU workaround for 843419 was missing!
INFO:    BL31: cortex_a53: CPU workaround for 855873 was applied
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x82000000
INFO:    SPSR = 0x3c9


U-Boot 2023.10-rc3 (Sep 23 2023 - 05:51:23 +0000)

SoC:  LS1088AE Rev1.0 (0x87030010)
Clock Configuration:
       CPU0(A53):1600 MHz  CPU1(A53):1600 MHz  CPU2(A53):1600 MHz  
       CPU3(A53):1600 MHz  CPU4(A53):1600 MHz  CPU5(A53):1600 MHz  
       CPU6(A53):1600 MHz  CPU7(A53):1600 MHz  
       Bus:      700  MHz  DDR:      2100 MT/s
Reset Configuration Word (RCW):
       00000000: 4000541c 00000040 00000000 00000000
       00000010: 00000000 000a0000 00300000 00000000
       00000020: 010011a0 00002580 00000000 00000000
       00000030: 013fe60a 00000000 00002c03 00000000
       00000040: 00000000 00000000 00000000 00000000
       00000050: 00000000 00000000 00000000 00000000
       00000060: 00000000 00000000 00000089 000009e7
       00000070: 44110000 0d007755
DRAM:  31.9 GiB
DDR    31.9 GiB (DDR4, 64-bit, CL=15, ECC on)
       DDR Chip-Select Interleaving Mode: CS0+CS1
Core:  68 devices, 24 uclasses, devicetree: separate
WDT:   Not starting wdt@c000000
WDT:   Not starting wdt@c010000
WDT:   Not starting wdt@c020000
WDT:   Not starting wdt@c030000
WDT:   Not starting wdt@c100000
WDT:   Not starting wdt@c110000
WDT:   Not starting wdt@c120000
WDT:   Not starting wdt@c130000
Using SERDES1 Protocol: 29 (0x1d)
Using SERDES2 Protocol: 20 (0x14)
MMC:   FSL_SDHC: 0
Loading Environment from SPIFlash... SF: Detected en25s64 with page size 256 Bytes, erase size 64 KiB, total 8 MiB
OK
PCIe1: pcie@3400000 Root Complex: no link
PCIe2: pcie@3500000 Root Complex: x1 gen2
PCIe3: pcie@3600000 Root Complex: x2 gen3
In:    serial@21c0500
Out:   serial@21c0500
Err:   serial@21c0500
Model: Traverse Ten64
Board: 1064-0201C, boot from QSPI
Controller: firmware 1.1.5
DPMAC7: 00:0A:FA:xx:yy:z5
DPMAC8: 00:0A:FA:xx:yy:z6
DPMAC9: 00:0A:FA:xx:yy:z7
DPMAC10: 00:0A:FA:xx:yy:z8
DPMAC3: 00:0A:FA:xx:yy:z9
DPMAC4: 00:0A:FA:xx:yy:zA
DPMAC5: 00:0A:FA:xx:yy:zB
DPMAC6: 00:0A:FA:xx:yy:zC
DPMAC2: 00:0A:FA:xx:yy:zD
DPMAC1: 00:0A:FA:xx:yy:zE
Retimer: (retimer on, resetting...) DS110DF410 found
OK
Fan:  PWM reset done
USB Hub:    OK
Bus usb@3100000: Register 200017f NbrPorts 2
Starting the controller
USB XHCI 1.00
Bus usb@3110000: Register 200017f NbrPorts 2
Starting the controller
USB XHCI 1.00
scanning bus usb@3100000 for devices... 1 USB Device(s) found
scanning bus usb@3110000 for devices... 3 USB Device(s) found
Net:   eth9: DPMAC1@10gbase-r, eth8: DPMAC2@10gbase-r, eth4: DPMAC3@qsgmii, eth5: DPMAC4@qsgmii, eth6: DPMAC5@qsgmii, eth7: DPMAC6@qsgmii, eth0: DPMAC7@qsgmii, eth1: DPMAC8@qsgmii, eth2: DPMAC9@qsgmii, eth3: DPMAC10@qsgmii
SF: Detected en25s64 with page size 256 Bytes, erase size 64 KiB, total 8 MiB
device 0 offset 0x300000, size 0x200000
SF: 2097152 bytes @ 0x300000 Read: OK
device 0 offset 0x5c0000, size 0x40000
SF: 262144 bytes @ 0x5c0000 Read: OK
crc32+ MC firmware version 10.36.0

fsl-mc: Booting Management Complex ... SUCCESS
fsl-mc: Management Complex booted (version: 10.36.0, boot status: 0x1)
device 0 offset 0x580000, size 0x40000
SF: 262144 bytes @ 0x580000 Read: OK
default DPL loaded
Autoboot in 10 seconds, press 's' to stop and bring up boot menu
Scanning for bootflows in all bootdevs
Seq  Method       State   Uclass    Part  Name                      Filename
---  -----------  ------  --------  ----  ------------------------  ----------------
Scanning global bootmeth 'efi_mgr':
Hunting with: simple_bus
Found 0 extension board(s).
Unknown uclass 'openwrt_nand' in label
Scanning bootdev 'nvme#0.blk#1.bootdev':
No bootdevs for 'usb0'
Scanning bootdev 'esdhc@2140000.bootdev':
  0  efi          ready   mmc          1  esdhc@2140000.bootdev.par efi/boot/bootaa64.efi
** Booting bootflow 'esdhc@2140000.bootdev.part_1' with efi
Working FDT set to 90000000
TEN64: Using legacy fan device tree overlay
Missing TPMv2 device for EFI_TCG_PROTOCOL
INFO:    RNG Desc SUCCESS with status 0
INFO:    result f411a388b3aee7b8
Booting /efi\boot\bootaa64.efi
Welcome to GRUB!
DPMAC7@qsgmii Waiting for PHY auto negotiation to complete......... TIMEOUT !
DPMAC7@qsgmii: Could not initialize
DPMAC8@qsgmii Waiting for PHY auto negotiation to complete......... TIMEOUT !
DPMAC8@qsgmii: Could not initialize
DPMAC9@qsgmii Waiting for PHY auto negotiation to complete......... TIMEOUT !
DPMAC9@qsgmii: Could not initialize
DPMAC10@qsgmii Waiting for PHY auto negotiation to complete......... TIMEOUT !
DPMAC10@qsgmii: Could not initialize
error: serial port `com0' isn't found.
error: terminal `serial' isn't found.
error: terminal `serial' isn't found.

Then Grub boots, but it’s muVirt, not openwrt on NAND.

mcbridematt · March 28, 2025, 10:29pm

Apologies, I need to update the instructions.
The NAND boot isn’t (yet) integrated into the new “bootflow” system in the newer U-Boot versions.
I am working on a U-Boot update at the moment, so I might be able to fix this.

If you want to boot into NAND by default, you can override the default boot command in U-Boot like so (interrupt the boot sequence by hitting s when prompted to get a prompt):

setenv bootcmd 'run bootcmd_openwrt_nand'
saveenv

To do the reverse (NAND boot after all other options are exhausted), I think you could try something like:

setenv boot_targets '...' (without dhcp/pxe)
setenv bootcmd 'bootflow scan -lb; run bootcmd openwrt_nand'
saveenv

(This will only work if none of the NVME/SD/USB devices have no EFI or other bootable files, or you configure GRUB on those devices to exit back to firmware)

As far as I know, opkg will prioritize feeds in the order they are read, which I think is distfeeds → customfeeds

When upgrading, you will need to add the repository URLs again and re-install the packages.
(For both us and upstream OpenWrt, the repositories are built for the exact release or snapshot version, so you don’t want to have old repository references carry across)

Another method would be use ImageBuilder to build your desired packages into an image which you then upload to your system.
I keep a collection of scripts for various custom images and have them setup so I just download the new copy of ImageBuilder and change the script to operate on the new ImageBuilder folder.

It’s not quite “backup”, but when I have to move VMs from one host to another, I do something like this:

# From sending system
/etc/init.d/muvirt stop vmmname
dd if=/dev/mapper/vmdata-vmname | gzip -c | nc -q 0 <otherhost> 1234
# Receiving system
lvcreate --name vmname --size XGiB vmdata
nc -l 1234 | gunzip -c | dd of=/dev/mapper/vmdata-vmname

If you are using the LVM storage manged by muvirt, you could use LVM snapshots to make a copy of a live VM.