Tips on installing NixOS

I just figured I’d document the (rough) configuration I have for my Ten64 that I use as a router.

Working:

  • Ethernet ports
  • NVMe SSD
  • Rebooting
  • USB Ports
  • Upstream kernel

Not tested:

  • Any WiFi/other internal devices not mentioned
  • Front panel LED’s, buttons, etc.
  • Fan controls or internal GPIO
  • SIM/SD slot

Didn’t work about 2 years ago, but might now:

  • SFP+

When I originally installed it, I simply grabbed the NixOS aarch64 installer, booted it up UEFI from a flash drive, modified the kernel command line to ignore the IOMMU issue (arm-smmu.disable_bypass=n), and it booted into a command line.

You will need to format the disk GPT, add the two partitions (see hardware-configuration.nix below for ideas), and replace the disk UUID’s with the ones you have. I have /boot at 1GB, and that’s okay but remember to clean up old generations as the kernels quickly fill the partition otherwise.

For my hardware-configuration.nix:

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernelModules = [ "nvme" "usb_storage" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/badae59e-a3b4-4499-aa2c-050c29051f3f";
      fsType = "ext4";
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/A695-045B";
      fsType = "vfat";
    };

  swapDevices = [ ];

  powerManagement.cpuFreqGovernor = lib.mkDefault "ondemand";
}

As for the configuration.nix, you may have some changes to make but here’s my template:

{ config, pkgs, ... }:

{
  imports =
    [
      ./hardware-configuration.nix
    ];

  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.kernelParams = [ "arm-smmu.disable_bypass=n" ];

  boot.loader.grub.enable = true;
  boot.loader.grub.efiSupport = true;
  boot.loader.grub.efiInstallAsRemovable = true;
  boot.loader.grub.device = "nodev";

  networking.hostName = "your hostname here";
  time.timeZone = "America/New_York";

  boot.kernel.sysctl = {
    "fs.inotify.max_user_watches" = 1048576;
    "net.ipv4.tcp_congestion_control" = "bbr";
    "net.ipv4.tcp_ecn" = 1;
    "net.core.default_qdisc" = "fq_codel";
  };

  systemd.network.enable = true;
  systemd.network.links = {
    "00-ge0" = {
      matchConfig = { PermanentMACAddress = "xx"; };
      linkConfig = { Name = "ge0"; };
    };
    "01-ge1" = {
      matchConfig = { PermanentMACAddress = "xx"; };
      linkConfig = { Name = "ge1"; };
    };
    "02-ge2" = {
      matchConfig = { PermanentMACAddress = "xx"; };
      linkConfig = { Name = "ge2"; };
    };
    "03-ge3" = {
      matchConfig = { PermanentMACAddress = "xx"; };
      linkConfig = { Name = "ge3"; };
    };
    "04-ge4" = {
      matchConfig = { PermanentMACAddress = "xx"; };
      linkConfig = { Name = "ge4"; };
    };
    "05-ge5" = {
      matchConfig = { PermanentMACAddress = "xx"; };
      linkConfig = { Name = "ge5"; };
    };
    "06-ge6" = {
      matchConfig = { PermanentMACAddress = "xx"; };
      linkConfig = { Name = "ge6"; };
    };
    "07-ge7" = {
      matchConfig = { PermanentMACAddress = "xx"; };
      linkConfig = { Name = "ge7"; };
    };
    "08-xg0" = {
      matchConfig = { PermanentMACAddress = "xx"; };
      linkConfig = { Name = "xg0"; };
    };
    "09-xg1" = {
      matchConfig = { PermanentMACAddress = "xx"; };
      linkConfig = { Name = "xg1"; };
    };
  };

  networking.useDHCP = false;
  networking.interfaces.ge0.useDHCP = true;
  networking.interfaces.ge1.useDHCP = false;
  networking.interfaces.ge2.useDHCP = false;
  networking.interfaces.ge3.useDHCP = false;
  networking.interfaces.ge4.useDHCP = false;
  networking.interfaces.ge5.useDHCP = false;
  networking.interfaces.ge6.useDHCP = false;
  networking.interfaces.ge7.useDHCP = false;
  networking.interfaces.xg0.useDHCP = false;
  networking.interfaces.xg1.useDHCP = false;

  networking.dhcpcd.extraConfig = "noipv6";

  networking.interfaces.br1.ipv4.addresses =
    [ { address = "10.0.0.1"; prefixLength = 24; } ];

  networking.bridges = {
    br1.interfaces = ["ge1" "ge2" "ge3" "ge4" "ge5" "ge6" "ge7" "xg0" "xg1"];
  };

  networking.nat.enable = true;
  networking.nat.externalInterface = "ge0";
  networking.nat.internalInterfaces = [ "br1" ];

  networking.firewall.checkReversePath = true;
  networking.firewall.logReversePathDrops = true;

  networking.firewall.allowedTCPPorts = [
    22  # ssh
  ];

  networking.firewall.interfaces.br1.allowedTCPPorts = [
    22
  ];

  i18n.defaultLocale = "en_US.UTF-8";

  users.users.yourusername= {
    isNormalUser = true;
    extraGroups = [ "wheel" ];
  };

  #security.sudo.wheelNeedsPassword = false;

  environment.systemPackages = with pkgs; [
    vim wget htop git ethtool tcpdump
  ];

  services.openssh.enable = true;
  services.openssh.openFirewall = false;
  services.openssh.listenAddresses = [ { addr = "0.0.0.0"; port = 22; } ];
  services.openssh.settings.PermitRootLogin = "no";

  services.irqbalance.enable = true;

  system.stateVersion = "unstable";
}

Do note that I went out of my way to both pin the MAC addresses and also to specify where I wanted DHCP to come in. I did that simply because I wanted to make sure it didn’t change on me, but also it renames the interfaces when the system comes up, so instead of those annoying procedurally generated names, I get interface names that match the printing on the front panel. Neat! One could totally set this up in other operating systems too, it’s just using a systemd feature.

I’d previously been compiling the kernel on the box with some kernel features suggested in the documentation - here’s what you would add to configuration.nix if you really want that; I was able to query DPAA2 with it but to be honest it was not useful to me in the end so I just boot the normal upstream kernel. It takes several hours to compile on the Ten64, but it can be done with just 8GB of RAM if you’re patient.

boot.kernelPatches = [ {
  name = "ten64-patches";
  patch = null;
  extraConfig = ''
    ARM64_VA_BITS_48 y
    ARM64_VA_BITS 48
    ARCH_LAYERSCAPE y
    FSL_MC_BUS y
    CRYPTO_HW y
    CRYPTO_DEV_FSL_CAAM_COMMON y
    CRYPTO_DEV_FSL_CAAM_CRYPTO_API_DESC y
    CRYPTO_DEV_FSL_CAAM_AHASH_API_DESC y
    CRYPTO_DEV_FSL_CAAM m
    CRYPTO_DEV_FSL_CAAM_JR m
    CRYPTO_DEV_FSL_CAAM_JR m
    CRYPTO_DEV_FSL_CAAM_CRYPTO_API y
    CRYPTO_DEV_FSL_CAAM_CRYPTO_API_QI y
    CRYPTO_DEV_FSL_CAAM_AHASH_API y
    CRYPTO_DEV_FSL_CAAM_PKC_API y
    CRYPTO_DEV_FSL_CAAM_RNG_API y
    CRYPTO_DEV_FSL_DPAA2_CAAM y
    GPIO_MPC8XXX y
    NET_VENDOR_FREESCALE y
    FSL_PQ_MDIO y
    FSL_DPAA_ETH m
    FSL_DPAA2_ETH m
    FSL_DPAA2_PTP_CLOCk m
    PCI_LAYERSCAPE y
    PCIE_LAYERSCAPE y
    FSL_MC_DPIO y
    FSL_MC_UAPI_SUPPORT y
  '';
} ];

This isn’t exactly a complete setup guide but I hope that it helps indicate some of the changes you’ll want to make when installing NixOS.