Building a dual 10 Gbit + 2.5G + 1G router with Debian and a Lenovo M720q

Last updated on Thursday, 23 May 2024 27 min read

Around two years ago, my parents’ internet connection was upgraded, as a new ISP became active in their area. Instead of just cable or DSL, they can now also use a fiber connection. Initially, they got access to a symmetric 1 Gbps connection. Fast-forward to this month, the ISPs connection was upgraded to XGS-PON, offering up to 8 Gbps speeds up and down. Unfortunately, at my own house, I am still stuck with slow cable and DSL speeds. Therefore, I run my self-hosted services on their connection. However, the ISP router is very limited in its capability and doesn’t offer a bridge option, which interferes with my parents’ and my self-hosting needs. Therefore, I have decided to build my own using general-purpose hardware and fully open source software. This post elaborately details how I did this and is an anonymised, edited version of my personal documentation. To make navigating through it easier, you can make use of the Table of Contents listed below.

This router is currently running on the Delta / Caiway network. However, I wrote the config in the main part of the post to be as generic as possible. For specific setup instructions for this provider, please refer to the section at the bottom of this post. If you are not using the Delta Fiber network, but do need to use VLANs to setup an internet connection with another provider, the section may also be useful as it contains instructions on how to do so.

Table of Contents

As the software stack, many people choose to use PFsense, OPNsense or OpenWRT when building a DIY router. After evaluating all three, I decided I didn’t like any of them. Although all three offer great GUIs, they also have their limitations which would still force you to use the command line. PFsense and OPNsense are BSD-based and when used as a 10 gbps capable router, they require manual tweaking of kernel settings and trial-and-error to get actually good and stable throughput. The advanced GUI these two options offer very much appeal to me, however, I’m unfamiliar with BSD and don’t want to learn about the intricacies of an entirely new OS and tuning it using the CLI. Moreover, generally, the driver support on BSD is not as good as it is on Linux, especially for newer and consumer-oriented hardware. Finally, another fiber provider will probably become active in the near future and I’d like to be able to switch. The other provider uses PPPoE and on BSD, the PPPoE daemon operates single-threaded, whereas on Linux it is multi-threaded. With single-threaded PPPoE I would not be able to saturate a 8 Gbps connection. Therefore, I decided against using PFsense or OPNsense. OpenWRT, on the other hand, is Linux-based and doesn’t have these issues. However, OpenWRT does not support automatically upgrading and retaining installed packages when doing so. Therefore, OpenWRT was also out of the question.

With the conclusions that:

  1. I didn’t want to learn the intricacies of a new OS specifically for a router
  2. PFsense and OPNsense would require me to dive into their CLIs and have single-threaded PPPoE
  3. OpenWRT is not easily upgradable.
  4. All three platforms would still require me to occasionally dive into the CLI.

I decided to look at vanilla Debian instead. Although it doesn’t have a ready-made router GUI, I do have many years of experience with it and it is easily upgradable. Moreover, out-of-the-box, Linux doesn’t require tuning the network stack to get proper results. It has multi-threaded PPPoE and generally supports more and newer hardware. The trade-off I’m making here is that I’d rather have good hardware support, upgradability and tuning defaults than an advanced but incomplete GUI.

Hardware

Having decided on the software stack, I now needed to find hardware to run it on. To directly plug into the fiber connection provided by the ISP, I needed a machine supporting at least two SFP+ transceiver modules.

SFP (small form-factor pluggable) is a standard supporting hot-swappable ethernet and fiber modules up to 1 Gbps. SFP+ is an updated version, backwards compatible with SFP, supporting up to 10 Gbps modules.

The first SFP+ slot will be used to connect to the outside fiber connection, the second to connect to a network switch.

To select hardware, initially, I looked at various firewall mini pc appliances sold on AliExpress. On AliExpress, for example, Qotom sells the passively cooled or rack mount Q20332G9 based on the eight-core Intel Atom C3758 / C3758R starting at around €400,- excluding storage and RAM (ECC supported). The Q20332G9 supports four SFP+ ports and five 2.5 GBe ports. Additionally, it has a SIM slot which could be used to expand it with a wireless modem for backup purposes. Finally, it supports 2 SATA drives, 2 NVMe drives and a mini SAS interface so it could even be used as a combined router / small NAS. I think I would have bought this appliance if this was my own connection and the budget set for this project was sufficient for that, but unfortunately, adding all additional required components up made me realise it wasn’t.

I put this plan of building a DIY router on hold for a while, until I learned about ServeTheHome’s TinyMiniMicro project.

Lenovo M720q

After reading the Lenovo tiny reference thread on their forums, I discoverd that the Lenovo M720q supports low-profile ordinary PCIe cards with an optional riser. This made me realise that I could try to find an affordable SFP+ PCIe card and turn this 1L PC into a router.

Armed with this knowledge, I scoured through online marketplaces and managed to pick up a used Lenovo M720q with the following specs for €110:

  • CPU: i3 8100T, 4 cores, 4 threads, 3.10GHz base frequency.
  • RAM: 2x4GB DDR4 RAM
  • Storage: 256GB NVMe SSD
  • Wired networking: Intel I219-V 1 GBit ethernet adapter
  • WiFi: Intel Dual Band Wireless-AC 3165

PCIe riser

If you want to place a PCIe card into an M720q, you lose the ability to equip it with a SATA drive as these share the same physical space. As I requested the seller I bought the M720q from to equip it with an NVMe rather than the standard SATA drive, I fortunately didn’t have to replace it.

To fit a PCIe card into a Lenovo M720q, a riser card is required. It is important to buy the exact riser article number required by the M720q. Other cards suited for other SKUs may fit mechanically but will be incompatible electrically and can destroy your motherboard. The specific type you want for the M720q is the 01AJ940, available through AliExpress and costs between €10-€20.

SFP+ card

The M720q is a small 1l PC in which only low-profile PCIe cards can fit. Moreover, it uses a custom bracket to secure PCIe cards and due to its size is limited in how much heat it can dissipate. Therefore, I wanted to use an SFP+ card from the Intel X710 series. Compared to previous cards such as the X550, it is a little bit more picky in accepting off-brand SFP+ modules, but this can easily be mitigated by disabling the module white list using the xl710-unlocker software (this action is reversible). On the flip side, it was launched a lot more recently and is based on newer chips, requiring far less power and dissipating far less heat, making it fit for purpose. Additionally, the X710-based cards support ASPM, while many cheaper, older cards don’t, which can cause the entire system to have a much higher idle power consumption. Higher power consumption also means more heat. Especially because the M720q will only have passive cooling for the PCIe card, it is important to keep power consumption as low as possible.

After saving an eBay search for a few weeks, I manage to import a Dell-branded Intel X710-DA2 card (type number 5N7Y5), sporting 2 SFP+ slots, for $45 from the US. Including shipping and import taxes, I paid $79.57 in total.

Although I can’t secure the card in place using screws, as I don’t have a custom bracket fitting the M720q, the stock PCIe bracket has a folded tab, which fits exactly over the cover of the PC when closed. This way, the card is well supported.

Extra 2.5GBe NIC

In addition to the M.2 NVMe drive slot, The Lenovo M720q has a M.2 A+E key slot, used for the WiFi module. As I didn’t need any WiFi capabilities on this router as this is handled by separate access points, I decided to replace it with a 2.5GBe wired ethernet card, the Realtek RTL8125BG, bought on AliExpress.

Replacing the wifi card with the ethernet card turned out to be a bit of a hassle, as the screw securing it was completely stuck. After many tries to get it out with a screwdriver, I resorted to carefully removing the motherboard from the case and drilling out the screw with a small drill, 2.5 mm if I recall. By doing this and then vacuuming carefully to remove any remaining metal splinters, I managed to get it out without damaging the threads.

Creating a nice bracket for this extra NIC is still on the TODO list, as for now, the RJ45 jack is still dangling out of the machine from its riser cable.

VESA mounts

Last but not least, on Black Friday, I managed to order the original VESA brackets for the M720q and its power supply from Lenovo for around €20 including shipping. This way, I can conveniently hang it in the meter cupboard in which the provider’s fiber connection is terminated.

Total cost of hardware

In total, I spent the following to gather all hardware, excluding SFP modules:

Software

First, I installed a Debian Bookworm base system with a BTRFS root file system to support efficient Incus reflink snapshots. Normally, I always install systems with full disk encryption enabled. For the router, I don’t, as I don’t want to deal with unlocking disks to get a working internet connection after a reboot or power failure. I did try out using the TPM module or other frameworks such as Clevis and Tang to automatically unlock the system, but found the implementations I was able to make with those to be too brittle for router purposes. For me, a router should just work and reboot automatically, without requiring any user invention. Therefore, I will lock down the storage differently, by physically securing it behind lock and key instead of relying on cryptography.

BIOS

Go into the BIOS and

  • Enable Secure Boot.
  • Delete all old keys and set Secure Boot to setup mode.

Install Debian

I created a USB stick with the Debian netinst image and Balena Etcher. I inserted this stick into the M720q and booted from it by pressing F12 on boot.

As I’m assuming you’re somewhat familiar with installing Debian when following this tutorial, I have summarised the install process in bullet points.

  • On start of the installer, go to advanced: Start the expert installer.
  • Users
    • Disable login as root user
    • Create a user account with your desired admin username and password.
  • enter, enter, enter
  • Partitioning disks
    • Partition disks: Manual
    • Select /dev/nvme0n1p1, hit enter to create new partition table.
      • Select Yes, choose gpt.
    • Go to FREE SPACE, hit enter to create new partition.
      • 1G, beginning.
      • Name: ESP
      • Use as: EFI System Partition
    • Go to largest FREE SPACE, hit enter to create new partition
      • 1G, beginning.
      • Name: Boot
      • Use as: Ext4 journaling file system
      • Format the partiton: yes, format it
      • Mount point: /boot
    • Go to largest FREE SPACE, hit enter to create new partition
      • Partition size: max
      • Use as: btrfs journaling file system
      • Mount point: /
    • Finish partitioning and write changes to disk.
    • SWAP message: choose No
    • Write changes to disk: Yes
  • Install base system
    • Choose linux-image-amd64, generic
    • Enable apt repositories you desire.
    • Choose install security updates automatically.
    • Participate in the package usage survey: Yes.
      • You’re using free software. This is one of the smallest possible ways to contribute back to the free software you’re using.
    • Disable desktop environment, enable SSH, standard system utilities.
  • Bootloader
    • Force GRUB: yes
    • Update NVRAM: yes
    • Run os prober: no
  • Is the system clock set to UTC: yes
  • Reboot: Continue

After rebooting, add your SSH key to .ssh/authorized_keys. Optionally, disable sudo password and install Zsh.

Harden SSH

sudo sed -i "s/#PasswordAuthentication yes/PasswordAuthentication no/g" /etc/ssh/sshd_config
sudo sed -i "s/#PermitRootLogin prohibit-password/PermitRootLogin no/g" /etc/ssh/sshd_config
sudo sed -i "s/AcceptEnv LANG LC_*/#AcceptEnv LANG LC_*/g" /etc/ssh/sshd_config

sudo systemctl restart ssh

BTRFS compression

BTRFS supports filesystem compression. Optionally, this can be enabled using these steps.

sudo nano /etc/fstab

Add to mount options

,compress=zstd:2

sudo reboot

Realtek drivers

Without Realtek’s proprietary drivers and manually enabling ASPM, the Realtek RTL8125BG will prevent the system to be able to reach lower power states than C3.

sudo apt install firmware-linux firmware-realtek

Reboot

sudo reboot

sudo lspci -vv

sudo sh -c "echo 1 > /sys/bus/pci/devices/0000:03:00.0/link/l1_aspm"

Check the output for the Realtek RTL8125 and check if ASPM is enabled.

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
    [...]
    LnkCtl: ASPM L1 Enabled; [....]
    [...]

Let’s make sure this happens on every reboot

sudo nano /etc/systemd/system/enable_aspm_realtek.service

[Unit]
Description=Enable RTL8125 ASPM power saving

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo 1 > /sys/bus/pci/devices/0000:03:00.0/link/l1_aspm'

[Install]
WantedBy=multi-user.target

sudo systemctl enable enable_aspm_realtek --now

sudo systemctl status enable_aspm_realtek

Git and miscellaneous applications

Git is a version control system. JQ is a json processing tool. Nmap is a network scanner.

sudo apt install git build-essential curl bc nano sudo htop jq nmap -y

Tops

These utilities can help in providing various diagnostic information.

sudo apt install htop powertop lm-sensors strace fancontrol read-edid i2c-tools cpufrequtils

Upgrade the X710-DA2 firmware

Through eBay, I managed to buy a Dell-branded Intel X710-DA2. Unfortunately, it was not possible to upgrade this using the firmware provided by Intel. Fortunately, Dell does not put their firmware upgrades behind a paywall. However, when I tried to upgrade the X710-DA2 on the M720q, the Dell upgrade didn’t want to upgrade the card. I resolved that by unpacking the Dell binary.

First, I downloaded the Dell firmware upgrade from the Dell website, which I copied to the router using SFTP. I had to do it this way, as for some reason, Dell forbids direct downloads using curl or wget. Next, I allowed the binary to be executed using chmod +x and extracted it to a firmware folder.

chmod +x ./Network_Firmware_1R0W0_LN_22.5.7_A00.BIN
sudo ./Network_Firmware_1R0W0_LN_22.5.7_A00.BIN --extract ./firmware

In the extracted firmware folder, a Dell-specific Intel nvmupdate64e binary is included. I upgraded the Dell 5N7Y5 Intel X710-DA2 using this file by executing it and following the steps.

cd firmware

sudo ./nvmupdate64e

After upgrading the firmware, reboot.

sudo reboot

Unlock the X710-DA2

To use the Intel X710-DA2 with non-whitelisted SFP modules, the whitelist needs to be disabled (this action is reversible).

Clone the xl710-unlocker and build the program

git clone https://github.com/bibigon812/xl710-unlocker

cd xl710-unlocker

make

Unlock your X710-DA2

./xl710_unlock

Select the network interface card you’d like to unlock. Select y when it asks you to confirm.

After unlocking the X710-DA2, reboot.

sudo reboot

WireGuard support

WireGuard will provide VPN tunnel capabilities to the router.

sudo apt install wireguard -y
sudo modprobe wireguard

Unbound

Unbound is a DNS resolver, which we’ll use to provide DNS records to the network. We’ll use systemd-resolved on the local system for easy integration with systemd-networkd.

sudo apt install unbound

sudo nano /etc/unbound/unbound.conf

server:

interface: 127.0.0.1
interface: ::1
access-control: 127.0.0.0/8 allow
access-control: ::1/128 allow

# enable remote-control (useful when utilising monitoring tools such as netdata)
remote-control:
    control-enable: yes


sudo unbound-checkconf

sudo systemctl restart unbound

sudo apt install systemd-resolved

add

sudo nano /etc/systemd/resolved.conf

DNS=127.0.0.1 ::1
DNSSEC=yes

sudo systemctl restart systemd-resolved

Check:

cat /etc/resolv.conf

sudo resolvectl status

VLAN support

Many providers require VLAN support to setup a connection. Moreover, using VLANs can help in splitting up the internal network into different segments without having to pull extra cables.

sudo apt install vlan

sudo modprobe 8021q
lsmod | grep 8021q

sudo sh -c "echo 8021q >> /etc/modules"

A useful resource for understanding VLANs is the Arch Linux wiki.

Install bridge-utils & migrate to systemd-networkd

Network bridges can be used to pool network interfaces together. Systemd-networkd is the modern way on Debian to do networking. As of writing, Debian Bookworm still uses ifupdown by default, we move away from this.

sudo apt install bridge-utils

Move / backup the old ifupdown configuration.

sudo mv /etc/network/interfaces /etc/network/interfaces.save

Interface settings

Add the new systemd-networkd interfaces. In my setup, the following interfaces are available:

  • eno1 is the 1 GBit Intel I219-V NIC
    • Right now, after installing Debian, this is the default network interface. This interface will be reconfigured to provide the LAN network, by attaching it to br0.
  • enp3s0 is the 2.5 Gbit Realtek RTL8125BG
    • This interface will initially be configured, so the router can be a client, attached to the existing ISP-provided router. After everything works, this could be changed by adding enp3s0 to br0, just like eno1 and enp1s0f1.
  • enp1s0f0 SFP+ slot 1
    • enp1s0f0 will be used as the WAN interface.
  • enp1s0f1 SFP+ slot 2
    • enp1s0f1 will be used as a LAN interface, attached to br0.

I am first setting up the eno1 and enp1s0f1 interfaces as LAN interfaces, providing a DHCP server and DNS resolver. The enp3s0 interface is temporarily set up as a DHCP client to the existing ISP-provided router, but is, in terms of firewall settings, also part of the LAN zone.

sudo nano /etc/systemd/network/br0.netdev

[NetDev]
Name=br0
Description="Home LAN bridge"
SkipForwardingDelay=true
Kind=bridge

sudo nano /etc/systemd/network/br0.network

“SkipForwardingDelay is important because it will allow the bridge interface to come up when the computer boots even if it cannot contact the gateway.”

[Match]
Name=br0

[Network]
Description="Home LAN bridge"
Address=192.168.70.1/24
IPMasquerade=both
DHCPServer=yes
ConfigureWithoutCarrier=true
ActivationPolicy=always-up

[DHCPServer]
PoolOffset=20
PoolSize=150
DNS=192.168.70.1

The ActivationPolicy=always-up argument is needed to ensure the network is still brought up even when devices on the other end are not initialised yet. Manually setting IPForward=yes is not required in br0.network, as this is already implied by IPMasquerade.

Bridge the eno1 interface into br0

sudo nano /etc/systemd/network/eno1.network

[Match]
Name=eno1

[Network]
Bridge=br0

Do the same for the enp1s0f1 interface.

sudo nano /etc/systemd/network/enp1s0f1.network

[Match]
Name=enp1s0f1

[Network]
Bridge=br0

sudo nano /etc/systemd/network/enp3s0.network

Bridge=br0 is commented out, as for now, we don’t want this to be in the LAN network and offering the DHCP server. Instead, we want the router on enp3s0 to be a DHCP client in an existing network.

[Match]
Name=enp3s0

[Network]
Description="WAN Client via modem"
DHCP=ipv4
MTUBytes=1500
#Bridge=br0

sudo systemctl enable systemd-networkd
sudo systemctl disable networking 

sudo reboot

While rebooting, make sure to change the network cable connecting to your existing router from eno1 to enp3s0.

Expand Unbound config to resolve for LAN

We added the LAN home and internal network to the router. Let’s allow the router to resolve DNS resolving requests for these networks.

sudo nano /etc/unbound/unbound.conf

interface: 192.168.70.1
access-control: 192.168.70.0/24 allow

interface: 10.10.6.1
access-control: 10.10.6.0/24 allow

sudo unbound-checkconf

sudo systemctl restart unbound

Additional admin packages

These tools can help in diagnosing network issues.

sudo apt install iperf3 net-tools tmux curl git pwgen build-essential gpg

Say “yes” to enable iperf3 system daemon.

Fail2ban

Fail2ban is a daemon to ban hosts that cause multiple authentication errors, e.g. on SSH. This is a great extra defense against bots trying to break into the router.

sudo apt install fail2ban

By default, Fail2ban on Debian Bookworm has a bug. Adjust the config to resolve it.

sudo nano /etc/fail2ban/jail.conf

backend = systemd

sudo systemctl restart fail2ban

Cockpit

Cockpit is a web-based graphical interface for servers.

Cockpits official documentation recommends to install from the backports repository. We don’t want it to install NetworkManager as we are directly using systemd-networkd, therefore, we use the --no-install-recommends flag.

sudo apt install cockpit -t bookworm-backports --no-install-recommends

Cockpit provides many additional modules. To list available plugins, use:

apt search cockpit-

Install software packages addon

sudo apt install cockpit-packagekit -t bookworm-backports

Install Cockpit performance co-pilot

sudo apt install cockpit-pcp -t bookworm-backports

Install Cockpit diagnostic reports

sudo apt install cockpit-sosreport -t bookworm-backports

Create BTRFS snapshot

To save the current state of the system, BTRFS supports making snapshots. These can be great when testing changes and provide the capability of easily rolling back to a previous state. In other words, if you make a configuration mistake, this can save you from having to reinstall.

sudo mkdir /.snapshots
sudo chmod 750 /.snapshots
sudo btrfs subvolume snapshot / /.snapshots/base-install

Incus

Incus is a powerful system container and virtual machine manager. I don’t run many things on my router, but do like to run some miscellaneous Docker containers. However, I want to keep the base OS of the router as simple and clean as possible. Therefore, I install as little as possible on it, and use Incus to separate the Docker services from the main router by running it in unprivileged containers as detailed in this “nesting Incus in Docker” post.

First, I add a new network bridge.

sudo nano /etc/systemd/network/br1.netdev

[NetDev]
Name=br1
Description="Incus LAN bridge"
SkipForwardingDelay=true
Kind=bridge

sudo nano /etc/systemd/network/br1.network

[Match]
Name=br1

[Network]
Description="Incus LAN bridge"
Address=10.10.6.1/24
IPMasquerade=both
DHCPServer=yes
ConfigureWithoutCarrier=true
ActivationPolicy=always-up

[DHCPServer]
PoolOffset=20
PoolSize=150
DNS=10.10.6.1

Add the Incus apt repository.

curl -fsSL https://pkgs.zabbly.com/key.asc | gpg --show-keys --fingerprint

sudo mkdir -p /etc/apt/keyrings/
sudo curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc

sudo sh -c 'cat <<EOF > /etc/apt/sources.list.d/zabbly-incus-stable.sources
Enabled: yes
Types: deb
URIs: https://pkgs.zabbly.com/incus/stable
Suites: $(. /etc/os-release && echo ${VERSION_CODENAME})
Components: main
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/zabbly.asc

EOF'

sudo apt update

Install Incus and the Incus UI.

sudo apt install incus incus-ui-canonical

Add your admin user to the incus-admin group.

sudo gpasswd -a $USER incus-admin

Logout and login again. Initialise Incus.

incus admin init

Would you like to use clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (btrfs, dir) [default=btrfs]: 
Would you like to create a new btrfs subvolume under /var/lib/incus? (yes/no) [default=yes]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: no
Would you like to use an existing bridge or host interface? (yes/no) [default=no]: yes
Name of the existing bridge or host interface: br1
Would you like the server to be available over the network? (yes/no) [default=no]: yes
Address to bind to (not including port) [default=all]:             
Port to bind to [default=8443]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: no
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]:

Change some kernel settings to optimise Incus’s operation and improve stability by preventing hitting server limits.

sudo nano /etc/sysctl.conf

Add

# Incus
# https://linuxcontainers.org/incus/docs/main/reference/server_settings/
fs.aio-max-nr = 524288
fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_instances = 1048576
fs.inotify.max_user_watches = 1048576
kernel.dmesg_restrict = 1
kernel.keys.maxbytes = 2000000
kernel.keys.maxkeys = 2000
net.core.bpf_jit_limit = 1000000000
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv6.neigh.default.gc_thresh3 = 8192
vm.max_map_count = 262144

To propagate the kernel settings instantly, run

sudo sysctl -p

Increase the limits for Incus.

sudo nano /etc/security/limits.conf

# Incus
# https://linuxcontainers.org/incus/docs/main/reference/server_settings/
*               soft    nofile          1048576
*               hard    nofile          1048576
root            soft    nofile          1048576
root            hard    nofile          1048576
*               soft    memlock         unlimited
*               hard    memlock         unlimited
root            soft    memlock         unlimited
root            hard    memlock         unlimited

sudo systemctl restart incus

Incus bash completion, assuming you’ve installed zsh.

incus completion zsh > .incus_completion.zsh

echo "source ~/.incus_completion.zsh" >> ~/.zshrc

Firewalld

Firewalld is a dynamically managed firewall.

The firewall is setup with three zones:

  • Public
    • WAN
  • Home
    • Home LAN
  • Internal
    • Incus containers and IoT devices
sudo apt install firewalld

Add the br0 interface to the home zone. Allow DNS and DHCP requests from the home zone. Set up masquerading for traffic leaving the public zone. Add the WAN interface to the public zone. Reload FirewallD and check the new setup. Add enp3s0 separately, as it is not currently part of the home LAN but will be in the future. For now it’s on the existing LAN, which can also be trusted, so this is fine.

My ISP, Delta, doesn’t provide an untagged internet connection. Instead it provides their internet connection on VLAN 100. Therefore, instead of adding enp1s0f0, I need to add enp1s0f0.100 to the public zone. If you are not on the Delta Fiber network, just use enp1s0f0.

sudo firewall-cmd --zone=home --add-interface=br0 --permanent
sudo firewall-cmd --zone=home --add-interface=enp3s0 --permanent
sudo firewall-cmd --zone=home --add-service=dns --permanent
sudo firewall-cmd --zone=home --add-service=dhcp --permanent

sudo firewall-cmd --zone=internal --add-interface=br1 --permanent
sudo firewall-cmd --zone=internal --add-service=dns --permanent
sudo firewall-cmd --zone=internal --add-service=dhcp --permanent


sudo firewall-cmd --zone=public --add-masquerade --permanent
sudo firewall-cmd --zone=public --add-interface=enp1s0f0.100 --permanent

sudo firewall-cmd --reload

sudo firewall-cmd --list-all-zones

With these settings, no connectivity from the home or internal zone to the public zone is possible as FirewallD doesn’t allow traffic between zones by default. In other words, the router doesn’t provide an WAN internet connections to clients on the LAN. This can be solved by adding a policy.

sudo firewall-cmd --new-policy internet-access --permanent
sudo firewall-cmd --policy internet-access --add-ingress-zone home --permanent
sudo firewall-cmd --policy internet-access --add-ingress-zone internal --permanent
sudo firewall-cmd --policy internet-access --add-egress-zone public --permanent
sudo firewall-cmd --policy internet-access --set-target ACCEPT --permanent

sudo firewall-cmd --reload

sudo firewall-cmd --list-all-policies

Allow iPerf3 from LAN. This way, it’s possible to do reliable speedtests on the internal network.

sudo firewall-offline-cmd --new-service=iperf3
sudo firewall-offline-cmd --service=iperf3 --set-description="iPerf3 server"
sudo firewall-offline-cmd --service=iperf3 --add-port=5201/tcp
sudo firewall-offline-cmd --service=iperf3 --add-port=5201/udp

sudo firewall-cmd --zone=home --add-service=iperf3 --permanent

sudo firewall-cmd --reload

Optional: Set up the router for use on the Delta Fiber network

In my setup, the following interfaces are available:

  • eno1 is the 1 GBit Intel I219-V NIC
    • LAN interface, providing DHCP server
  • enp3s0 is the 2.5 Gbit Realtek RTL8125BG
    • LAN interface, configured as DHCP client in existing LAN network.
  • enp1s0f0 SFP+ slot 1
    • WAN interface, at the start of this section still unconfigured.
  • enp1s0f1 SFP+ slot 2
    • LAN interface, providing DHCP server

To limit the downtime of the internet connection, I first configured the router and only swapped out the physical connections afterwards.

Setting up the WAN connection on the router

Until now, I’ve used the router as a client on my existing ISP-provided router, attached to enp3s0. The enp1s0f0 interface is not used untagged. It does, however, need to be configured, in order for VLANs to work. The order of interface configuration is important. Systemd-networkd processes config files in lexicographical order, which is why I can use numbers to determine their order.

Firstly , the enp1s0f0 interface needs to be set up.

sudo nano /etc/systemd/network/10-enp1s0f0.network

[Match]
Name=enp1s0f0

[Link]
RequiredForOnline=yes

[Network]
DHCP=no
VLAN=enp1s0f0.100
LinkLocalAddressing=no

Secondly, a VLAN 100 can be added.

sudo nano /etc/systemd/network/20-enp1s0f0.100.netdev

[NetDev]
Name=enp1s0f0.100
Kind=vlan

[VLAN]
Id=100

Finally, the network on VLAN 100 can be configured. We set it to use DHCP to retrieve an IP address from the provider, but set UseNTP and UseDNS to no as we want to use our own DNS resolver, Unbound.

sudo nano /etc/systemd/network/30-enp1s0f0.100.network

[Match]
Name=enp1s0f0.100

[Link]
RequiredForOnline=yes

[Network]
DHCP=ipv4
LinkLocalAddressing=no

[DHCPv4]
SendHostname=no
UseNTP=no
UseDNS=no

Although Delta specifies separate VLANs for IPTV in their documentation, I have not to configured keep the setup simple and have not experienced any issues. They also provide a separate VLAN for VOIP, but as I don’t use their VOIP services, I haven’t configured that either.

Physically connecting the router to the Delta AON / P2P network

At the moment of writing this section, I am still om the old P2P / AON 1 Gbit network. This means that I can get a connection using this cheap bidirectional 1 GBit SFP module, sold by FS.com, which should be ordered with:

  • Brand: Intel
  • Device number: X710-DA2
  • Connector: SC

The SFP module needs to be connected with the Attema FTU (fiber termination unit), through which the Delta Fiber connection is provided. A matching cable is required as well, which can be attached in the existing Attema connector sled of the ISP-provided router to connect it to the FTU. The blue connector should go into the SFP module. Do NOT switch these ends around, as the green end is using SC APC standard, while the blue end follows the SC UPC standard.

In total, the SFP module, fiber cable and shipping cost me €20.70.

Carefully unplug the ISP-provided router and connect the new router. An internet connection should automatically be established.

Physically connecting the router to the Delta XGS-PON network

Next week, I will be migrated to the Delta XGS-PON network. This new network technology allows for much faster speeds, currently up to 8Gbps. However, it also means I need to invest in a newer, more expensive SFP+ module.

People are reporting good experiences with the Zaram XGS-PON ONU SFP+, which currently retails for around €130.00. The cable remains the same.

Unfortunately, the new technology also requires you to register the SFP+ module with Delta using the steps outlined in this forum post. I have not tried this myself yet.

Optional: WireGuard client

I am using a few VPSes and wanted this router to be a VPN client to one of these servers, below you can find how to configure the router as a WireGuard client device.

sudo nano /etc/systemd/network/wg-oracle.netdev

[NetDev]
Name=wg-vps
Kind=wireguard
Description=wg-vps - wireguard tunnel

[WireGuard]
PrivateKey = REPLACE_THIS

[WireGuardPeer]
PublicKey = REPLACE_THIS
PresharedKey = REPLACE_THIS
AllowedIPs = 10.10.84.1/24, 10.10.88.1/24
Endpoint = example.example.org:51820
PersistentKeepalive = 25

sudo nano /etc/systemd/network/wg-vps.network

[Match]
Name=wg-vps

[Network]
Address=10.10.84.5/24

TODO firewall settings

Optional: WireGuard server

When I’m away from home, I’d still like to be able to login to my home network using some devices. To do so, I also add a WireGuard server to the router using the following steps.

First, WireGuard keys can be generated manually by running the following commands as root:

wg genkey | (umask 0077 && tee "/etc/wireguard/peer_A.key") | wg pubkey > "/etc/wireguard/peer_A.pub"
wg genpsk | (umask 0077 && tee "/etc/wireguard/peer_A.psk")

This becomes tiresome with multiple hosts. Let’s make it easier to generate keys by creating a helper script.

sudo nano /usr/local/sbin/wg-create-keys

#!/bin/bash

# Directory where the keys will be stored
KEY_DIR="/etc/wireguard"

# Check if an argument was provided
if [ $# -eq 0 ]; then
    echo "Usage: $0 <peer_name>"
    exit 1
fi

# Use the first argument as the peer name
PEER_NAME=$1

# Check if the KEY_DIR exists, if not create it
if [ ! -d "$KEY_DIR" ]; then
    echo "$KEY_DIR does not exist, creating it..."
    mkdir -p "$KEY_DIR"
    chmod 700 "$KEY_DIR"
fi

# Function to generate keys
generate_keys() {
    # Generate the private key, set permissions, and output to the specified peer name files
    wg genkey | (umask 0077 && tee "$KEY_DIR/${PEER_NAME}.key") | wg pubkey > "$KEY_DIR/${PEER_NAME}.pub"

    # Generate a preshared key and output to the specified peer name file
    wg genpsk | (umask 0077 && tee "$KEY_DIR/${PEER_NAME}.psk")

    echo "Generated keys for $PEER_NAME stored in $KEY_DIR:"
    echo "Private Key: ${PEER_NAME}.key"
    echo "Public Key: ${PEER_NAME}.pub"
    echo "Preshared Key: ${PEER_NAME}.psk"
}

# Check if any of the files already exist
if [ -f "$KEY_DIR/${PEER_NAME}.key" ] || [ -f "$KEY_DIR/${PEER_NAME}.pub" ] || [ -f "$KEY_DIR/${PEER_NAME}.psk" ]; then
    echo "One or more key files for $PEER_NAME already exist in $KEY_DIR."
    echo "Do you want to overwrite them? (y/n)"
    read answer
    if [ "$answer" = "y" ]; then
        generate_keys
    else
        echo "Operation cancelled."
    fi
else
    generate_keys
fi


sudo chmod +x /usr/local/sbin/wg-create-keys

Setting up server keys

Let’s create a new WireGuard network interface called “wg0”. We’ll set the WireGuard network IP range to 192.168.14.1/24

sudo wg-create-keys wg0

sudo nano /etc/systemd/network/wg0.netdev

In the PrivateKey field, add the /etc/wireguard/wg0.key private key that was just generated.

[NetDev]
Name = wg0
Kind = wireguard
Description = Wireguard tunnel wg0

[WireGuard]
PrivateKey = <private_key>
ListenPort = 51820

Configuring the WireGuard network

With the network interface added, we now configure its network settings.

sudo nano /etc/systemd/network/wg0.network

[Match]
Name=wg0

[Network]
Address=192.168.14.1/24
IPMasquerade=both

Adding WireGuard VPN clients

sudo wg-create-keys

sudo nano /etc/systemd/network/wg0.netdev

add

[WireGuardPeer]
# Laptop
PublicKey = <public_key>
PresharedKey = <preshared_key>
AllowedIPs = 192.168.14.2/32

sudo systemctl restart systemd-networkd

sudo firewall-cmd --zone=public --add-service=wireguard --permanent
sudo firewall-cmd --zone=home --add-interface=wg0 --permanent
sudo firewall-cmd --reload

Optional: Speedtest

To test network speed to the outside world, we can install the speedtest binary from speedtest.net

sudo apt install speedtest-cli

speedtest

Optional: Clevis & Tang

Although Clevis & Tang were unsuitable for unlocking the router, they can be very useful to automatically unlock Linux machines within a network. This section describes how the Tang server can be setup on the router and how Clevis can be installed on client machines to provide automatic disk unlocking on startup. More information on Clevis & Tang can be found in Redhat’s documentation.

Tang server setup

sudo apt install tang

sudo systemctl edit tangd.socket

Add:

[Socket]
ListenStream=
ListenStream=7500

The empty ListenStream= argument is required to unset tangd listening on port 80.

sudo systemctl daemon-reload
sudo systemctl show tangd.socket -p Listen

Output should be:

Listen=[::]:7500 (Stream)

sudo systemctl restart tangd.socket

sudo firewall-cmd --add-port=7500/tcp --zone=home --permanent
sudo firewall-cmd --reload

Example clevis unlock setup on Debian-based systems

sudo apt install clevis clevis-luks clevis-initramfs

Use blkid to find out the UUID of the disk you’d like to auto unlock.

sudo blkid

Use the UUID to create a new Clevis binding.

sudo clevis luks bind -d /dev/disk/by-uuid/<UUID> tang '{"url":"http://192.168.70.1:7500"}'\n

Update the initramfs with the new Clevis setup

sudo update-initramfs -u -k all

Sources

Linux VLANs

https://www.baeldung.com/linux/vlans-create https://www.sherbers.de/diy-linux-router-part-2-interfaces-dhcp-and-vlan/

Pieter Hollander
Pieter Hollander