Proxmox – Giovanni F. Mazzeo De Santolo https://desantolo.com That italian IT guy Sun, 27 Dec 2020 05:38:52 +0000 en-US hourly 1 https://wordpress.org/?v=6.8.3 123042357 Allowing OpenVPN to create tun device on LXC / Proxmox https://desantolo.com/2018/11/allowing-openvpn-to-create-tun-device-on-lxc-proxmox/ Mon, 19 Nov 2018 01:56:57 +0000 https://desantolo.com/?p=569 Continue reading →]]> Due to built-in security of LXC, trying to setup a tunnel interface inside a container is by blocked by default.

ERROR: Cannot open TUN/TAP dev /dev/net/tun

To allow this for a specific container in Proxmox, we need to make a few tweaks to allow this interface to work in a specific container (we don’t want to allow all containers to be able to setup a tunnel – hackers can hide their tracks using it).

How to do this:

ADD these lines to /etc/pve/lxc/<container-id>.conf

lxc.cgroup.devices.allow = c 10:200 rwm
lxc.hook.autodev = sh -c "modprobe tun; cd ${LXC_ROOTFS_MOUNT}/dev; mkdir net; mknod net/tun c 10 200; chmod 0666 net/tun"

 

]]> 569 OPNsense firewall on Proxmox fix ‘no internet’ https://desantolo.com/2018/11/opnsense-firewall-on-proxmox-fix-no-internet/ Sat, 17 Nov 2018 18:17:25 +0000 https://desantolo.com/?p=566 Continue reading →]]> Quick post to note how I determined and then fixed the internet access issue I was having when I installed OPNsense on Proxmox.

OPNsense virtual machine is configured with VirtiO network drivers.

Other than the obvious “I can’t access anything on the internet” or can’t reach external IP addresses problem I looked at troubleshooting via nmap – because the devices on the network could ping externally (8.8.8.8) and also resolve DNS requests.

In a broken state you may see ‘tcpwrapper’ when testing a known host serving HTTP, like so:

root@test:~# nmap -p 80 -sV 216.58.194.206

Starting Nmap 7.40 ( https://nmap.org ) at 2018-11-17 17:54 UTC

Nmap scan report for sfo03s01-in-f206.1e100.net (216.58.194.206)

Host is up (0.010s latency).

PORT   STATE SERVICE    VERSION

80/tcp open  tcpwrapped

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .

Nmap done: 1 IP address (1 host up) scanned in 7.91 seconds

To fix this issue, ensure that “Disable hardware checksum offload” is  enabled in the OPNsense interface, then reboot the firewall for changes to take effect.

After a reboot, doing another test via nmap will actually respond with HTTP fingerprints, as expected and internet is back.

root@test:~# nmap -p 80 -sV 216.58.194.206

Starting Nmap 7.40 ( https://nmap.org ) at 2018-11-17 18:00 UTC

Nmap scan report for sfo03s01-in-f14.1e100.net (216.58.194.206)

Host is up (0.0096s latency).

PORT   STATE SERVICE VERSION

80/tcp open  http    gws

1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service :

SF-Port80-TCP:V=7.40%I=7%D=11/17%Time=5BF0574C%P=x86_64-pc-linux-gnu%r(Get

SF:Request,8A7A,"HTTP/1\.0\x20200\x20OK\r\nDate:\x20Sat,\x2017\x20Nov\x202

SF:018\x2018:00:43\x20GMT\r\nExpires:\x20-1\r\nCache-Control:\x20private,\

SF:x20max-age=0\r\nContent-Type:\x20text/html;\x20charset=ISO-8859-1\r\nP3

SF:P:\x20CP=\"This\x20is\x20not\x20a\x20P3P\x20policy!\x20See\x20g\.co/p3p

SF:help\x20for\x20more\x20info\.\"\r\nServer:\x20gws\r\nX-XSS-Protection:\

SF:x201;\x20mode=block\r\nX-Frame-Options:\x20SAMEORIGIN\r\nSet-Cookie:\x2

SF:01P_JAR=2018-11-17-18;\x20expires=Mon,\x2017-Dec-2018\x2018:00:43\x20GM

SF:T;\x20path=/;\x20domain=\.google\.com\r\nSet-Cookie:\x20NID=146=0dp1WLb

SF:UhFIr1MIVwhAglx_4O6x-0eJHrmYFTov9a3oFxE2-lZSUI_9mmKBFXQZjYbjKbSRiirLZ-U

SF:cfybTiNQR_vmHD2MY4RBHP-hj4K7oyQX4lXuCgrSU7ESRXiX2Jn0qwoLWvvEItnC2hgDHEb

SF:oLJffQrfiEazdGDp5XppPU;\x20expires=Sun,\x2019-May-2019\x2018:00:43\x20G

SF:MT;\x20path=/;\x20domain=\.google\.com;\x20HttpOnly\r\nAccept-Ranges:\x

SF:20none\r\nVary:\x20Accept-Encoding\r\n\r\n<!doctype\x20html><html\x20it

SF:emscope=\"\"\x20itemtype=\"http://schema\.org/WebPage\"\x20lang=\"en\">

SF:<head><meta\x20content=\"Search\x20the\x20world's\x20information,\x20in

SF:cluding\x20webpages,\x20images,\x20videos\x20and\x20more\.\x20Google\x2

SF:0has\x20ma")%r(HTTPOptions,71B,"HTTP/1\.0\x20405\x20Method\x20Not\x20Al

SF:lowed\r\nAllow:\x20GET,\x20HEAD\r\nDate:\x20Sat,\x2017\x20Nov\x202018\x

SF:2018:00:44\x20GMT\r\nContent-Type:\x20text/html;\x20charset=UTF-8\r\nSe

SF:rver:\x20gws\r\nContent-Length:\x201592\r\nX-XSS-Protection:\x201;\x20m

SF:ode=block\r\nX-Frame-Options:\x20SAMEORIGIN\r\n\r\n<!DOCTYPE\x20html>\n

SF:<html\x20lang=en>\n\x20\x20<meta\x20charset=utf-8>\n\x20\x20<meta\x20na

SF:me=viewport\x20content=\"initial-scale=1,\x20minimum-scale=1,\x20width=

SF:device-width\">\n\x20\x20<title>Error\x20405\x20\(Method\x20Not\x20Allo

SF:wed\)!!1</title>\n\x20\x20<style>\n\x20\x20\x20\x20\*{margin:0;padding:

SF:0}html,code{font:15px/22px\x20arial,sans-serif}html{background:#fff;col

SF:or:#222;padding:15px}body{margin:7%\x20auto\x200;max-width:390px;min-he

SF:ight:180px;padding:30px\x200\x2015px}\*\x20>\x20body{background:url\(//

SF:www\.google\.com/images/errors/robot\.png\)\x20100%\x205px\x20no-repeat

SF:;padding-right:205px}p{margin:11px\x200\x2022px;overflow:hidden}ins{col

SF:or:#777;text-decoration:none}a\x20img{border:0}@media\x20screen\x20and\

SF:x20\(max-width:772px\){body{background:none;margin-top:0;max-width:none

SF:;padding");

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .

Nmap done: 1 IP address (1 host up) scanned in 52.89 seconds

root@test:~#

]]> 566 Fix zfs-mount.service failing after reboot on Proxmox https://desantolo.com/2017/07/fix-zfs-mount-service-failing-after-reboot-on-proxmox/ https://desantolo.com/2017/07/fix-zfs-mount-service-failing-after-reboot-on-proxmox/#comments Sat, 01 Jul 2017 01:33:33 +0000 https://desantolo.com/?p=545 Continue reading →]]> In my new homelab migration to Proxmox I came across a bug that will prevent you from being able to mount all your ZFS mount points and be a pain in the ass even more if you host containers in that folder.

Cause of the problem: When you use a different zpool than the default rpool, and setup a directory mount for PVE to use for ISO datastore, VZ dump, etc on reboot if the zfs mount points have not completed mounting at boot time. Proxmox will attempt to create the directory path structure.

The problem with creating a directory for something before is mounted is that when zfs-mount.service runs and attempts to mount the zfs mount points you will get these kind of errors:

root@pve:~# systemctl status zfs-mount.service
● zfs-mount.service - Mount ZFS filesystems
Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2017-06-30 18:10:21 PDT; 21s ago
Process: 6590 ExecStart=/sbin/zfs mount -a (code=exited, status=1/FAILURE)
Main PID: 6590 (code=exited, status=1/FAILURE)

Jun 30 18:10:19 pve systemd[1]: Starting Mount ZFS filesystems...
Jun 30 18:10:20 pve zfs[6590]: cannot mount '/gdata/pve/subvol-102-disk-1': directory is not empty
Jun 30 18:10:20 pve zfs[6590]: cannot mount '/gdata/pve/subvol-106-disk-1': directory is not empty
Jun 30 18:10:20 pve zfs[6590]: cannot mount '/gdata/pve/subvol-109-disk-1': directory is not empty
Jun 30 18:10:21 pve systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Jun 30 18:10:21 pve systemd[1]: Failed to start Mount ZFS filesystems.
Jun 30 18:10:21 pve systemd[1]: zfs-mount.service: Unit entered failed state.
Jun 30 18:10:21 pve systemd[1]: zfs-mount.service: Failed with result 'exit-code'.

Fixing the root of the problem: change how proxmox deals with mounts by editing /etc/pve/storage.cfg – you need to add “mkdir 0” and “is_mountpoint” to the directory mount. Example:

dir: gdata-dump
path /gdata/vz
content iso,vztmpl,backup
maxfiles 0
shared 0
mkdir 0
is_mountpoint 1

Now we need to do some system cleanup before we reboot and confirm the problem is fixed.

Let’s check which mount points have failed:
root@pve:~# zfs list -r -o name,mountpoint,mounted

Now let’s umount all zfs mount points (except rpool of course – assuming the rootfs is zfs)

# zfs umount -a

After making sure ZFS mount points are unmounted, now we can delete the empty folders. Recall the failed mount points that the zfs list command gave you and one by one delete them like so:

# rm -rf /gdata/pve/subvol-102-disk-1

Do this for each folder that showed issues mounting. You have a choice to remount everything with zfs mount -O -a — or better… reboot the system and check its fixed. I like the later better. So reboot.

After it boots back up check that service was able to mount zfs without issues:

# systemctl status zfs-mount.service
# zfs list -r -o name,mountpoint,mounted

That’s all folks… if you made the edit to storage.cfg and added the two variables this should not occur again. This was an annoying bug to deal with but good to have found a better solution than a startup script doing some dirty tricks!

]]> https://desantolo.com/2017/07/fix-zfs-mount-service-failing-after-reboot-on-proxmox/feed/ 2 545 LXC allow non-root users to bind to port 80 (couchpotato example) https://desantolo.com/2017/06/lxc-allow-non-root-users-to-bind-to-port-80-couchpotato-example/ Thu, 29 Jun 2017 08:37:42 +0000 https://desantolo.com/?p=541 Continue reading →]]> A follow-up to my last post dealing with unprivileged port access on linux containers.

This time, I have a couchpotato container that I want to change its default port from 5050 to port 80, so that it is as simple as http://mycouch/ to access from the local network.

Since CouchPotato is a python script, my other method of whitelisting the binary won’t work, an alternative is to use authbind to get around this by granting a user/group privileges to bind to one of those restricted ports (non-root can’t bind to ports 1024 or less).

Environment: LXC Container (Debian 9.0 Stretch) image, with couchpotato defaults running on port 5050 and systemd init script setup (couchpotato user is named gmedia)

#  groupadd -g 3200 gmedia
# useradd -u 3200 -g gmedia -M gmedia
# apt-get install authbind
# touch /etc/authbind/byport/80
# chown gmedia /etc/authbind/byport/80
# chmod 500 /etc/authbind/byport/80

Now edit the startup settings (Exec/user/group):
# nano /etc/systemd/system/couchpotato.service

Should look something like this:

[Unit]
Description=CouchPotato application instance
After=network.target

[Service]
ExecStart=/usr/bin/authbind --deep /opt/CouchPotatoServer/CouchPotato.py
Type=simple
User=gmedia
Group=gmedia

[Install]
WantedBy=multi-user.target

Now its time to test:

# systemctl daemon-reload
# systemctl start couchpotato.service
# systemctl status couchpotato.service

Confirm all is hunky dory.

root@couchpotato:~# systemctl status couchpotato.service
● couchpotato.service - CouchPotato application instance
Loaded: loaded (/etc/systemd/system/couchpotato.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2017-06-29 08:35:32 UTC; 2s ago
Main PID: 1203 (python)
Tasks: 9 (limit: 4915)
CGroup: /system.slice/couchpotato.service
└─1203 python /opt/CouchPotatoServer/CouchPotato.py

Jun 29 08:35:32 couchpotato systemd[1]: Started CouchPotato application instance.
root@couchpotato:~# lsof -i :80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 1203 gmedia 49u IPv4 6008724 0t0 TCP *:http (LISTEN)
python 1203 gmedia 52u IPv4 6024843 0t0 TCP 192.168.200.140:http->192.168.200.5:56928 (ESTABLISHED)
root@couchpotato:~#

]]> 541 Install proxmox on a partition instead of a full-disk https://desantolo.com/2017/06/zfs-proxmox-on-a-partition-instead-of-a-full-disk/ Sun, 11 Jun 2017 11:24:31 +0000 https://desantolo.com/?p=532 Continue reading →]]> By default, installing Proxmox with ZFS during the installation process will force you to use the entire disk for the root zpool. For most installs this is good enough. However, I like to do things differently sometimes.

I have a pair of Samsung 840 Pro 256GB SSDs that I wanted to use for my new homelab that I am currently building (moving from vmware to proxmox). You may be wondering why I want to install the operating system on a partition instead of an entire disk. Several reasons:

1. Proxmox (ZFS-on-Linux) does not yet support SSD TRIM, FreeBSD does support it so migrating from FreeNAS into Proxmox I should be aware of it.
2. Data redundancy for the root filesystem does not need to be large. Even if I do RAID1 with my two SSDs I won’t be storing my critical data or VMs in the rpool – I want a smaller sized root pool that has fault-tolerance (RAID1). A partition of 60GB mirrored in two SSDs should fit the bill here.
3. ZIL Intent Log experimentation, I also want to experiment by using the same two SSDs to speed up my ZFS writes. I want a small partition in a stripe (RAID0) for performance, 45GB total (22.5gb per ssd) is plenty for this.
4. The left over unused space will be left untouched so that the SSD will have more available blocks during the controller’s built-in garbage collection (not the same as TRIM)

I don’t have enough time to go into a lot of details (it’s past 4am), so I will get to how to do it. If you are trying to follow my same steps, you will need at least 3 hard drives.

1. On a hard drive or device you don’t care to use in the final outcome, install Proxmox as you would normally. Wipe the entire partition table and let it install RAID0 on the whole disk.
2. Boot into your new installation, have the two new disks you want to keep attached to the system and ensure linux sees them fdisk should help with this.
3. You will now need to create the partitions on the new disks (not rpool):

You will need to know how to calculate hard disk sectors and multiply by your block size. I don’t have time to go over it but I will do a quick TL;DR example to give you an idea:

We want 25GB slice so that is around 25000000000 bytes / 512 (block size) = 48828125 total sectors to allocate this storage amount.

Take a look at the partition table to make sure you create something similar, fdisk -l /dev/sd$ (your rpool disk). We will leave 8MB disk at the end of the partition, Proxmox by default creates 3 partitions: GRUB_BOOT, ZFS data, Solaris 8MB.

This command creates the partitions for my new array, I’ve described them for you by the -c command. It should be self-explanatory.

# sgdisk -z /dev/sdb
# sgdisk -a1 -n1:34:2047 -t1:EF02 -c1:”BIOS boot” -n2:2048:156252048 -t2:BF01 -c2:”mirror” -n3:156252049:205080174 -t3:BF01 -c3:”stripe” -n4:205080175:205096559 -t4:BF0 /dev/sda

# sgdisk -a1 -n1:34:2047 -t1:EF02 -c1:”BIOS boot” -n2:2048:156252048 -t2:BF01 -c2:”mirror” -n3:156252049:205080174 -t3:BF01 -c3:”stripe” -n4:205080175:205096559 -t4:BF0 /dev/sdc
# zpool create -f stripe -o ashift=13 /dev/sda3 /dev/sdc3
# zpool create -f newroot -o ashift=13 mirror /dev/sda2 /dev/sdc2
# grub-install /dev/disk/by-id/ata-Samsung_SSD_840_PRO_Series_S1ATNSADB46090M
# grub-install /dev/disk/by-id/ata-Samsung_SSD_840_PRO_Series_S12RNEACC59063B

Backup & moving stuff.
# zfs snapshot -r rpool@fullbackup
# zfs list -t snapshot
# zfs send -R rpool@fullbackup | zfs recv -vFd newroot
root@pve:/# zpool get bootfs
NAME PROPERTY VALUE SOURCE
newroot bootfs – default
rpool bootfs rpool/ROOT/pve-1 local
stripe bootfs – default
root@pve:/# zpool set bootfs=newroot/ROOT/pve-1 newroot
zpool export newroot
zpool import -o altroot=/mnt newroot
— rebooted with freenas live cd, enter shell, import newroot with new name rpool. rebooted
— boot into proxmox recovery — once it boots, do recovery
grub-install /dev/sdb
grub-install /dev/sda
update-grub2
update-initramfs -u

#zpool set bootfs=newroot rpool could also work without renaming via FreeNAS but didn’t try.

]]> 532 Troubleshooting networking issues after fresh install of proxmox VE 4.4 https://desantolo.com/2017/02/troubleshooting-networking-issues-after-fresh-install-of-proxmox-ve-4-4/ Fri, 10 Feb 2017 06:04:02 +0000 https://desantolo.com/?p=504 Continue reading →]]> Writing a quick troubleshooting guide and informative post to address an issue I came across when installing Proxmox VE 4.4 on two of my machines.

On servers with more than two network interfaces Debian/Proxmox renames all interfaces and does not properly detect eth0 as the on-board ethernet as many other linux flavors. This may cause a mild headache if you just installed Proxmox with static IP addresses using the installer and upon reboot you can’t access any network resources.

I already explained the cause and you could argue that on the Proxmox installer they could add a built-in network detection check to properly label eth0 as eth0 as the device is named in many other linux distros. That currently does not exist so I will walk you around the troubleshooting.

Upon reboot or first boot after the installation is complete:
# ip link

The bridge interface (vmbr0) should read “NO-CARRIER, MULTICAST, UP” as well as “state down” a few words further to the left of the results.

# dmesg | grep eth

Read the entries in the dmesg logs, it tells you the name of network interfaces on your system.

“NO-CARRIER” indicates it does not detect an uplink, the interface is configured but none of its bridge members have a network cable or connection being detected.

To fix this you will want to run the following commands:
# ifdown -a
# vi /etc/network/interfaces

By default the installer sets up “eth0” as your only bridge member since the network card numbering got setup differently, the logical name on proxmox for eth0 is actually eth2.

Edit the single instance of eth0 with eth2 – save the file and exit the editor.

# ifup -a
should try to bring back up your interfaces. Trying pinging your network gateway, it should be working now. Cheers.

]]> 504 Virtualization hypervisor and containers all in one https://desantolo.com/2017/01/virtualization-hypervisor-docker-containers-all-in-one/ https://desantolo.com/2017/01/virtualization-hypervisor-docker-containers-all-in-one/#comments Sun, 08 Jan 2017 10:01:26 +0000 https://desantolo.com/?p=475 Continue reading →]]> I’m a big fan of virtualization, the ability to run multiple platforms and operating systems (called guests) in a single server (called host) is probably one of the best computing technologies of the past 10 years.

Personally, I have been using virtualization circa 2004. It all took off after 2006 when chip manufacturer’s started bundling virtualization technologies in their processors (Intel VT-x or AMD-v). The reason why “cloud” computing is so popular can also be attributed to virtualization.

In a container world…

However, in the past couple of years a new technology has been making making the rounds everywhere, the words “containers”, “docker”, “orchestration” is picking up steam in the past year. They say that containers are changing the landscape for system administrators and application developers.

Claims that containers can be built and deployed in seconds, share a common storage layer and allow you to resize the container in real-time when you need more performance or capacity are really exciting concepts and I think the time is now for me to jump in and learn a thing of two about this new technology when its hot a new.

Time to ditch vmware ESXi for a hybrid hypervisor?

You may remember my blog entry building a low-power sandy bridge ESXi server with ZFS – now 5 years later it is time to find a new platform that will allow me to keep my legacy virtual machines (VMs) as well as allow me to host containers using Docker.

The process of finding a suitable replacement for ESXi may take awhile and more than just a single entry on my blog. This is the first entry on my journey.

Before replacing something that works with a new platform I think it is good to point out the strengths and weaknesses of vmware ESXi (which has been my platform of choice for 6 years)

Strengths of ESXi

  • awesome windows GUI vSphere client that allows you to manage your hypervisor without the need for console or ssh
  • a web-interface to manage it too if you install a plugin
  • virtual switch with VLAN support
  • support for PCI passthrough (Intel VT-d) allowing you to assign PCI devices to virtual guests

Weaknesses

  • does not support docker containers (unless you wish to create a virtual machine and run docker from there – but I prefer a central platform if possible)
  • vmware continues to remove features from the free version of ESX – vSphere client interface is no longer availabel in their latest release
  • Nothing exciting has been released by vmware in the past 2 years (in terms of ESXi) and they push esxi users into paid licensees

The alternatives?

SmartOS is a fork off OpenIndiana/OpenSolaris, it seems to have a lot of great security features and features from Solaris that enjoy (you may have read of my love for the ZFS filesystem which is native to SmartOS). Joyent has recently open-sourced their SmartDataCenter “SDC” or they are now calling it Triton Enterprise.

What I like about it other than the fact it uses native Solaris and it uses the ZFS filesystem for storage is the fact that it is a hybrid hypervisor. It can host containers and VMs (using technology similar to virtualbox since virtualbox is also from solaris).

The downside of this platform seems to be the complexity needed to deploy containers with this tool. You need to have a “head node” to be the brains of the platform, the “head node” does a lot of critical things. It monitors the network, the other compute nodes (where you host your vms/containers), it also hosts the database for all the nodes. In dev mode you can force the head node to also be able to host VMs but this is not recommended or good practice.

The web interface (SmartDataCenter) to manage your containers and VMs is also very rudementary, there is no built-in console to your guests. You need to run a lot of commands in the head node’s shell to make JSON queries to grab the data you want like the VNC server and port address for your guests.

Honestly I have not dug much deeper into SmartOS but I probably should, it looks like an awesome project. I am sure for people that want to use their platform for scalable container/hypervisor deployments it makes sense, but to replace my single server at home doing virtualization it does not look very likely this may be a good choice given the complexity.

Proxmox is another platform I am looking at, you may recall that 7 years ago I discovered proxmox virtual environment and started using it on my lab. That was Proxmox VE 1.5 I think and I recently discovered they have made a lot of strides in the right direction.

Just a few weeks ago they released their latest PVE 4.4 and they are now supporting ZFS data pools (via FUSE/zfsonlinux), not to mention that they have replaced OpenVZ with LXC (linux containers). It may be worth it for me to download their latest release and check out their platform again.

Other than Proxmox or SmartOS, I have not come across any other ‘hybrid’ hypervisors. Please share in the comments if there is something else I should check out.

]]> https://desantolo.com/2017/01/virtualization-hypervisor-docker-containers-all-in-one/feed/ 1 475 Add additional IP’s on different subnets using same Ethernet card on PVE https://desantolo.com/2010/05/add-additional-ips-on-different-subnets-using-same-ethernet-card-on-pve/ https://desantolo.com/2010/05/add-additional-ips-on-different-subnets-using-same-ethernet-card-on-pve/#respond Fri, 21 May 2010 03:42:23 +0000 http://gioflux.wordpress.com/2010/05/21/add-additional-ips-on-different-subnets-using-same-ethernet-card-on-pve/ Continue reading →]]> To do this, we need to add a custom route to the server, we need to add the network and netmask addresses, to test and see if it works:

route add -net 10.5.0.0 netmask 255.255.255.0 dev vmbr0

if it works, add the following to your /etc/network/interfaces file

iface vmbr0 inet static
…
bridge_fd 0
up route add -net 10.5.0.0 netmask 255.255.255.0 dev vmbr0
down route del -net 10.5.0.0 netmask 255.255.255.0 dev vmbr0
…

did not work? Remove route with:

route del -net 10.5.0.0 netmask 255.255.255.0 dev vmbr0

]]> https://desantolo.com/2010/05/add-additional-ips-on-different-subnets-using-same-ethernet-card-on-pve/feed/ 0 21