Linux – Giovanni F. Mazzeo De Santolo

How to setup dual-stack IPv4 IPv6 Azure VM without a load-balancer

Giovanni — Sun, 11 Apr 2021 20:32:31 +0000

I wanted to document my Microsoft Azure saga in getting a public IPv6 address to work in a virtual machine without a load balancer in front of it. My needs were pretty simple and straightforward I wanted a virtual server that had a static IPv4 and IPv6 public addresses so that I can monitor my home network and other websites.

You would think this would be pretty easy, a few clicks and done? That wasn’t my experience on Azure and setting this up isn’t easy nor straightforward. Below is how to get it done, if this helps you – you can buy me a coffee or beer.

What Microsoft documentation said

You can use public IPv6 address on a load balancer which needs to be a separate paid server/product in front of your servers. This isn’t what I was looking – I complained about it on github with screenshots issue #69167.

Disclaimer: I have no expertise on Microsoft Azure prior to this (I currently work at Google and GCP is what we do). I chose to use Azure for my hobby/playground primarily because I get $50 a month free credit with my MSDN subscription.

My hacky (undocumented) way of getting it set up

Create a new Virtual Machine

Use portal.azure.com – I am not going to go in depth here but I will call out the items you need to make sure to enable or change at setup.

Do pay attention to these when creating VM:

Use "create a resource" select "Ubuntu Server 18.04 LTS" and should default to virtual server.
Make sure to select ‘create new resource group’ to bundle everything of this server together.
Disable default ‘scheduled shutdown’

Deploy it. Go back to the newly created resource group.

Edit Virtual Network to add IPv6 (address space)

You should see only IPv4 listed here. Like 10.0.0.0/24 – add IPv6.

Input: ace:cab:deca::/48

Hit save.

Edit Virtual Network to add IPv6 (subnets)

You should see default click it. On the right dialog that opens click Add IPv6 address space

Input: ace:cab:deca::/64

Make sure to select a network Security Group. (whatever the name you gave it). Save.

Create a dual-stack IPv4 and IPv6 public address

Search Azure for Public IP addresses create one.

Pay attention at creation:

Select IP version both
Select SKU standard
Ensure you associate it to your VM resource group and zone otherwise it won’t work

Stop virtual machine.

Self explanatory, or during step 1 make sure to ensure it won’t be auto started.

Associate the new NIC and delete the old NIC from the VM

With the VM shutdown, networking settings > "Attach network interface" menu. Select create and attach network interface.

At NIC create:

NIC security group select NONE.
Select Private IPv6 address. Give it a name "v6" for me.

Detach the old NIC and delete from resource group (self-explanatory).

Associate the public IPv6 and IPv6 to the network interface

Go to the new NIC we created and associated, "IP configuration" menu.

You should see ‘ipconfig1’ is IPv4 and "v6" is secondary with our local IPv6 we gave on step 2.

Associate IPv4 public address by clicking ipconfig1 a new menu with a drop-down box and the new IPv4 (dual-stack) shows up. Select and save.

Associate IPv6 public address same as above. You should end up with something like this

Incoming firewall rules

Make sure to add necessary firewall rules, if you created the default settings on the security group you probably already have SSH (port 22) and that’s it.

You probably want to add a rule for ICMP traffic (ping).

If you like to have no security at all (or implement your own firewall on the virtual server) you can add a blanked incoming rule for all ports 0-65535 and this should open everything.

Note microsoft IPv6 implementation sucks and ICMP ping on IPv6 incoming/outgoing WILL NOT WORK! This is what tripped me out and I spent several hours trying to troubleshoot something that Microsoft could have easily documented… but here we are… I spent hours frustrated but hopefully with this guide I wrote for you it saved you all this time. If you appreciated it – remember you can buy me a coffee

Go ahead start your virtual server and you should be able to use nmap on its IPv6 address or SSH remotely and see it work. See above in:re ping on IPv6.

Updates

June 2021: Reader “Ben R” contacted me about this article and shared some noteworthy information for folks using older VM images or installations. DHCPv6 may be disabled and must be manually enabled. See this article for enabling DHCPv6 on Azure.

Linux IPv6 command cheatsheet

Giovanni — Fri, 05 Mar 2021 05:02:14 +0000

This post is intended to be a quick note to self on linux IPv6 commands, for quick reference when needed.

# Find the route decision for a given address.
ip route get 2a00:1450:4001:820::200e

ip -6 route
route -A inet6 (apt-get install net-tools) # equivalent to netstat -rn -A inet6
netstat -A inet6 -rn
netstat -r
ip -6 route show table 51820
ip -6 neigh
ip -6 neigh show proxy

# DHCPv6 release / renew
dhclient -6 -r ens18
dhclient -6 ens18

# Flush IPv6 neighbors cache
ip -family inet6 neigh flush any

# Find the route decision for a given address.
ip route get 2a00:1450:4001:820::200e

# Request a router advertisement (apt-get install -y ndisc6)
rdisc6 eth0

# Perform a Neighbor Solicitation (arp IPv4 equivalent) (apt-get install -y ndisc6)
ndisc6 2001:db8::42 eth0

# ping multicast address to have all link-local hosts to respond.
ping ff02::1%eth0

Good articles about IPv6:

IPv6 range blocks
IPv6 unicast, anycast, multicast – well known addresses.

Troubleshooting tools:

IPv6 ping from multiple locations.

Bonus section ‘ufw’ firewall

Check status of firewall and enable or disable.
# ufw status
# ufw enable
# ufw disable

View firewall rules and number to delete a specific rule.
# ufw status verbose
# ufw status numbered

Delete a firewall rule (from number of last cmd)
# ufw delete 12

Rules needed to allow IPv6 routing via wireguard to endpoint (no iptables forwarding would be needed on IPv6 as the other peer carries the public IP – they also should implement a firewall before using the below command!)
# ufw route allow in on eth0 out on wghub to 2602:xxx:730b:xxx:cafe::12

IPv4 firewall and routing rule to open Plex Media Server port 32400 and internal IPv4 space of wireguard peer.
# ufw allow in on eth0 out on wghub to 10.100.100.11 port 32400
# ufw route allow in on eth0 out on wghub to 10.100.100.11 port 32400

Add rule and get logs for matches of that rule:
# ufw allow log proto any from any to 192.168.1.x

View logs:
# tail -f /var/log/ufw.log

Extra bonus iptables NAT forwarding

On VPS (public endpoint) forward IPv4 traffic on tcp/32400 to wireguard peer. (recall IPv6 doesn’t need a rule)
# iptables -A PREROUTING -t nat -i eth0 -p tcp –dport 32400 -j DNAT –to 10.100.100.11:32400

Test your port forwards

For IPv6 I use:
http://www.ipv6scanner.com/cgi-bin/main.py

For IPv4:
https://www.yougetsignal.com/tools/open-ports/

Fix ZFSonLinux pool auto expanding

Giovanni — Mon, 24 Jul 2017 05:00:16 +0000

If you’re having issues with zfsonlinux and your pool not expanding after replacing your hard drives with larger ones then here is a trick to fix it.

# zpool set autoexpand=on {pool name}

# zpool online -e {pool-name} {disk name/id as displayed on zpool status}

Your pool should resize after running the second command. The first command was only to make sure you had set the zfs property that’s needed for the second command to expand the pool.

Fix zfs-mount.service failing after reboot on Proxmox

Giovanni — Sat, 01 Jul 2017 01:33:33 +0000

In my new homelab migration to Proxmox I came across a bug that will prevent you from being able to mount all your ZFS mount points and be a pain in the ass even more if you host containers in that folder.

Cause of the problem: When you use a different zpool than the default rpool, and setup a directory mount for PVE to use for ISO datastore, VZ dump, etc on reboot if the zfs mount points have not completed mounting at boot time. Proxmox will attempt to create the directory path structure.

The problem with creating a directory for something before is mounted is that when zfs-mount.service runs and attempts to mount the zfs mount points you will get these kind of errors:

root@pve:~# systemctl status zfs-mount.service
● zfs-mount.service - Mount ZFS filesystems
Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2017-06-30 18:10:21 PDT; 21s ago
Process: 6590 ExecStart=/sbin/zfs mount -a (code=exited, status=1/FAILURE)
Main PID: 6590 (code=exited, status=1/FAILURE)

Jun 30 18:10:19 pve systemd[1]: Starting Mount ZFS filesystems...
Jun 30 18:10:20 pve zfs[6590]: cannot mount '/gdata/pve/subvol-102-disk-1': directory is not empty
Jun 30 18:10:20 pve zfs[6590]: cannot mount '/gdata/pve/subvol-106-disk-1': directory is not empty
Jun 30 18:10:20 pve zfs[6590]: cannot mount '/gdata/pve/subvol-109-disk-1': directory is not empty
Jun 30 18:10:21 pve systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Jun 30 18:10:21 pve systemd[1]: Failed to start Mount ZFS filesystems.
Jun 30 18:10:21 pve systemd[1]: zfs-mount.service: Unit entered failed state.
Jun 30 18:10:21 pve systemd[1]: zfs-mount.service: Failed with result 'exit-code'.

Fixing the root of the problem: change how proxmox deals with mounts by editing /etc/pve/storage.cfg – you need to add “mkdir 0” and “is_mountpoint” to the directory mount. Example:

dir: gdata-dump
path /gdata/vz
content iso,vztmpl,backup
maxfiles 0
shared 0
mkdir 0
is_mountpoint 1

Now we need to do some system cleanup before we reboot and confirm the problem is fixed.

Let’s check which mount points have failed:
root@pve:~# zfs list -r -o name,mountpoint,mounted

Now let’s umount all zfs mount points (except rpool of course – assuming the rootfs is zfs)

# zfs umount -a

After making sure ZFS mount points are unmounted, now we can delete the empty folders. Recall the failed mount points that the zfs list command gave you and one by one delete them like so:

# rm -rf /gdata/pve/subvol-102-disk-1

Do this for each folder that showed issues mounting. You have a choice to remount everything with zfs mount -O -a — or better… reboot the system and check its fixed. I like the later better. So reboot.

After it boots back up check that service was able to mount zfs without issues:

# systemctl status zfs-mount.service
# zfs list -r -o name,mountpoint,mounted

That’s all folks… if you made the edit to storage.cfg and added the two variables this should not occur again. This was an annoying bug to deal with but good to have found a better solution than a startup script doing some dirty tricks!

LXC allow non-root users to bind to port 80 (couchpotato example)

Giovanni — Thu, 29 Jun 2017 08:37:42 +0000

A follow-up to my last post dealing with unprivileged port access on linux containers.

This time, I have a couchpotato container that I want to change its default port from 5050 to port 80, so that it is as simple as http://mycouch/ to access from the local network.

Since CouchPotato is a python script, my other method of whitelisting the binary won’t work, an alternative is to use authbind to get around this by granting a user/group privileges to bind to one of those restricted ports (non-root can’t bind to ports 1024 or less).

Environment: LXC Container (Debian 9.0 Stretch) image, with couchpotato defaults running on port 5050 and systemd init script setup (couchpotato user is named gmedia)

# groupadd -g 3200 gmedia
# useradd -u 3200 -g gmedia -M gmedia
# apt-get install authbind
# touch /etc/authbind/byport/80
# chown gmedia /etc/authbind/byport/80
# chmod 500 /etc/authbind/byport/80

Now edit the startup settings (Exec/user/group):
# nano /etc/systemd/system/couchpotato.service

Should look something like this:

[Unit]
Description=CouchPotato application instance
After=network.target

[Service]
ExecStart=/usr/bin/authbind --deep /opt/CouchPotatoServer/CouchPotato.py
Type=simple
User=gmedia
Group=gmedia

[Install]
WantedBy=multi-user.target

Now its time to test:

# systemctl daemon-reload
# systemctl start couchpotato.service
# systemctl status couchpotato.service

Confirm all is hunky dory.

root@couchpotato:~# systemctl status couchpotato.service
● couchpotato.service - CouchPotato application instance
Loaded: loaded (/etc/systemd/system/couchpotato.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2017-06-29 08:35:32 UTC; 2s ago
Main PID: 1203 (python)
Tasks: 9 (limit: 4915)
CGroup: /system.slice/couchpotato.service
└─1203 python /opt/CouchPotatoServer/CouchPotato.py

Jun 29 08:35:32 couchpotato systemd[1]: Started CouchPotato application instance.
root@couchpotato:~# lsof -i :80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 1203 gmedia 49u IPv4 6008724 0t0 TCP *:http (LISTEN)
python 1203 gmedia 52u IPv4 6024843 0t0 TCP 192.168.200.140:http->192.168.200.5:56928 (ESTABLISHED)
root@couchpotato:~#

Allow non-root processes to bind to privileged (ports

Giovanni — Wed, 28 Jun 2017 07:53:49 +0000

As I work on my homelab migration from FreeNAS into Linux containers, I need to move my freebsd jails to LXC.

In *nix any usage of well-known ports (aka 1024 or less) requires special privileges or a kernel setting. In FreeBSD a simple sysctl net.inet.ip.portrange.reservedhigh =1 was enough to allow the BSD jail to use any port on the jail.

On LXC, I had to figure out how to do the same thing and its quite different. My environment is a debian stretch LXC container but should work on other linux versions.

# apt-get install libcap2-bin
# setcap 'cap_net_bind_service=+ep' /usr/bin/transmission-daemon

In the example above, the binary /usr/bin/transmission-daemon is now able to open any port, or port 80 http in my case all while running a service as a non-root user.

Hopefully these helps folks out there, the answer took some digging but I already had an idea on what was needed thanks to my FreeBSD experience in zones

Install proxmox on a partition instead of a full-disk

Giovanni — Sun, 11 Jun 2017 11:24:31 +0000

By default, installing Proxmox with ZFS during the installation process will force you to use the entire disk for the root zpool. For most installs this is good enough. However, I like to do things differently sometimes.

I have a pair of Samsung 840 Pro 256GB SSDs that I wanted to use for my new homelab that I am currently building (moving from vmware to proxmox). You may be wondering why I want to install the operating system on a partition instead of an entire disk. Several reasons:

1. Proxmox (ZFS-on-Linux) does not yet support SSD TRIM, FreeBSD does support it so migrating from FreeNAS into Proxmox I should be aware of it.
2. Data redundancy for the root filesystem does not need to be large. Even if I do RAID1 with my two SSDs I won’t be storing my critical data or VMs in the rpool – I want a smaller sized root pool that has fault-tolerance (RAID1). A partition of 60GB mirrored in two SSDs should fit the bill here.
3. ZIL Intent Log experimentation, I also want to experiment by using the same two SSDs to speed up my ZFS writes. I want a small partition in a stripe (RAID0) for performance, 45GB total (22.5gb per ssd) is plenty for this.
4. The left over unused space will be left untouched so that the SSD will have more available blocks during the controller’s built-in garbage collection (not the same as TRIM)

I don’t have enough time to go into a lot of details (it’s past 4am), so I will get to how to do it. If you are trying to follow my same steps, you will need at least 3 hard drives.

1. On a hard drive or device you don’t care to use in the final outcome, install Proxmox as you would normally. Wipe the entire partition table and let it install RAID0 on the whole disk.
2. Boot into your new installation, have the two new disks you want to keep attached to the system and ensure linux sees them fdisk should help with this.
3. You will now need to create the partitions on the new disks (not rpool):

You will need to know how to calculate hard disk sectors and multiply by your block size. I don’t have time to go over it but I will do a quick TL;DR example to give you an idea:

We want 25GB slice so that is around 25000000000 bytes / 512 (block size) = 48828125 total sectors to allocate this storage amount.

Take a look at the partition table to make sure you create something similar, fdisk -l /dev/sd$ (your rpool disk). We will leave 8MB disk at the end of the partition, Proxmox by default creates 3 partitions: GRUB_BOOT, ZFS data, Solaris 8MB.

This command creates the partitions for my new array, I’ve described them for you by the -c command. It should be self-explanatory.

# sgdisk -z /dev/sdb
# sgdisk -a1 -n1:34:2047 -t1:EF02 -c1:”BIOS boot” -n2:2048:156252048 -t2:BF01 -c2:”mirror” -n3:156252049:205080174 -t3:BF01 -c3:”stripe” -n4:205080175:205096559 -t4:BF0 /dev/sda

# sgdisk -a1 -n1:34:2047 -t1:EF02 -c1:”BIOS boot” -n2:2048:156252048 -t2:BF01 -c2:”mirror” -n3:156252049:205080174 -t3:BF01 -c3:”stripe” -n4:205080175:205096559 -t4:BF0 /dev/sdc
# zpool create -f stripe -o ashift=13 /dev/sda3 /dev/sdc3
# zpool create -f newroot -o ashift=13 mirror /dev/sda2 /dev/sdc2
# grub-install /dev/disk/by-id/ata-Samsung_SSD_840_PRO_Series_S1ATNSADB46090M
# grub-install /dev/disk/by-id/ata-Samsung_SSD_840_PRO_Series_S12RNEACC59063B

Backup & moving stuff.
# zfs snapshot -r rpool@fullbackup
# zfs list -t snapshot
# zfs send -R rpool@fullbackup | zfs recv -vFd newroot
root@pve:/# zpool get bootfs
NAME PROPERTY VALUE SOURCE
newroot bootfs – default
rpool bootfs rpool/ROOT/pve-1 local
stripe bootfs – default
root@pve:/# zpool set bootfs=newroot/ROOT/pve-1 newroot
zpool export newroot
zpool import -o altroot=/mnt newroot
— rebooted with freenas live cd, enter shell, import newroot with new name rpool. rebooted
— boot into proxmox recovery — once it boots, do recovery
grub-install /dev/sdb
grub-install /dev/sda
update-grub2
update-initramfs -u

#zpool set bootfs=newroot rpool could also work without renaming via FreeNAS but didn’t try.

Xenserver 7.1 a good contender to replace esxi

Giovanni — Sun, 26 Feb 2017 21:37:13 +0000

I have been a big fan and user of vmware ESXi for years, I started playing with it since circa 2008. The bare metal hypervisor and its easy to use GUI on Windows makes virtualization management extremely easy.

However in the past couple of years the free version of esxi has moved to HTML5 web management, and in the latest ESXi version the Windows client (vSphere client) requires you to pay for a license a run a central vcenter server/vm in order to manage via GUI (non-web).

There are a few articles posted on how limited the web UI may be in contrast to all the features of the original windows GUI so I started pondering other products that may replace ESXi on my homelab (been an ESXi user since 3.x) but it looks like Xenserver 7.1 was just released and Citrix has been including a lot of Enterprise features on their free Xenserver product and offer a very comparable GUI on Windows as well.

A few days ago (February 23) Citrix just released the latest Xenserver 7.1 version (release link/downloads/changelog). The feature I am most excited about? Docker integration via installing “container management supplemental pack”

Xenserver 7.1 seems to be based on Linux CentOS so installing packages (via rpm) should be pretty easy. I’ll post more information when I get more time to dig into it deeper as I am just installing it on my homelab for testing.

Troubleshooting networking issues after fresh install of proxmox VE 4.4

Giovanni — Fri, 10 Feb 2017 06:04:02 +0000

Writing a quick troubleshooting guide and informative post to address an issue I came across when installing Proxmox VE 4.4 on two of my machines.

On servers with more than two network interfaces Debian/Proxmox renames all interfaces and does not properly detect eth0 as the on-board ethernet as many other linux flavors. This may cause a mild headache if you just installed Proxmox with static IP addresses using the installer and upon reboot you can’t access any network resources.

I already explained the cause and you could argue that on the Proxmox installer they could add a built-in network detection check to properly label eth0 as eth0 as the device is named in many other linux distros. That currently does not exist so I will walk you around the troubleshooting.

Upon reboot or first boot after the installation is complete:
# ip link

The bridge interface (vmbr0) should read “NO-CARRIER, MULTICAST, UP” as well as “state down” a few words further to the left of the results.

# dmesg | grep eth

Read the entries in the dmesg logs, it tells you the name of network interfaces on your system.

“NO-CARRIER” indicates it does not detect an uplink, the interface is configured but none of its bridge members have a network cable or connection being detected.

To fix this you will want to run the following commands:
# ifdown -a
# vi /etc/network/interfaces

By default the installer sets up “eth0” as your only bridge member since the network card numbering got setup differently, the logical name on proxmox for eth0 is actually eth2.

Edit the single instance of eth0 with eth2 – save the file and exit the editor.

# ifup -a
should try to bring back up your interfaces. Trying pinging your network gateway, it should be working now. Cheers.

How to flash Seagate firmware onto HP enterprise MB2000EAMZF drives

Giovanni — Thu, 12 Jan 2017 07:06:40 +0000

In my previous post I talked about how HP locks down their hard drives (made by Seagate) with custom firmware that is exclusive to HP and with the huge caveat that if you do not have an HP storage controller the software update package provided by HP won’t work for you at all.

I spent the past 48 hours researching and trying different things. I unpacked the official HP firmware and tried to find the binary/firmware dump from within the .scexe file provided by HP on their website to no avail. I even contacted HP for help and they would not be willing to help if the drives were not connected to HP hardware.

After some research I discovered that the HP MB2000EAMZF 2TB drive I had was almost exactly the same (part number) as the official Seagate ST32000644NS drive after failing to be able to flash HPs firmware using HP’s bootable Firmware Update 8.3 (injecting the .scexe files inside this live CD) – I decided I really had nothing to lose but try the method I will explain below.

The information provided below is provided to you with no guarantees it will work for you. By following my steps you are risking bricking your hard drive and I won’t be liable for it. Continue at your own risk.

The tale of two drives

*Seagate original* Constellation ES ST32000644NS has part number: 9JW168-039

*HP enterprise* MB2000EAMZF has part number: 9JW168-280

It is possible that these two drives have different PCB control boards (basically what controls the disk heads and communicates to the physical disk). If I load a firmware that is written for a different PCB – my drive will most likely be bricked.

The part numbers were too close to eachother, only the last three digits were different. I did not find any other information online from anyone trying what I was about to attempt. HP had issued a *Critical* firmware update for these drives and my only choice was to take the risk now or put this drive on my ZFS array and wait for it to fail short term.

Tools needed and where to get the firmware

I had another ST32000644NS drive around and used it’s serial number on seagate’s website. It provided me with firmware version SN12 (ConstellationES1-Muskie-StdOEM-SATA-SN12.zip). Download to your desktop, unpack the zip and look inside the firmware folder SN12.lod file.
Download and make a bootable CD of system-rescue live cd – or if you have a Dell iDRAC like me boot from the .iso remotely
Ensure that only the hard drive to be flashed is connected to the server/system.
Boot into the linux live CD, once in console check drive smart data to see the current model and version with smartct -x /dev/sda (or whatever the /dev/ name of your drive is)
Once you have verified the smart data of the drive you want to flash (it will have a name of /dev/sda or /dev/sdb or /dev/sd$)
Make sure to have a USB thumb drive with the SN12.lod file from step one – plug it in and mount the drive (if its linux formatted it will be as simple as mount /dev/sdb1 /mnt/windows)
Now we have the files and we’re ready to flash. Make sure disk is idle (no smart tests running in background). Push the firmware with hdparm –fwdownload SN12.lod –yes-i-know-what-i-am-doing –please-destroy-my-drive /dev/sdb
Reboot the server.

You can repeat steps 4 to 5 and check smart data report and you should see the drive running the firmware you just flashed. In my case here you can see my firmware changed to SN12 on my HP hard drive.

Good luck!