# zpool set autoexpand=on {pool name}

# zpool online -e {pool-name} {disk name/id as displayed on zpool status}

Your pool should resize after running the second command. The first command was only to make sure you had set the zfs property that’s needed for the second command to expand the pool.

The problem with creating a directory for something before is mounted is that when zfs-mount.service runs and attempts to mount the zfs mount points you will get these kind of errors:

root@pve:~# systemctl status zfs-mount.service
● zfs-mount.service - Mount ZFS filesystems
Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2017-06-30 18:10:21 PDT; 21s ago
Process: 6590 ExecStart=/sbin/zfs mount -a (code=exited, status=1/FAILURE)
Main PID: 6590 (code=exited, status=1/FAILURE)

Jun 30 18:10:19 pve systemd[1]: Starting Mount ZFS filesystems...
Jun 30 18:10:20 pve zfs[6590]: cannot mount '/gdata/pve/subvol-102-disk-1': directory is not empty
Jun 30 18:10:20 pve zfs[6590]: cannot mount '/gdata/pve/subvol-106-disk-1': directory is not empty
Jun 30 18:10:20 pve zfs[6590]: cannot mount '/gdata/pve/subvol-109-disk-1': directory is not empty
Jun 30 18:10:21 pve systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Jun 30 18:10:21 pve systemd[1]: Failed to start Mount ZFS filesystems.
Jun 30 18:10:21 pve systemd[1]: zfs-mount.service: Unit entered failed state.
Jun 30 18:10:21 pve systemd[1]: zfs-mount.service: Failed with result 'exit-code'.

Fixing the root of the problem: change how proxmox deals with mounts by editing /etc/pve/storage.cfg – you need to add “mkdir 0” and “is_mountpoint” to the directory mount. Example:

dir: gdata-dump
path /gdata/vz
content iso,vztmpl,backup
maxfiles 0
shared 0
mkdir 0
is_mountpoint 1

Now we need to do some system cleanup before we reboot and confirm the problem is fixed.

Let’s check which mount points have failed:
root@pve:~# zfs list -r -o name,mountpoint,mounted

Now let’s umount all zfs mount points (except rpool of course – assuming the rootfs is zfs)

# zfs umount -a

After making sure ZFS mount points are unmounted, now we can delete the empty folders. Recall the failed mount points that the zfs list command gave you and one by one delete them like so:

# rm -rf /gdata/pve/subvol-102-disk-1

Do this for each folder that showed issues mounting. You have a choice to remount everything with zfs mount -O -a — or better… reboot the system and check its fixed. I like the later better. So reboot.

After it boots back up check that service was able to mount zfs without issues:

# systemctl status zfs-mount.service
# zfs list -r -o name,mountpoint,mounted

That’s all folks… if you made the edit to storage.cfg and added the two variables this should not occur again. This was an annoying bug to deal with but good to have found a better solution than a startup script doing some dirty tricks!

I have a pair of Samsung 840 Pro 256GB SSDs that I wanted to use for my new homelab that I am currently building (moving from vmware to proxmox). You may be wondering why I want to install the operating system on a partition instead of an entire disk. Several reasons:

1. Proxmox (ZFS-on-Linux) does not yet support SSD TRIM, FreeBSD does support it so migrating from FreeNAS into Proxmox I should be aware of it.
2. Data redundancy for the root filesystem does not need to be large. Even if I do RAID1 with my two SSDs I won’t be storing my critical data or VMs in the rpool – I want a smaller sized root pool that has fault-tolerance (RAID1). A partition of 60GB mirrored in two SSDs should fit the bill here.
3. ZIL Intent Log experimentation, I also want to experiment by using the same two SSDs to speed up my ZFS writes. I want a small partition in a stripe (RAID0) for performance, 45GB total (22.5gb per ssd) is plenty for this.
4. The left over unused space will be left untouched so that the SSD will have more available blocks during the controller’s built-in garbage collection (not the same as TRIM)

I don’t have enough time to go into a lot of details (it’s past 4am), so I will get to how to do it. If you are trying to follow my same steps, you will need at least 3 hard drives.

1. On a hard drive or device you don’t care to use in the final outcome, install Proxmox as you would normally. Wipe the entire partition table and let it install RAID0 on the whole disk.
2. Boot into your new installation, have the two new disks you want to keep attached to the system and ensure linux sees them fdisk should help with this.
3. You will now need to create the partitions on the new disks (not rpool):

You will need to know how to calculate hard disk sectors and multiply by your block size. I don’t have time to go over it but I will do a quick TL;DR example to give you an idea:

We want 25GB slice so that is around 25000000000 bytes / 512 (block size) = 48828125 total sectors to allocate this storage amount.

Take a look at the partition table to make sure you create something similar, fdisk -l /dev/sd$ (your rpool disk). We will leave 8MB disk at the end of the partition, Proxmox by default creates 3 partitions: GRUB_BOOT, ZFS data, Solaris 8MB.

This command creates the partitions for my new array, I’ve described them for you by the -c command. It should be self-explanatory.

# sgdisk -z /dev/sdb
# sgdisk -a1 -n1:34:2047 -t1:EF02 -c1:”BIOS boot” -n2:2048:156252048 -t2:BF01 -c2:”mirror” -n3:156252049:205080174 -t3:BF01 -c3:”stripe” -n4:205080175:205096559 -t4:BF0 /dev/sda

# sgdisk -a1 -n1:34:2047 -t1:EF02 -c1:”BIOS boot” -n2:2048:156252048 -t2:BF01 -c2:”mirror” -n3:156252049:205080174 -t3:BF01 -c3:”stripe” -n4:205080175:205096559 -t4:BF0 /dev/sdc
# zpool create -f stripe -o ashift=13 /dev/sda3 /dev/sdc3
# zpool create -f newroot -o ashift=13 mirror /dev/sda2 /dev/sdc2
# grub-install /dev/disk/by-id/ata-Samsung_SSD_840_PRO_Series_S1ATNSADB46090M
# grub-install /dev/disk/by-id/ata-Samsung_SSD_840_PRO_Series_S12RNEACC59063B

Backup & moving stuff.
# zfs snapshot -r rpool@fullbackup
# zfs list -t snapshot
# zfs send -R rpool@fullbackup | zfs recv -vFd newroot
root@pve:/# zpool get bootfs
NAME PROPERTY VALUE SOURCE
newroot bootfs – default
rpool bootfs rpool/ROOT/pve-1 local
stripe bootfs – default
root@pve:/# zpool set bootfs=newroot/ROOT/pve-1 newroot
zpool export newroot
zpool import -o altroot=/mnt newroot
— rebooted with freenas live cd, enter shell, import newroot with new name rpool. rebooted
— boot into proxmox recovery — once it boots, do recovery
grub-install /dev/sdb
grub-install /dev/sda
update-grub2
update-initramfs -u

#zpool set bootfs=newroot rpool could also work without renaming via FreeNAS but didn’t try.

Personally, I have been using virtualization circa 2004. It all took off after 2006 when chip manufacturer’s started bundling virtualization technologies in their processors (Intel VT-x or AMD-v). The reason why “cloud” computing is so popular can also be attributed to virtualization.

In a container world…

However, in the past couple of years a new technology has been making making the rounds everywhere, the words “containers”, “docker”, “orchestration” is picking up steam in the past year. They say that containers are changing the landscape for system administrators and application developers.

Claims that containers can be built and deployed in seconds, share a common storage layer and allow you to resize the container in real-time when you need more performance or capacity are really exciting concepts and I think the time is now for me to jump in and learn a thing of two about this new technology when its hot a new.

Time to ditch vmware ESXi for a hybrid hypervisor?

You may remember my blog entry building a low-power sandy bridge ESXi server with ZFS – now 5 years later it is time to find a new platform that will allow me to keep my legacy virtual machines (VMs) as well as allow me to host containers using Docker.

The process of finding a suitable replacement for ESXi may take awhile and more than just a single entry on my blog. This is the first entry on my journey.

Before replacing something that works with a new platform I think it is good to point out the strengths and weaknesses of vmware ESXi (which has been my platform of choice for 6 years)

Strengths of ESXi

• awesome windows GUI vSphere client that allows you to manage your hypervisor without the need for console or ssh
• a web-interface to manage it too if you install a plugin
• virtual switch with VLAN support
• support for PCI passthrough (Intel VT-d) allowing you to assign PCI devices to virtual guests

Weaknesses

• does not support docker containers (unless you wish to create a virtual machine and run docker from there – but I prefer a central platform if possible)
• vmware continues to remove features from the free version of ESX – vSphere client interface is no longer availabel in their latest release
• Nothing exciting has been released by vmware in the past 2 years (in terms of ESXi) and they push esxi users into paid licensees

The alternatives?

SmartOS is a fork off OpenIndiana/OpenSolaris, it seems to have a lot of great security features and features from Solaris that enjoy (you may have read of my love for the ZFS filesystem which is native to SmartOS). Joyent has recently open-sourced their SmartDataCenter “SDC” or they are now calling it Triton Enterprise.

What I like about it other than the fact it uses native Solaris and it uses the ZFS filesystem for storage is the fact that it is a hybrid hypervisor. It can host containers and VMs (using technology similar to virtualbox since virtualbox is also from solaris).

The downside of this platform seems to be the complexity needed to deploy containers with this tool. You need to have a “head node” to be the brains of the platform, the “head node” does a lot of critical things. It monitors the network, the other compute nodes (where you host your vms/containers), it also hosts the database for all the nodes. In dev mode you can force the head node to also be able to host VMs but this is not recommended or good practice.

The web interface (SmartDataCenter) to manage your containers and VMs is also very rudementary, there is no built-in console to your guests. You need to run a lot of commands in the head node’s shell to make JSON queries to grab the data you want like the VNC server and port address for your guests.

Honestly I have not dug much deeper into SmartOS but I probably should, it looks like an awesome project. I am sure for people that want to use their platform for scalable container/hypervisor deployments it makes sense, but to replace my single server at home doing virtualization it does not look very likely this may be a good choice given the complexity.

Proxmox is another platform I am looking at, you may recall that 7 years ago I discovered proxmox virtual environment and started using it on my lab. That was Proxmox VE 1.5 I think and I recently discovered they have made a lot of strides in the right direction.

Just a few weeks ago they released their latest PVE 4.4 and they are now supporting ZFS data pools (via FUSE/zfsonlinux), not to mention that they have replaced OpenVZ with LXC (linux containers). It may be worth it for me to download their latest release and check out their platform again.

Other than Proxmox or SmartOS, I have not come across any other ‘hybrid’ hypervisors. Please share in the comments if there is something else I should check out.

Let me start by giving you a brief overview of my old system:

• Intel Core 2 Quad Q9450 2.66Ghz LGA 775 95W TDP
• Corsair Builder CX430 430Watt Power Supply
• 4GB PC2-6400 667Mhz DDR2 Ram 1.5v
• ATI Radeon 4800 basic PCI-E graphics (no PCI-E power needed)
• Biostar Tpower I45 Motherboard (I45 Chipset)
• LSI SAS3041E-R SATA II 300Mbps RAID controller
• OS: Ubuntu Linux 10.04
• Storage: Oracle ZFS (via FUSE-ZFS) **
• RAID-Z with four drives (3x 2TB plus one 1TB drive)

As you can see, while idle the system is drawing 130 Watts constantly at a minimum. This becomes a problem since the server is online for 24×7 and thanks to EZ Kill-A-Watt it costs an estimated $30 a month to run the server. About $0.80 a day in electricity alone.

With today’s green technology, Sandy Bridge new processor steppings (Intel Speedstep) and green hard drives (replacing my old 1TB 7200 RPM drive with quieter, lower-power 2TB hitachi), I hope to reduce my idle power consumption at least 20%. Based on $30 a month total cost to run my old server, this means I would be saving $6+ a month on my electric bill with this new build.

Giovanni@server:~# format -e
Searching for disks…done
AVAILABLE DISK SELECTIONS:
0. c8t0d0 <DEFAULT cyl 60797 alt 2 hd 255 sec 252>
/pci@0,0/pci15d9,d380@1f,2/disk@0,0
1. c8t1d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
/pci@0,0/pci15d9,d380@1f,2/disk@1,0
2. c8t2d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
/pci@0,0/pci15d9,d380@1f,2/disk@2,0
3. c8t3d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
/pci@0,0/pci15d9,d380@1f,2/disk@3,0
4. c8t4d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
/pci@0,0/pci15d9,d380@1f,2/disk@4,0
5. c8t5d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
/pci@0,0/pci15d9,d380@1f,2/disk@5,0
Specify disk (enter its number):

Select a drive, lets pick 5 from the list.

Specify disk (enter its number): 5
selecting c8t5d0
[disk formatted]
/dev/dsk/c8t5d0s0 is part of active ZFS pool gpool. Please see zpool(1M).
FORMAT MENU:
disk       – select a disk
type       – select (define) a disk type
partition  – select (define) a partition table
current    – describe the current disk
format     – format and analyze the disk
fdisk      – run the fdisk program
repair     – repair a defective sector
label      – write label to the disk
analyze    – surface analysis
defect     – defect list management
backup     – search for backup labels
verify     – read and display labels
inquiry    – show vendor, product and revision
scsi       – independent SCSI mode selects
cache      – enable, disable or query SCSI disk cache
volname    – set 8-character volume name
!<cmd>     – execute <cmd>, then return
quit
format>

Now let’s do the checking

Enter “cache” to enter cache menu.

CACHE MENU:
write_cache – display or modify write cache settings
read_cache  – display or modify read cache settings
!<cmd>      – execute <cmd>, then return
quit
cache>

Type: “write_cache” or “read_cache” depending on what you would like to see, lets use write:

cache> write_cache
WRITE_CACHE MENU:
display     – display current setting of write cache
enable      – enable write cache
disable     – disable write cache
!<cmd>      – execute <cmd>, then return
quit
write_cache> display
Write Cache is enabled
write_cache>

Use the same for read_cache and to disable and enable.

Giovanni@server:~/Downloads/filebench-1.4.8# pkg install SUNWfilebench
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                    1/1       60/60     0.32/0.32

PHASE                                        ACTIONS
Install Phase                                  82/82
Giovanni@server:~/Downloads/filebench-1.4.8#

and it was installed Use pkg search to search for packages.

Giovanni@server:/usr/benchmarks/filebench# bin/go_filebench
FileBench Version 1.4.4
filebench> load varmail
742: 3.707: Varmail Version 2.1 personality successfully loaded
742: 3.707: Usage: set $dir=<dir>
742: 3.707:        set $filesize=<size>    defaults to 16384
742: 3.707:        set $nfiles=<value>     defaults to 1000
742: 3.707:        set $nthreads=<value>   defaults to 16
742: 3.707:        set $meaniosize=<value> defaults to 16384
742: 3.707:        set $readiosize=<size>  defaults to 1048576
742: 3.707:        set $meandirwidth=<size> defaults to 1000000
742: 3.707: (sets mean dir width and dir depth is calculated as log (width, nfiles)
742: 3.707:  dirdepth therefore defaults to dir depth of 1 as in postmark
742: 3.707:  set $meandir lower to increase depth beyond 1 if desired)
742: 3.707:
742: 3.707:        run runtime (e.g. run 60)
filebench> set $dir=/gpool
filebench> run 60
742: 27.078: Creating/pre-allocating files and filesets
742: 27.081: Fileset bigfileset: 1000 files, 0 leafdirs avg dir = 1000000, avg depth = 0.5, mbytes=15
742: 27.096: Removed any existing fileset bigfileset in 1 seconds
742: 27.096: making tree for filset /gpool/bigfileset
742: 27.096: Creating fileset bigfileset…
742: 35.092: Preallocated 812 of 1000 of fileset bigfileset in 8 seconds
742: 35.092: waiting for fileset pre-allocation to finish
742: 35.092: Starting 1 filereader instances
744: 36.102: Starting 16 filereaderthread threads
742: 39.112: Running…
742: 99.712: Run took 60 seconds…
742: 99.713: Per-Operation Breakdown
closefile4                449ops/s   0.0mb/s      0.0ms/op        3us/op-cpu
readfile4                 449ops/s   7.0mb/s      0.0ms/op       19us/op-cpu
openfile4                 449ops/s   0.0mb/s      0.0ms/op       18us/op-cpu
closefile3                449ops/s   0.0mb/s      0.0ms/op        3us/op-cpu
fsyncfile3                449ops/s   0.0mb/s     17.4ms/op       20us/op-cpu
appendfilerand3           449ops/s   3.5mb/s      0.0ms/op       27us/op-cpu
readfile3                 449ops/s   7.0mb/s      0.0ms/op       18us/op-cpu
openfile3                 449ops/s   0.0mb/s      0.0ms/op       18us/op-cpu
closefile2                449ops/s   0.0mb/s      0.0ms/op        3us/op-cpu
fsyncfile2                449ops/s   0.0mb/s     17.9ms/op       17us/op-cpu
appendfilerand2           449ops/s   3.5mb/s      0.0ms/op       23us/op-cpu
createfile2               449ops/s   0.0mb/s      0.1ms/op       52us/op-cpu
deletefile1               449ops/s   0.0mb/s      0.0ms/op       33us/op-cpu

742: 99.713:
IO Summary:      353667 ops, 5836.1 ops/s, (898/898 r/w)  21.0mb/s,     78us cpu/op,   8.9ms latency
742: 99.713: Shutting down processes
filebench>
742: 110.144: Aborting…

Going back to normal

Giovanni@server:~# zpool create gpool raidz c8t1d0 c8t2d0 c8t3d0 c8t4d0 c8t5d0
Giovanni@server:~# zpool status
pool: gpool
state: ONLINE
scrub: none requested
config:

NAME        STATE     READ WRITE CKSUM
gpool       ONLINE       0     0     0
raidz1    ONLINE       0     0     0
c8t1d0  ONLINE       0     0     0
c8t2d0  ONLINE       0     0     0
c8t3d0  ONLINE       0     0     0
c8t4d0  ONLINE       0     0     0
c8t5d0  ONLINE       0     0     0

errors: No known data errors

pool: rpool
state: ONLINE
scrub: none requested
config:

NAME        STATE     READ WRITE CKSUM
rpool       ONLINE       0     0     0
c8t0d0s0  ONLINE       0     0     0

errors: No known data errors
Giovanni@server:~#

OK.

Giovanni@server:~# format
Searching for disks…done
AVAILABLE DISK SELECTIONS:
0. c8t0d0 <DEFAULT cyl 60797 alt 2 hd 255 sec 252>
/pci@0,0/pci15d9,d380@1f,2/disk@0,0
1. c8t1d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
/pci@0,0/pci15d9,d380@1f,2/disk@1,0
2. c8t2d0 <ATA-Hitachi HDS72202-A28A-1.82TB>
/pci@0,0/pci15d9,d380@1f,2/disk@2,0
3. c8t3d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
/pci@0,0/pci15d9,d380@1f,2/disk@3,0
4. c8t4d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
/pci@0,0/pci15d9,d380@1f,2/disk@4,0
5. c8t5d0 <ATA-Hitachi HDS72202-A3EA-1.82TB>
/pci@0,0/pci15d9,d380@1f,2/disk@5,0
Specify disk (enter its number):

Hard drives are located on /dev/dsk in Opensolaris. Compare to the zpool status and add drives that are new to the system (not yet in any storage pools)