Run CoreOS on FreeBSD's bhyve
by Emile `iMil' Heitor - 2016-06-21
No, I’m not following the hype, only I like to test things plus I feel there will be a growing demand for docker at ${DAYWORK}. I read here and there that CoreOS was the Linux distribution of choice to play with docker, so while at it, I picked up this one to dive into the container world. Finally, I’ve been willing to put my hands on bhyve for quite a while, so I took this opportunity to learn all those new (to me) technologies at once.
Here I’ll write down a quick set of commands and configuration files I found, discovered or modified.
First thing, we’ll have to prepare our FreeBSD system for bhyve. Load the vmm and mndm, respectively bhyve and the serial device module. We’ll also need a tap device and bridging capability in order to bring network to our future virtual machine:
$ cat /boot/loader.conf
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
In /etc/rc.conf, we’ll bring up the tap interface and attach it to a bridge, which will be bond to our physical interface:
cloned_interfaces="bridge0 tap0"
ifconfig_bridge0="addm em0 addm tap0 up"
In order to bring up the tap interface when it’s opened, add the following to /etc/sysctl.conf:
net.link.tap.up_on_open=1
Now that the host OS is ready, let’s fetch CoreOS ISO image. The following link might change, check https://coreos.com/os/docs/latest/booting-with-iso.html
$ curl -O https://stable.release.core-os.net/amd64-usr/current/coreos_production_iso_image.iso
Let’s create a ZFS volume to receive the system:
$ sudo zfs create -V16G -o volmode=dev zroot/coredisk0
To boot a Linux bhyve virtual machine, you’ll need the grub-bhyve package, simply install it using pkg(8):
$ sudo pkg install grub2-bhyve
You’ll then have to create a device.map file:
$ cat device.map
(hd0) /dev/zvol/zroot/coredisk0
(cd0) /usr/home/imil/iso/coreos_production_iso_image.iso
Copy the commands to give to grub in a text file:
$ cat grub-cd0.cfg
linux (cd0)/coreos/vmlinuz coreos.autologin
initrd (cd0)/coreos/cpio.gz
boot
And load the kernel:
$ sudo grub-bhyve -m device.map -r cd0 -M 1024M coreos < grub-cd0.cfg
You can now fire up the virtual machine:
$ sudo bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/coredisk0 -l com1,stdio -c 2 -m 1024M coreos
Once the OS has booted, you should be able to interact with the virtual machine serial console. If there’s no DHCP server on your network, you might have to add an IP address, default route and nameserver:
$ sudo -s
# ip a a 192.168.1.100/24 dev eth0
# route add default gw 192.168.1.254
# echo 'nameserver 8.8.8.8' > /etc/resolv.conf
It is now time to install the OS to the ZFS volume, seen as a block device from the virtual machine point of view. You’ll find the associated device in /dev/disk/by-id:
$ sudo coreos-install -d /dev/disk/by-id/virtio-BHYVE-580E-3135-7E05 -C stable
Once finished, just shutdown the VM you booted from the ISO image:
$ sudo halt
And kill the virtual machine:
$ sudo bhyvectl --destroy --vm=coreos
Create a new set of grub commands in order to boot from the ZFS volume:
$ cat grub-hd0.cfg
linux (hd0,gpt1)/coreos/vmlinuz-a console=ttyS0 ro root=LABEL=ROOT usr=LABEL=USR-A coreos.autologin
boot
Note the usr=LABEL=USR-A parameter, it took me a little while to figure out why the system would fail at mounting /usr, as no documentation I found mentionned this.
Load the commands using grub-bhyve:
sudo grub-bhyve -m device.map -r hd0 -M 1024M coreos < grub-hd0.cfg
And finally, fire up the resulting operating system:
$ sudo bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/coredisk0 -l com1,stdio -c 2 -m 1024M coreos
You might want to run a headless system, in which case you’ll have to replace stdio with /dev/nmdm0A and connect to the other end of the serial device (/dev/nmdm0B) with any serial console utility like cu(1) or screen.
Resources: