The bhyve BSD-licensed hypervisor became part of the base system with FreeBSD 10.0-RELEASE. This hypervisor supports a number of guests, including FreeBSD, OpenBSD, and many Linux® distributions. By default, bhyve provides access to serial console and does not emulate a graphical console. Virtualization offload features of newer CPUs are used to avoid the legacy methods of translating instructions and manually managing memory mappings.
The bhyve design requires a
processor that supports Intel® Extended Page Tables
(EPT) or AMD® Rapid Virtualization Indexing
(RVI) or Nested Page Tables
(NPT). Hosting Linux® guests or FreeBSD guests
with more than one vCPU requires
VMX unrestricted mode support
(UG). Most newer processors, specifically
the Intel® Core™ i3/i5/i7 and Intel® Xeon™
E3/E5/E7, support these features. UG support
was introduced with Intel's Westmere micro-architecture. For a
complete list of Intel® processors that support
EPT, refer to http://ark.intel.com/search/advanced?s=t&ExtendedPageTables=true.
RVI is found on the third generation and
later of the AMD Opteron™ (Barcelona) processors. The easiest
way to tell if a processor supports
bhyve is to run
dmesg or look in
/var/run/dmesg.boot for the
POPCNT processor feature flag on the
Features2 line for AMD® processors or
EPT and UG on the
VT-x line for Intel® processors.
The first step to creating a virtual machine in bhyve is configuring the host system. First, load the bhyve kernel module:
#kldload vmm
Then, create a tap interface for the
network device in the virtual machine to attach to. In order
for the network device to participate in the network, also
create a bridge interface containing the
tap interface and the physical interface
as members. In this example, the physical interface is
igb0:
#ifconfigtap0create#sysctl net.link.tap.up_on_open=1net.link.tap.up_on_open: 0 -> 1#ifconfigbridge0create#ifconfigbridge0addmigb0addmtap0#ifconfigbridge0up
Create a file to use as the virtual disk for the guest machine. Specify the size and name of the virtual disk:
#truncate -s16Gguest.img
Download an installation image of FreeBSD to install:
#fetchFreeBSD-10.3-RELEASE-amd64-bootonly.iso 100% of 230 MB 570 kBps 06m17sftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.3/FreeBSD-10.3-RELEASE-amd64-bootonly.iso
FreeBSD comes with an example script for running a virtual
machine in bhyve. The script will
start the virtual machine and run it in a loop, so it will
automatically restart if it crashes. The script takes a
number of options to control the configuration of the machine:
-c controls the number of virtual CPUs,
-m limits the amount of memory available to
the guest, -t defines which
tap device to use, -d
indicates which disk image to use, -i tells
bhyve to boot from the
CD image instead of the disk, and
-I defines which CD image
to use. The last parameter is the name of the virtual
machine, used to track the running machines. This example
starts the virtual machine in installation mode:
#sh /usr/share/examples/bhyve/vmrun.sh -c1-m1024M-ttap0-dguest.img-i -IFreeBSD-10.3-RELEASE-amd64-bootonly.isoguestname
The virtual machine will boot and start the installer. After installing a system in the virtual machine, when the system asks about dropping in to a shell at the end of the installation, choose .
Reboot the virtual machine. While rebooting the virtual
machine causes bhyve to exit, the
vmrun.sh script runs
bhyve in a loop and will automatically
restart it. When this happens, choose the reboot option from
the boot loader menu in order to escape the loop. Now the
guest can be started from the virtual disk:
#sh /usr/share/examples/bhyve/vmrun.sh -c4-m1024M-ttap0-dguest.imgguestname
In order to boot operating systems other than FreeBSD, the sysutils/grub2-bhyve port must be first installed.
Next, create a file to use as the virtual disk for the guest machine:
#truncate -s16Glinux.img
Starting a virtual machine with
bhyve is a two step process. First
a kernel must be loaded, then the guest can be started. The
Linux® kernel is loaded with
sysutils/grub2-bhyve. Create a
device.map that
grub will use to map the virtual
devices to the files on the host system:
(hd0) ./linux.img (cd0) ./somelinux.iso
Use sysutils/grub2-bhyve to load the Linux® kernel from the ISO image:
#grub-bhyve -m device.map -r cd0 -M1024Mlinuxguest
This will start grub. If the installation
CD contains a
grub.cfg, a menu will be displayed.
If not, the vmlinuz and
initrd files must be located and loaded
manually:
grub>ls(hd0) (cd0) (cd0,msdos1) (host) grub>ls (cd0)/isolinuxboot.cat boot.msg grub.conf initrd.img isolinux.bin isolinux.cfg memtest splash.jpg TRANS.TBL vesamenu.c32 vmlinuz grub>linux (cd0)/isolinux/vmlinuzgrub>initrd (cd0)/isolinux/initrd.imggrub>boot
Now that the Linux® kernel is loaded, the guest can be started:
#bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0-s 3:0,virtio-blk,./linux.img\ -s 4:0,ahci-cd,./somelinux.iso-l com1,stdio -c4-m1024Mlinuxguest
The system will boot and start the installer. After installing a system in the virtual machine, reboot the virtual machine. This will cause bhyve to exit. The instance of the virtual machine needs to be destroyed before it can be started again:
#bhyvectl --destroy --vm=linuxguest
Now the guest can be started directly from the virtual disk. Load the kernel:
#grub-bhyve -m device.map -r hd0,msdos1 -Mgrub>1024Mlinuxguestls(hd0) (hd0,msdos2) (hd0,msdos1) (cd0) (cd0,msdos1) (host) (lvm/VolGroup-lv_swap) (lvm/VolGroup-lv_root) grub>ls (hd0,msdos1)/lost+found/ grub/ efi/ System.map-2.6.32-431.el6.x86_64 config-2.6.32-431.el6.x 86_64 symvers-2.6.32-431.el6.x86_64.gz vmlinuz-2.6.32-431.el6.x86_64 initramfs-2.6.32-431.el6.x86_64.img grub>linux (hd0,msdos1)/vmlinuz-2.6.32-431.el6.x86_64 root=/dev/mapper/VolGroup-lv_rootgrub>initrd (hd0,msdos1)/initramfs-2.6.32-431.el6.x86_64.imggrub>boot
Boot the virtual machine:
#bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0\ -s 3:0,virtio-blk,./linux.img-l com1,stdio -c4-m1024Mlinuxguest
Linux® will now boot in the virtual machine and eventually present you with the login prompt. Login and use the virtual machine. When you are finished, reboot the virtual machine to exit bhyve. Destroy the virtual machine instance:
#bhyvectl --destroy --vm=linuxguest
In addition to bhyveload and grub-bhyve, the bhyve hypervisor can also boot virtual machines using the UEFI userspace firmware. This option may support guest operating systems that are not supported by the other loaders.
In order to make use of the UEFI support in bhyve, first obtain the UEFI firmware images. This can be done by installing sysutils/bhyve-firmware port or package.
With the firmware in place, add the flags
-l bootrom,
to your bhyve command line.
The actual bhyve command may look
like this:/path/to/firmware
#bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \ -s 2:0,virtio-net,tap1-s 3:0,virtio-blk,./disk.img\ -s 4:0,ahci-cd,./install.iso-c4-m1024M\ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd\guest
sysutils/bhyve-firmware also contains a CSM-enabled firmware, to boot guests with no UEFI support in legacy BIOS mode:
#bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \ -s 2:0,virtio-net,tap1-s 3:0,virtio-blk,./disk.img\ -s 4:0,ahci-cd,./install.iso-c4-m1024M\ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CSM.fd\guest
The UEFI firmware support is particularly useful with predominantly graphical guest operating systems such as Microsoft Windows®.
Support for the UEFI-GOP framebuffer may also be enabled
with the -s 29,fbuf,tcp=
flags. The framebuffer resolution may be configured with
0.0.0.0:5900w= and
800h=, and
bhyve can be instructed to wait for
a VNC connection before booting the guest
by adding 600wait. The framebuffer may be
accessed from the host or over the network via the
VNC protocol.
The resulting bhyve command would look like this:
#bhyve -AHP -s 0:0,hostbridge -s 31:0,lpc \ -s 2:0,virtio-net,tap1-s 3:0,virtio-blk,./disk.img\ -s 4:0,ahci-cd,./install.iso-c4-m1024M\ -s 29,fbuf,tcp=0.0.0.0:5900,w=800,h=600,wait \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd\guest
Note, in BIOS emulation mode, the framebuffer will cease receiving updates once control is passed from firmware to guest operating system.
If ZFS is available on the host machine, using ZFS volumes instead of disk image files can provide significant performance benefits for the guest VMs. A ZFS volume can be created by:
#zfs create -V16G-o volmode=devzroot/linuxdisk0
When starting the VM, specify the ZFS volume as the disk drive:
#bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0-s3:0,virtio-blk,/dev/zvol/zroot/linuxdisk0\ -l com1,stdio-c4-m1024Mlinuxguest
It is advantageous to wrap the
bhyve console in a session
management tool such as sysutils/tmux or
sysutils/screen in order to detach and
reattach to the console. It is also possible to have the
console of bhyve be a null modem
device that can be accessed with cu. To do
this, load the nmdm kernel module and
replace -l com1,stdio with
-l com1,/dev/nmdm0A. The
/dev/nmdm devices are created
automatically as needed, where each is a pair, corresponding
to the two ends of the null modem cable
(/dev/nmdm0A and
/dev/nmdm0B). See nmdm(4) for more
information.
#kldload nmdm#bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0-s 3:0,virtio-blk,./linux.img\ -l com1,/dev/nmdm0A-c4-m1024Mlinuxguest#cu -lConnected Ubuntu 13.10 handbook ttyS0 handbook login:/dev/nmdm0B
A device node is created in /dev/vmm for each virtual
machine. This allows the administrator to easily see a list
of the running virtual machines:
#ls -al /dev/vmmtotal 1 dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./ dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../ crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest
A specified virtual machine can be destroyed using
bhyvectl:
#bhyvectl --destroy --vm=guestname
In order to configure the system to start bhyve guests at boot time, the following configurations must be made in the specified files:
/etc/sysctl.conf
net.link.tap.up_on_open=1
/etc/rc.conf
cloned_interfaces="bridge0tap0" ifconfig_bridge0="addmigb0addmtap0kld_list="nmdm vmm"
All FreeBSD documents are available for download at https://download.freebsd.org/ftp/doc/
Questions that are not answered by the
documentation may be
sent to <freebsd-questions@FreeBSD.org>.
Send questions about this document to <freebsd-doc@FreeBSD.org>.