Requirements
Xen3 is supported from NetBSD-4.0 onward. If you plan on using NetBSD-CURRENT, please read the article How to build NetBSD-current to do so. Also keep a spare partition for each guest operating system you intend to install.
This tutorial describes how to:
install and configure NetBSD as a DOM0
install and run a Debian system as a DOMU
Installing Xen tools and kernels
Xen tools
To run and administer xen domains, we need the xentools3 packages.
You can get them directly from pkgsrc, under sysutils/xentools3 for traditional xentools, and sysutils/xentools3-hvm for //HVM// goodies used to run un-modified OSes, through //HVM// support.
# cd /usr/pkgsrc/sysutils/xentools3
# make install
Xen kernel
Next, we will get the xen hypervisor kernel itself (sysutils/xenkernel3 in pkgsrc):
# cd /usr/pkgsrc/sysutils/xenkernel3
# make install
And copy it into / directory, like this:
# cp /usr/pkgsrc/sysutils/xenkernel3/work/xen.gz /xen
Xen DOM0 kernel
Lastly, we need a XEN-enabled kernel for our DOM0 domain. There are two possibilities: downloading the binary version, or building it from source.
Downloading the binary version
From NetBSD-4.0 onward, NetBSD supports Xen, and provides some XEN-enabled kernel directly from [1], in each binary/kernel directory associated with a particular release.
For example, with NetBSD-4.0, You can grab one from (ftp.NetBSD.org/pub/NetBSD/NetBSD-4.0/i386/binary/kernel/netbsd-XEN3_DOM0.gz [2]):
# ftp -a ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-4.0/i386/binary/kernel/netbsd-XEN3_DOM0.gz
The netbsd-XEN3_DOM0.gz file contains a gzipped version of the kernel. Just copy and move it into the root directory like this:
# cp netbsd-XEN3_DOM0.gz /netbsd-XEN3.DOM0
Building it from source
Building a kernel from source is out of the scope of this section. Please consult How to build a kernel from NetBSD’s FAQ for more details.
Once building is done, you can find them in /usr/src/obj/_releasedir_/i386/binary/kernel/. Identically to binary version, copy or move netbsd-XEN3_DOM0 in the root directory /.
Setting up Grub
Before installing and configuring XEN3, we need to install grub, the bootloader required to boot our XEN-enabled system.
To install it, use its package provided in pkgsrc (sysutils/grub):
# cd /usr/pkgsrc/sysutils/grub
# make install
Building grub may take a while.
When installation is finished, you can proceed to grub’s configuration in the /grub directory.
GRUB configuration
First, create /grub (if it is not present), and copy paste the menu.lst file below used by grub to read its configuration:
# mkdir /grub
# vi /grub/menu.lst
This typical /grub/menu.lst file should have following content:
# The default entry to load will be
# default=0
# However we do set it as saved
default saved
# boot after 7 seconds the default entry if the user didn’t hit keyboard
timeout=7
# Configure serial port to use as console. Ignore if you’ll use VGA only
# serial –unit=0 –speed=115200 –word=8 –parity=no –stop=1
# Let the user select which console to use (serial or VGA), default
# to serial after 10s
terminal –timeout=10 serial console
# First, entry for NetBSD/xen, using /netbsd-XEN3_DOM0 as the domain0 kernel, using VGA console.
# Domain0 will have 512MB RAM allocated.
# You can use this entry as first example.
# In case you have smaller RAM size you can insert desired value:
# kernel (hd0,a)/xen.gz dom0_mem=65536
# kernel (hd0,a)/xen.gz dom0_mem=131072
title Xen 3.0 / NetBSD_XEN3_DOM0 (hda0, vga)
root(hd0,0)
kernel (hd0,a)/xen dom0_mem=514624
module (hd0,a)/netbsd-XEN3_DOM0 root=/dev/hda1 ro console=tty0
# An entry for NetBSD/xen, using /netbsd as the domain0 kernel, and serial
# console. Domain0 will have 64MB RAM allocated.
# Assume NetBSD is installed in the first MBR partition.
title Xen 3.0 / NetBSD (hda0, serial)
root(hd0,0)
kernel (hd0,a)/xen dom0_mem=65536 com1=115200,8n1
module (hd0,a)/netbsd root=/dev/hda1 ro console=ttyS0
# An entry for NetBSD/xen, using /netbsd as the domain0 kernel, and serial
# console.
title Xen 3.0 / NetBSD_xen-serial_com1 (hda0, serial)
root(hd0,0)
kernel (hd0,a)/xen dom0_mem=65536 com1=115200,8n1
module (hdO,a)/netbsd root=/dev/hda1 ro console=ttyS0
# Same as above, but using VGA console
# We can use console=tty0 (Linux syntax) or console=pc (NetBSD syntax)
title Xen 3.0 / NetBSD-xen-vga (hda0, vga)
root(hd0,0)
kernel (hd0,a)/xen dom0_mem=65536
module (hd0,a)/netbsd root=/dev/hda1 ro console=tty0
# NetBSD/xen using a backup domain0 kernel (in case you installed a
# nonworking kernel as /netbsd
title X 3.0 / NetBSD-backup-serial_com1 (hda0, backup, serial)
root(hd0,0)
kernel (hd0,a)xen dom0_mem=65536 com1=115200,8n1
module (hd0,a)netbsd.backup root=/dev/hda1 ro console=ttyS0
# NetBSD/xen using a backup domain0 kernel and serial console (in case you installed a
# nonworking kernel as /netbsd.
title Xen 3.0 / NetBSD-backup-vga (hda0, backup, VGA)
root(hd0,0)
kernel (hd0,a)/xen dom0_mem=65536
module (hd0,a)/netbsd.backup root=/dev/hda1 ro console=tty0
# Load a regular NetBSD/i386 kernel. Can be useful if you end up with a
# nonworking /xen and for test purposes.
title NetBSD 4.0 pure load
root (hd0,a)
kernel /netbsd
# Load the NetBSD bootloader, letting it load the NetBSD/i386 kernel.
# May be better than the above, as grub can’t pass all required infos
# to the NetBSD/i386 kernel (e.g. console, root device, …)
title NetBSD chailnoalder
root (hd0,a)
chainloader +1
# Reboot computer
title Reboot computer
reboot
# Power off computer
title Power off computer
halt
# Load last saved boot
savedefault
Tip: If you hard drive is partitioned in some complicated ways (many partitions and slices for example), use the command-line grub:
# grub –no-floppy
The syntax used is the exact same as the one in the above config file, and you can use TAB-completion features. Very handy in case you do not know how to label the root partition with grub’s syntax, or check that the commands you type are properly written in your menu.lst config file.
Once you have finished editing menu.lst, proceed with the stage files copy, required for grub to work:
# grub-install /dev/rwd0d # considering wd0 is the boot device
Last, we need to install grub in the MBR. We will use the grub shell like commands to do so:
# grub –no-floppy
grub> root (hd0,a)
Filesystem type is ffs, partition type 0xa9
grub> setup (hd0)
Checking if “/boot/grub/stage1″ exists… no
Checking if “/grub/stage1″ exists… yes
Checking if “/grub/stage2″ exists… yes
Checking if “/grub/ffs_stage1_5″ exists… yes
Running “embed /grub/ffs_stage1_5 (hd0)”… 14 sectors are embedded.
succeeded
Running “install /grub/stage1 (hd0) (hd0)1+14 p (hd0,0,a)/grub/stage2 /grub/menu.lst”…
succeeded
Done.
Setting up DOM0
Creating xen devices
To create all xen devices, change to /dev and run ./MAKEDEV xen
cd /dev
./MAKEDEV xen
This should create the devices xencons, xenevt and xsd_kva. If any of these are missing you may not have updated to CURRENT using the latest sources and you will have to create the missing device files.
Configuring the bridge interface
The bridge(4) interface is used to provide network access to DOMUs.
To use one, edit (or create) the file /etc/ifconfig.bridge0 and insert following lines to:
create
!brconfig $int add bge0 stp bge0 up
Where ‘bge0′ should be changed to the name of the interface you want to use with your guest operating systems. use ifconfig(8) to get more details about your actual interfaces.
Rebooting into DOM0
Time to reboot:
# shutdown -r now
If all has gone well, you should have booted into the XEN3_DOM0 kernel. Check this with uname(1):
# uname -v
NetBSD 4.0 (XEN3_DOM0) #0: Sun Dec 16 01:20:31 PST 2007
builds@wb34:/home/builds/ab/netbsd-4-0-RELEASE/i386/200712160005Z-obj/home/builds/ab/netbsd-4-0-RELEASE/src/sys/arch/i386/compile/XEN3_DOM0
You should have XEN3 DOM0 mentioned somewhere.
Configuring rc scripts
Copy or symlink xend, xenbackendd and xendomains from /usr/pkg/share/examples/rc.d to /etc/rc.d.
# cp /usr/pkg/share/examples/rc.d/xend /etc/rc.d/
# cp /usr/pkg/share/examples/rc.d/xenbackendd /etc/rc.d/
# cp /usr/pkg/share/examples/rc.d/xendomains /etc/rc.d/
Edit /etc/rc.conf and add the following lines:
xend=YES
xenbackendd=YES
xendomains=”dom1″
Later on, when you have created a configuration file for ‘dom1′, the xendomains variable specified above will trigger ‘dom1′ to be started when the system is booted. At this point no configuration exists for dom1, therefore it does nothing at this point. If you choose to name your configuration file something else, adapt the name accordingly.
To avoid rebooting a second, start all three services:
# /etc/rc.d/xend start
# /etc/rc.d/xenbackendd start
# /etc/rc.d/xendomains start
Run ifconfig -a to ensure the bridge interface is present and issue a ps ax | grep xen to ensure you have a similar output.
12 ? DK 0:00.00 [xenwatch]
13 ? DK 0:00.00 [xenbus]
411 ? I 0:00.24 xenstored –pid-file=/var/run/xenstore.pid
594 ? IWa 0:00.26 xenconsoled
629 ? IW 0:00.00 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start
631 ? IWsa 0:00.02 /usr/pkg/sbin/xenbackendd
639 ? IWa 0:00.52 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start
The DOM0 configuration is now done. We will proceed to configuring DOMU domains.
Configuring DOMU
Configuring and installing a NetBSD DOMU
Create (or modify) /usr/pkg/etc/xen/dom1 and include this:
kernel = “/usr/src/obj/releasedir/i386/binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz”
#kernel = “/netbsd-XEN3_DOMU”
memory = 64
name = “dom1″
#vcpus = 1
disk = [ ‘phy:/dev/wd0g,0x03,w’,'file:/usr/src/obj/releasedir/i386/installation/cdrom/netbsd-i386.iso,0x04,r’ ]
vif = [ ‘bridge=bridge0′ ]
root = “/dev/wd0d”
This configuration boots into the NetBSD sysinst program and allows you to install a NetBSD DOMU using the normal sysinst method. This configuration uses a DOMU_INSTALL kernel and an ISO image provided by a successful ‘build release’ and ‘build iso-image’. You may be able to locate a valid Xen3 DOMU_INSTALL kernel from ftp://ftp.netbsd.org/pub/NetBSD-daily/ but if not, building a release is your best bet.
In this configuration file, /dev/wd0g is the reserved partition for the guest operating system. This should be changed to the partition you reserved prior to following the instructions within this document.
If you would like to use a physical CDROM instead of an ISO image, change the disk line to:
disk = [ ‘phy:/dev/wd0g,0x03,w’,'phy:/dev/cd0a,0x04,r’ ]
Now boot into sysinst using the command:
xm create dom1 -c
The reserved partition will appear as /dev/xbd0. Proceed as you would with a normal NetBSD installation using xbd0 as the target drive and xbd1 as the CDROM.
When you have finished, run shutdown -hp now to dom1.
Now edit /usr/pkg/etc/xen/dom1. Comment the INSTALL kernel and uncomment the DOMU kernel.
You should now have a working NetBSD DOMU (dom1). Boot into dom1 again with the command:
xm create dom1 -c
and ensure the file, /etc/ttys contains only this line or has all other lines commented:
console “/usr/libexec/getty Pc” vt100 on secure
and the file /etc/wscons.conf is completely empty or has all lines commented out. These last two steps ensure no errors should be present on boot. Setting wscons=NO in /etc/rc.conf may effectively do the same thing.
From here, configure /etc/rc.conf and all runtime configuration files as you would normally. The network interface name should be xennet0. Use this name when configuring an IP address.
More information can be obtained by referencing the Xen user guide and the official NetBSD Xen Howto. Questions can be addressed to the port-xen@NetBSD.org mailling list.
Configuring and installing a GNU/Linux DOMU
We will do this in two steps:
install a GNU/Linux system, from a livecd or any installation media
configure the DOM0 so that it can create and start the Linux DOMU.
Installing a Linux distribution (soon-to-be DOMU)
Before proceeding with DOMU configuration, we will install our favorite GNU/Linux distribution on the computer.
In order to do it, we need at least two partitions (only one, if you do not consider the swap). These partitions must reside outside of the NetBSD slice, and may be either of primary or extended type. Of course, you can use more than two, but adapt your labels and partitions accordingly.
We do not cover the partition/slices manipulations through fdisk(8) and disklabel(8), as it depends strongly on how you manage your hard drive’s space.
For this tutorial, we will use this partitioning:
# fdisk /dev/wd0d
fdisk: removing corrupt bootsel information
fdisk: Cannot determine the number of heads
Disk: /dev/wd0d
NetBSD disklabel disk geometry:
cylinders: 486344, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
total sectors: 490234752
BIOS disk geometry:
cylinders: 1023, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
total sectors: 490234752
Partition table:
0: Linux native (sysid 131)
start 63, size 20482812 (10001 MB, Cyls 0-1274)
PBR is not bootable: All bytes are identical (0x00)
1: Linux swap or Prime or Solaris (sysid 130)
start 20482875, size 1959930 (957 MB, Cyls 1275-1396)
PBR is not bootable: All bytes are identical (0x00)
2: NetBSD (sysid 169)
start 61464690, size 428770062 (209360 MB, Cyls 3826-30515/178/63), Active
3:
Drive serial number: -286527765 (0xeeebeeeb)
Here, you notice that we decide to use two primary partitions for our future Linux DOMU:
partition 0 (for the root directory /)
partition 1 (for the swap)
Labels:
16 partitions:
# size offset fstype [fsize bsize cpg/sgs]
a: 30720816 61464690 4.2BSD 2048 16384 0 # (Cyl. 60976*- 91453*)
b: 1049328 92185506 swap # (Cyl. 91453*- 92494*)
c: 428770062 61464690 unused 0 0 # (Cyl. 60976*- 486343)
d: 490234752 0 unused 0 0 # (Cyl. 0 - 486343)
e: 20480000 93234834 4.2BSD 0 0 0 # (Cyl. 92494*- 112812*)
f: 20480000 113714834 4.2BSD 0 0 0 # (Cyl. 112812*- 133129*)
g: 20480000 134194834 4.2BSD 0 0 0 # (Cyl. 133129*- 153447*)
h: 335559918 154674834 4.2BSD 0 0 0 # (Cyl. 153447*- 486343)
i: 20482812 63 Linux Ext2 0 0 # (Cyl. 0*- 20320*)
j: 1959930 20482875 swap # (Cyl. 20320*- 22264*)
Bear in mind that we added two labels here, namely i and j, which maps respectively to partition 0 and partition 1 of the disk. We will use these labels later for DOMU configuration.
Now that we have partitioned the disk, proceed with installing your Linux distribution. We will not cover that part in this tutorial. You can either install it from an installation media (a cdrom from example), or copy files from an already installed distribution on your computer.
Tip: to manipulate ext2/3 filesystems (the traditional fs under Linux) from NetBSD, you can use sysutils/e2fsprogs from pkgsrc:
# cd /usr/pkgsrc/sysutils/e2fsprogs
# make install
And then use e2fsck, mke2fs and mount_ext2fs(8) directly from NetBSD.
Getting XEN aware Linux kernels
Once installation is done, reboot your computer and return to our Xen-NetBSD system.
To boot our Linux DOMU, we will need a Linux kernel supporting the XENU virtualisation. Depending on your Linux distribution, you can grab one from its repository (it is up to you to find it through aptitude, yum or whatever package manager you use), or get one from the Xen binary distribution.
To get a XENU Linux kernel from Xen binary distribution, get it directly from Xen website download page. Download the tarball and extract the vmlinuz-*-xen from it. In our case, with a 2.6.18 Linux kernel:
# ftp -a http://bits.xensource.com/oss-xen/release/3.1.0/bin.tgz/xen-3.1.0-install-x86_32.tgz
# cd /tmp
# tar -xzf xen-3.1.0-install-x86_32.tgz dist/install/boot/vmlinuz-2.6.18-xen
vmlinuz-2.6.18-xen is the kernel that Xen will use to start the DOMU. Move it to any directory you like (just remember it when configuring the kernel entry in the DOMU configuration file):
# mv dist/install/boot/vmlinuz-2.6.18-xen /vmlinuz-XEN3-DOMU
Configuring DOMU
Configuring the Linux DOMU is a bit different than a NetBSD one; some options tend to differ.
Edit (or create) the configuration file domu-linux, in /usr/pkg/etc/xen/:
# vi /usr/pkg/etc/xen/domu-linux
Here’s a typical config file for a Linux DOMU:
#—————————————————————————-
# Kernel image file. This kernel will be loaded in the new domain.
kernel = “/vmlinuz-XEN3-DOMU”
# Memory allocation (in megabytes) for the new domain.
memory = 256
# A handy name for your new domain. This will appear in ‘xm list’,
# and you can use this as parameters for xm in place of the domain
# number. All domains must have different names.
#
name = “domu-linux”
# Which CPU to start domain on (only relevant for SMP hardware). CPUs
# numbered starting from “0′’.
#
cpu = “^1″ # leave to Xen to pick
#—————————————————————————-
# Define network interfaces for the new domain.
# Number of network interfaces (must be at least 1). Default is 1.
vif = [ ‘’ ]
# Define MAC and/or bridge for the network interfaces.
#
# The MAC address specified in “mac'’ is the one used for the interface
# in the new domain. The interface in domain0 will use this address XOR’d
# with 00:00:00:01:00:00 (i.e. aa:00:00:51:02:f0 in our example). Random
# MACs are assigned if not given.
#
# “bridge'’ is a required parameter, which will be passed to the
# vif-script called by xend(8) when a new domain is created to configure
# the new xvif interface in domain0.
#
# In this example, the xvif is added to bridge0, which should have been
# set up prior to the new domain being created — either in the
# “network'’ script or using a /etc/ifconfig.bridge0 file.
#
vif = [ ‘mac=aa:00:00:50:02:f0, bridge=bridge0′ ]
#—————————————————————————-
# Define the disk devices you want the domain to have access to, and
# what you want them accessible as.
#
# Each disk entry is of the form:
#
# phy:DEV,VDEV,MODE
#
# where DEV is the device, VDEV is the device name the domain will see,
# and MODE is r for read-only, w for read-write. You can also create
# file-backed domains using disk entries of the form:
#
# file:PATH,VDEV,MODE
#
# where PATH is the path to the file used as the virtual disk, and VDEV
# and MODE have the same meaning as for “phy'’ devices.
#
# /dev/wd0i will be seen as “hda1″ under DOMU (the root partition)
# /dev/wd0j will be seen as “hda2″ under DOMU (the swap)
#
disk = [ ‘phy:/dev/wd0i,hda1,w’,'phy:/dev/wd0j,hda2,w’ ]
#—————————————————————————-
# Set the kernel command line for the new domain.
# Set root device.
root = “/dev/hda1″
Now, you should be able to start your first Linux DOMU!
# xm create -c /usr/pkg/etc/xen/domu-linux
If you intend to have more than one box configured with the above configuration on the same network, you will most likely have to specify a unique MAC address per guest OS, otherwise it is likely you will have a conflict. I’m not sure if the MAC assignment is random, incremental or if Xen is able to check for the existence of the proposed MAC address, so specifying the MAC address is recommended.
Here is a method to assign a MAC address to a newly created Xen Domu. First as described before, use the following vif parameter in your config file :
vif = [ ‘bridge=bridge0′ ]
Then, run the Xen DomU and, once logged run the following command :
# ifconfig xennet0
Output Sample :
xennet0: flags=8863 mtu 1500
capabilities=2800
enabled=0
address: 00:16:3e:2e:32:5f
inet 192.168.4.81 netmask 0xffffff00 broadcast 192.168.4.255
inet6 fe80::216:3eff:fe2e:325f%xennet0 prefixlen 64 scopeid 0x2
A MAC address is automaticaly generated, use it in your config file with the following syntax :
vif = [ ‘mac=00:16:3e:2e:32:5f, bridge=bridge0′ ]
And now you can restart the Xen DomU.
Please, note that the MAC Address MUST start with “00:16:3e”.