Playing with KVM virtualisation – part 1.1

Looks like I bumped into a small issue when updating one of the VMs (Debian) I was playing with: Shortage of diskspace on the root volume. Disaster when running dist-upgrade!

# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    1.5G  1.4G     0 100% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  224K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Fortunately, it’s an LVM so it’s not so difficult to extend this, even on-the-fly.

Let’s create our new disk first

virsh# vol-create-as MyVolume vm2-2.img 2G --format raw
Vol vm2-2.img created

virsh # vol-list --details MyVolume
Name                 Path                                  Type   Capacity  Allocation
--------------------------------------------------------------------------------------
Android-x86-4.0.img  /virtualmachines/Android-x86-4.0.img  file   4,00 GiB  498,75 MiB
cvm3.img             /virtualmachines/cvm3.img             file  16,00 GiB   61,81 MiB
vm2-2.img            /virtualmachines/vm2-2.img            file   2,00 GiB    2,00 GiB
vm2.img              /virtualmachines/vm2.img              file   4,00 GiB    1,83 GiB

Now that we have created the volume, let’s attach it to the VM. It is possible while it’s running live.

virsh # list
 Id    Name                           State
----------------------------------------------------
 4     vm2                            running

virsh # attach-disk vm2 /virtualmachines/vm2-2.img vdc
Disk attached successfully

Within the VM it’s automatically detected as well.

vm2# tail /var/log/kern.log
...
Dec 21 11:58:00 vm2 kernel: [ 3978.289939]  vdc: unknown partition table

The new disk is there, but it has not been partitioned yet. So let’s do that

root@vm2:/# fdisk /dev/vdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xc9f8eafe.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): m
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition's system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-4095999, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-4095999, default 4095999):
Using default value 4095999

Command (m for help): t
Selected partition 1

Hex code (type L to list codes): 8e

8e is the code for Linux LVM

Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Done! Now we need to add it to our LVM. First let’s check the volume group (vm2 here).

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               3.76 GiB
  PE Size               4.00 MiB
  Total PE              962
  Alloc PE / Size       962 / 3.76 GiB
  Free  PE / Size       0 / 0   
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

As you can see, no Free space left (Free PE / Size). We are ready to add our new disk and its freshly created partition

# vgextend vm2 /dev/vdb1
No physical volume label read from /dev/vdb1
Physical volume "/dev/vdb1" successfully created
Volume group "vm2" successfully extended

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               5.71 GiB
  PE Size               4.00 MiB
  Total PE              1461
  Alloc PE / Size       1218 / 4.76 GiB
  Free  PE / Size       243 / 972.00 MiB
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

Great! The new space has been allocated (VG Size, Free PE / Size). Now we are ready to exptend the root volume

# lvextend -L+1G /dev/vm2/root
Extending logical volume root to 2.47 GiB
Logical volume root successfully resized
# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    2.5G  1.4G  945M  60% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  228K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Just extending the logical volume does not automatically add the free space to the filesystem, that needs to be resized as well.

# resize2fs -p /dev/vm2/root 
resize2fs 1.42.8 (20-Jun-2013)
Filesystem at /dev/vm2/root is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/vm2/root is now 648192 blocks long.

root@vm2:/dev# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/vm2-root    2.5G  1.4G  945M  60% /
udev                     10M     0   10M   0% /dev
tmpfs                    25M  224K   25M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                    49M     0   49M   0% /run/shm
/dev/vda1               228M   58M  158M  27% /boot
/dev/mapper/vm2-home    2.1G  3.6M  2.0G   1% /home

Great success! One last look to the volume group stats shows that not all space has been allocated yet.

# vgdisplay
  --- Volume group ---
  VG Name               vm2
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               5.71 GiB
  PE Size               4.00 MiB
  Total PE              1461
  Alloc PE / Size       1218 / 4.76 GiB
  Free  PE / Size       243 / 972.00 MiB
  VG UUID               ksC1ha-y05g-E6gx-JshG-b09N-dw6q-kcF2t7

In case we need some tweaking to other logical volumes, it can be added accordingly.

Playing with KVM virtualisation

When I attended Loadays 2013 back in the beginning of April, I was introduced to Ansible, an interesting configuration management system. In order to properly play with it I wanted to have a couple of hosts that could be messed with. Virtual machines are perfect for that. For the exercise I opted for KVM. No particular reason, other options are just as fine for this. This is also mainly a beginning reference for myself, so the setup can probably be handled in a much better and efficient way (don’t hesitate to tell me, of course).

For the impatient, the source files as example below can be found in this Bitbucket repository

Prerequisites

Create bridge interface

First of all, in order to have different IP addresses assigned, and you want to have the VMs connect directly on your physical network, you can create a bridge interface br0, which all the VMs can use for networking. In the example scripts below, this way is used.

A good starting point for more information on setting this up on Debian can be found on the Debian Wiki BridgeNetworkConnections page.

LibVirt

For this setup, we’re using libVirt

Webserver to host files

For testing, I installed a simple Apache Webserver which hosted the files in its root directory. YMMV (which is also recommended)

Create virtual machines

Of course, this can all be done through the virt-manager GUI as well, but when working remote, without an X-server or VNC at your disposal, it’s handy to know some command-line as well. And to reduce typing the same thing over and over (although it’s not bad to get the commands in your muscle memory), let’s put it in a small script. (Inspiration was found here)

Creation scripts

Note: The example scripts are for testing purposes, and are meant to generate a small image. Your usage requirements will most probably differ.

#!/bin/sh
#
# A script to install Debian Wheezy on a KVM guest
# using automatic installation from installable distribution
# image on an HTTP server

# Defined MAC range 52:54:00:8E:FF:00 - 52:54:00:8E:FF:FF
# Choose the last number to your own preference
 
if [ $# -ne 1 ]
then
echo "Usage: $0 guest-name"
exit 1
fi

size=2
CPUs=2
RAM=512
MyVMMACRANGE='52:54:00:8e:ff'
VMMAC=${MyVMMACRANGE}:00
domain=home
wwwhost=http://example.com
 
virt-install \
--connect=qemu:///system \
--virt-type kvm \
--name=${1} \
--ram=${RAM} \
--vcpus=${CPUs} \
--disk path=/var/lib/libvirt/images/${1}.img,size=${size} \
-l http://ftp.be.debian.org/debian/dists/wheezy/main/installer-amd64/ \
--vnc \
--os-variant debianwheezy \
--noautoconsole \
--hvm \
--network bridge=br0,mac=${VMMAC} \
--extra-args="auto=true hostname=${1} domain=${domain} url=${wwwhost}/preseed-debian_vm.txt text console=tty1 console=ttyS0,115200"

Call this script like this

# create_vm-debianwheezy.sh vm1

and it will result in a call to libvirt to create a new virtual machine, with 2 CPUs, 512MB RAM and 2 GB image on disk. 1 CPU and 256MB also work fine, you can easily define that in the script (or later in the generated xml config file for the VM, located at /etc/libvirt/qemu/vm1.xml). It also has a console that you can attach to from within virsh.

To see the progress of the installation, type

# virsh console vm1

The -l parameter in the script tells us to connect to the Debian mirror to fetch the online installation image for Wheezy. It also adds some kernel parameters with –extra-args which tells the installer to fetch a pre-configured preseed file from the wwwhost you defined above.

Headless installation files

An example preseed.txt file has been put in the www subdirectory of the repository. It will create a minimal install of Debian Wheezy, with a belgian mirror configured. Its filesystem is only 1 partition without LVM. No accessible root user is configured, but the pre-defined “myuser” has sudo rights for root permissions. Some useful (YMMV) packages are also automatically installed.

CentOS headless installation

I’ve also added VM-creation script for CentOS. There is a corresponding Kickstart file which gives a hardened CentOS image. The main source of inspiration was found in RackerHackers Github repository

VM fun

Listing the VMs
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1     vm1                            running
 2     vm2                            shut off
Starting the VM

# virsh start vm1
Domain vm1 started

Stopping the VM

# virsh shutdown vm1
Domain vm1 is being shutdown

virsh commands and help
# virsh help 
... 
 Virsh itself (help keyword 'virsh'):
    cd                             change the current directory
    echo                           echo arguments
    exit                           quit this interactive terminal
    help                           print help
    pwd                            print the current directory
    quit                           quit this interactive terminal

Cloning fun

If you want to backup or clone your VM, you can use the virt-clone command.

# virt-clone -o vm1 -n vm2 -m 52:54:00:8E:FF:02 -f /var/lib/libvirt/images/vm2.img

A new VM will be created, exactly the same as the original image, except for the name, MAC address and image-file. Also note that the ssh-host keys will be identical, so they’d need to be regenerated.

The End

With enough tiny clones, you are ready for the other fun stuff: orchestration. Those first steps will probably be dealt with in a future blogpost about Ansible and/or Puppet

And to end, for those who want to go even further in this:

And most of all, have fun, if you feel something needs to be added or corrected, let me know!