HomeRamblings  ⁄  SystemsServers

Jumpstarting your Virtual tour with Oracle VM

Published: October 29, 2008 (about 9 years ago)
Updated: over 2 years ago

Server virtualization technology has come a long ways in the last 18 months (today being a wonderful October 28, 2008 crisp, cold Fall Sunny day).  Here, I am going to take a look at Oracle VM, one of the relative newcomers to the game and show you how to quickly get up and running with a Xen host and virtual server.

A little background…

Our shop has been using virtual technologies for just over two years. At the time we started taking it seriously, the open source offerings were just on the cusp of being production quality; Citrix had not yet bought into Xen; and VMWare had just rolled out their free VMWare Server option.  We had a mixture of Linux and Windows servers, all physical, and were really feeling some of the budget pince in our rapid growth. We decided it was time to rethink how we were growing our data-center and that led us to try out VMWare and Xen side-by-side, eventually siding with Xen since the bulk of our servers turned out to be Linux hosts while our predominant application, SQL Server 2005 was not all that conducive to being virtualized without some heavy, heavy investment in hardware purchases. The gameplan, get some beefy servers, run Xen on them, with images residing on SAN and migrate all our physical Linux hosts to virtualized environment. Once converted, we’d simply reclaim all the excess hardware to fuel the Windows SQL Server hunger. So, off we went. We purchased a three of Dell’s first dual quad-core servers at 1.66ghz and loaded them up with 16gb of RAM and a Dell AX150i iSCSI dual storage processor SAN to run our wonderfully envisioned HA cluster off of. Xen had just turned 3.0 at the time and we really went through a lot of permutations in our stress testing and daily use before we finally came up with a stable blending of several offerings from the open source community. As wild as it may seem, we settled on Oracle VM’s first release as the Xen server, CentOS 5 repositories to manage packages on the host, Red Hat’s kickstart project to build the VM’s and ran a mixed guest environment of CentOS 4, CentOS 5, and Fedora Core 8. The guest environment’s heterogeneity had a purpose for existing. You see, our clients often dictated what version of Red Hat they wanted us to support when we deployed our systems in their data center, so we needed these three items (all relatively free of compulsory licensing fees) to give us the ability to effectively emulate any of our clients’ data center in-house for Q/A, stress/load, and deployment “dry-run” testing purposes. After trying for many moons to effect a HA clustering solution that was rock-solid and backed by the SAN, we sadly had to regress from that vision (for now). VM images that were once all homed on the SAN and connected via iSCSI to the Xen hosts simply overwhelmed our SAN with I/O dropping low enough to force the filesystems of the guests into read-only state. We gave up the HA vision (for now) and purchased a bunch of very fast SCSI drives and RAID 5’d them, filling up each of the Xen Host’s drive bays and relocated the VM images to each Xen host’s local volumes. This infrastructure has been our “norm” for the better part of the year now and we have a rock-solid deployment running about 80 virtualized Linux hosts on just four Dell servers. The ease with which we can set up a new server or tear one down and dismiss it has had quite an effect on our ability to test widely varying deployment scenarios and automate the deployments themselves.

Fast forward October 2008

The newest version of Xen from Red Hat (and hence CentOS for us) is looking rather sharp. Virtually all the issues we were encountering with just standalone Xen hosts appear to be addressed (in our limited testing). One of the more critical one, where our Dom0 environment would become unstable and we’d lose ability to manage it has been fixed (our lone-remaining headache in the above setup). We have not, however, ventured to test whether our HA woes have been resolved as well. One other thing we did notice, though was that VM guests shutdown and launch a whole lot faster than they do on our production servers, so that could hint to some very good performance improvements lurking in the newest releases of Xen., The downside: Six, yes count ‘em, six CDs must be downloaded to install the CentOS Xen version. Oracle VM, by comparison clocks in at roughly ONE 500mb CD image plus two companion CDs that give you their VM Manager and some other handy tools that go hand in hand with managing a Virtual Server environment. Oracle wisely dispenses with all the developer tools and Gnome desktop and office productivity tools that bloat up the RHEL/CentOS distros. Good security practices recommend only running what you need to run on your server and nothing else! What follows below is how to get your first Virtualized server up and running by downloading just the one Oracle VM Xen Server CD. As far as I know, this is about as simple as it gets. The graphical environment is not used and none of the iSCSI or LVM (logical volume management) tools are employed. This is just a simple exercise that gets your hands dirty and playing with virtual technologies.

Step 1 - Install the Xen Server

Download the Oracle VM Server 32-bit Edition CD from Oracle and burn the ISO to CD. Then plop it into your CD drive on the machine you want and boot up. Follow the installation instructions on the screen (or the Online documentation). Once installed, boot up and log in as root. Note: At the time of this writing, we’re installing Oracle VM 2.1.2 You should now be able to issue:

1
2
3
# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   543     1     r-----   3036.7

And see an output similar to the above. Congratulations, you have a Xen Server up and running.

Step 2 - Initial File Preparation for the Guest

Initialize diskspace for your first Virtual Machine by performing the following commands (as root):

cd /OVS
mkdir -p local/numberone && cd local/numberone
mkdir numberone
cd numberone
dd if=/dev/zero of=numberone_hda1.img bs=1024k count=6000
dd if=/dev/zero of=numberone_hda2.img bs=1024k count=1500

This will set up a “local/numberone” folder on the Oracle Clustered File System partition, “/OVS” with first file being 6gb and 2nd file being 1.5gb. These will then become root partition and swap space of the guest VM, which we’re calling “numberone” here (you can name it any valid hostname you desire). Once these files are established, we create the filesystems for them with:

mkfs.ext3 numberone_hda1.img
mkswap numberone_hda2.img

Step 3 - Establish the Root partition of the Guest

Its time to push some files onto the root partition so that we have a valid system that can boot. Begin by issuing the following statements:

cd /OVS/local/numberone
mkdir hda1
mount -o loop numberone_hda1.img hda1
rsync  -avH /boot hda1
rsync  -avH /root hda1
rsync  -avH /dev hda1
rsync  -avH /var hda1
rsync  -avH /etc hda1
rsync  -avH /usr hda1
rsync  -avH /bin hda1
rsync  -avH /sbin hda1
rsync  -avH /lib hda1
rsync -avH /selinux hda1
mkdir hda1/{proc,sys,home,tmp}
chmod 777 hda1/tmp
unmount hda1
rmdir -rf hda1

Step 4 - Configure the Guest

You really don’t want the guest OS coming up with the same configuration as your Xen Host, so lets get into the guest’s /etc folder and do some tweaking: Edit /OVS/local/numberone/hda1/etc/fstab and set the root and swap partitions

1
2
3
4
5
6
/dev/hda1               /                       ext3    defaults        1 1
/dev/hda2               swap                    swap    defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
sysfs                   /sys                    sysfs   defaults        0 0

Edit /OVS/local/numberone/hda1/etc/sysconfig/network and change the hostname

1
2
3
4
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=numberone
GATEWAY=192.168.0.5

Note: Gateway is the firewall on your network providing Internet connectivity. Mine happens to be 192.168.0.5. Edit /OVS/local/numberone/hda1/etc/hosts and change the hostname

1
2
127.0.0.1               numberone localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6

Rename /OVS/local/numberone/hda1/etc/sysconfig/network-scripts/ifcfg-eth0 to /OVS/local/numberone/hda1/etc/sysconfig/network-scripts/ifcfg-eth0.bak so its not used when the guest first boots up. Similarly, rename /OVS/local/numberone/hda1/lib/tls to /OVS/local/numberone/hda1/lib/tls.disabled Then unmount the guest’s root partition:

cd /OVS/local/numberone
umount hda1

Create a domain configuration file for your new guest in /etc/xen/numberone on the Xen Server (not the guest’s root partition):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#  -*- mode: python; -*-
#============================================================================
# Python configuration setup for 'xm create'.
# This script sets the parameters used when a domain is created using 'xm create'.
# You use a separate script for each domain you want to create, or
# you can set the parameters for the domain on the xm command line.
#============================================================================

#----------------------------------------------------------------------------
kernel = "/boot/vmlinuz-2.6.18-8.1.15.1.16.el5xen"
ramdisk = "/boot/initrd-2.6.18-8.1.15.1.16.el5xen.img"
memory = 512
name = "numberone"

vif = [ 'bridge=xenbr0' ]

disk = ['file:/OVS/local/gox/gox_hda1.img,hda1,w',
        'file:/OVS/local/gox/gox_hda2.img,hda2,w' ]

root = "/dev/hda1 ro"

# Sets runlevel 4.
extra = "4"

The mac address was left out of the vif definition. We’ll boot up the guest to console and see what “reasonable value” the Xen host establishes for the server then come back and hard code the value so its not changing all the time.

Step 5 - Booting for the first time

To boot the guest, we’ll use the command-line tools, specifically “xm” to create (boot) the guest and shutdown rather than the Oracle VM Manager. So lets have a look:

xm create -c numberone

The “-c” flag boots you with console view of the guest. If all went well, you’ll see a typical Linux boot up process that leads all the way to a login prompt similar to this:

Oracle VM server release 2.1.2
Hypervisor running in UNDETERMINED bit mode (WARNING: XEND IS PROBABLY NOT RUNNING).

Network :
Management Interface  : 
If : eth1(Down)  Mac :  IP address : 

Configured Networks and Bridges :
If : eth0   Mac : 00:16:3E:22:F3:4F

CPU :
cpu family  : 15
model    : 31
model name  : AMD Athlon(tm) 64 Processor 3200+

numberone login: 

Copy that Mac address to clipboard (or write it down) because you’re going to need it for the next steps. Log in as root, same password as the Xen Host (since we copied everything off the host to begin with!). Edit /etc/sysconfig/network-scripts/ifcfg-eth0:

1
2
3
4
5
6
DEVICE=eth0
BOOTPROTO=static
HWADDR=00:16:3E:22:F3:4F
ONBOOT=yes
IPADDR=192.168.0.150
NETMASK=255.255.255.0

Once you’ve done this, you should be able to start your network and ping your gateway:

ifup eth0
ping 192.168.0.5
==>
PING 192.168.0.5 (192.168.0.5) 56(84) bytes of data.
64 bytes from 192.168.0.5: icmp_seq=1 ttl=64 time=3.33 ms
64 bytes from 192.168.0.5: icmp_seq=2 ttl=64 time=1.04 ms

If not, check for typos. Otherwise, you’re done with configuring the guest. Continue by logging out of your guest then break from console with CTRL-] (control-key plus left-bracket key). Edit /etc/xen/numberone and change the vif line to read:

1
vif = [ 'mac=00:16:3E:22:F3:4F, bridge=xenbr0' ]

This gels your mac address for the guest VM so the network will be reliably started on reboots for eth0 device. Now issue the following:

xm shutdown numberone
xm top 

And watch your guest disappear from the listing. Once its gone, issue the following to bring it back up:

xm create numberone

Finally, if everything’s gone well, you will be able to ssh to the guest VM with:

ssh 192.168.0.150 -l root

(Hey, I didn’t say it was uber secure or that we’re following best practices regarding root accounts!) If you can’t ssh into the guest from the xen server, you’ll have to get back into console and review logs and figure out what went wrong. You may need to fix typos in your configs or try another xen bridge such as xenbr1. If you are able to issue “ifup eth0” within the guest and can ping your firewall, its not network connectivity, but a typo in your /etc/sysconfig/* files somewhere. Probably the best place to double-check things is the mac address.

xm console numberone

One last tip. If you want to install packages that didn’t come on the Oracle VM install CD, you can add the CentOS5 repository to the VM guest and install various packages using yum thereafter. To do this, log into your Guest VM, and edit the following new file: /etc/yum.repos.d

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[CentOS5 base]
name=CentOS-5-Base
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=0
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

[CentOS5 updates]
name=CentOS-5-Updates
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=$basearch&repo=updates
gpgcheck=0
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

[CentOS5plus]
name=CentOS-5-Plus
mirrorlist=http://mirrorlist.centos.org/?release=5&arch=$basearch&repo=centosplus
gpgcheck=0
enabled=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

So there you have it. One of the simplest approaches to getting your first Xen guest Virtual Machine up and running and well on your way to experimenting with virtual technologies. _Special Note: Much credit must go to my colleague, Manyuan Jin, who spent many, many tireless hours researching early instability issues and treading through the jungle of open source solutions to assemble a working solution for our shop. _

comments powered by Disqus