I have been building a lot of Virtual Machines for a variety of reasons over the last 12 months, and up until now have been using external vendors, such as Digital Ocean and Host Virtual to provide the virtualisation and hosting services (both of these companies have been excellent).
I wanted to try bringing some of these services in-house and thought it would be a good opportunity to experiment with the available technology and learn something new.
The original plan was to build a platform similar to that of the above on-line providers. A web-based portal where I could log in, view and manage my existing Virtual Machines, and create new ones (from templates or ISOs) as required.
I looked in to VMWare, VirtualBox and XenServer for the Virtualisation technology. OpenFiler and FreeNAS for shared storage.
I have in no way completed this project, but thought I would start to document things in case I forget something or any of this might be useful to others. I am also by no means an expert in this area, and welcome any comments about how I should improve things.
- Existing Ubuntu 12.04 server providing DHCP & DNS (but this could come from your router etc)
- Mac Mini Workstation for management and installation tasks (including Windows 7 VM for XenCenter).
- 3 x HP Microserver G1610T Gen8 (Celeron, 16GB, 64GB SSD)
- 1 x HP Microserver Gen7 (Celeron, 4GB, 4 x 2TB)
- 1 x HP PS1810-8G Switch (no PSU!)
- 1 x TP-LINK TL-POE150S Power over Ethernet injector
- External USB DVD drive
- 32 GB USB Key
- XenServer 6.2
- Xen Orchestra (software for web-based administration)
The HP Gen 8’s will be the XenServer hosts. The Gen 7 will be a shared storage device. The HP switch connects everything up via GbE. The USB key will be used to host FreeNAS as I want to use all the internal drives for shared storage.
I am currently running everything (management and data) on a single subnet. This is mainly because I wanted to get everything working as quickly as possible. The next stage of this project is to separate things, since the Gen8’s have two NICs where as the Gen7 only has one.
Annoyingly, the HP switch did not come with a power supply, and even though the instructions state that it does (even in the ‘please check contents’ section) – contacting eBuyer and even HP resulted in them politely telling me that I needed to get power to the unit in another way. I ended up going for the POE injector since HP could not tell me which PSU to buy!
I also purchased the incorrect memory for the Gen8’s initially – since the machines seem to be very strict about what they will accept. I ended up going for the “CT4487631 16GB kit (8GBx2), 240-pin DIMM Upgrade for a HP Compaq Proliant MicroServer Gen8 System” from Crucial, at £127 + VAT.
I also wanted to run XenServer from the Gen8’s internal Micro SD card slot – but no matter what I tried, I could not get it to install successfully. After many, many lost hours I ended up getting some cheap SanDisk SSD drives from Novatech that made the process a whole lot easier. The added bonus here is that I can use the SSDs for some fast local VM storage if I need it.
These will be where the VMs do all their computing. XenServer is a breeze to install providing your hardware is all set up correctly. I decided on XenServer rather than Xen (hosted on Ubuntu etc) since I do not need to use these boxes for anything other than virtualisation.
- Download a copy of the latest free version of XenServer ISO.
- Burn the ISO to disk (or USB etc) and prepare the machine to boot from it.
- Power up the machine and follow the simple installation steps.
- Choose the local SSD drive for the destination media.
- Chose the local DVD for installation media.
- Chose not to install any additional ‘packs’.
- Provide a root password when prompted.
- Change the network settings to fixed (assign IP, gateway etc).
- Rename the hostname to something memorable (I used “xenserver-monroe”).
- Choose the timezone details.
- Enter an NTP server.
- Continue to install the software and when completed, remove the installation media and reboot.
- Repeat as required for each host.
This YouTube video might be useful for those installing XenServer for the first time.
This is currently a Windows only app, so I had to use my VM. Very easy to set up and use, and definitely worth the hassle of setting up Windows. You can even download a version of Windows 7 that works without registration for 30 days if you don’t already have a license.
- Download the XenCenter installer to the Windows OS (in my case, a VM on the Mac).
- Install using the defaults.
- Load the software up.
- Click the “Add Server” icon.
- Add the IP, username (root) and password.
- Choose to save and restore the connection state on start up.
- Create a master-password for all your XenServers (this allows you to access all the servers you connect with a single-sign-on).
- Repeat steps 1-6 for all XenServer hosts.
- Right-click one of the hosts.
- Choose “Add to Pool” -> “New Pool”.
- Enter a Pool name (I used “hollywood”).
- Right click the the other XenServer hosts and add to the same Pool.
After doing some light reading, I decided on FreeNAS as the shared storage platform. It is well documented, and free to use. It also supports iSCSI which seems to be the preferred method of connecting storage to XenServer.
- Disable the hardware RAID on the server. I want FreeNAS to handle all the disks itself.
- Download the latest IMG file for USB installation. This is not an installer, but rather a disk image that is a pre-installed version of FreeNAS. This means you put the USB in and it boots up a working version of FreeNAS.
- Copy the image to the USB stick. I used this guide.
- Install the USB in to a suitable slot (I used the internal), set any required BIOS settings to allow booting from USB, and boot up.
- Follow the simple on-screen instructions.
- When provided with the URL for web-based management, switch to your workstation, open your browser and navigate to that address.
- Log in (default credentials are “root” / “freenas”).
- Go to the Interfaces menu and Add new.
- Set up the interface with a fixed IP. After you change this, you will have to navigate to the new IP in your browser.
- Go to the Storage menu, open the Volumes option and open the ZFS volume manager.
- From here you can create a volume from the installed hard drives and choose from a variety of disk formats. I went with RAIDZ2 (after reading this) – but suggest you read in to what each of the different methods offer. From my 4 x 2TB disk, I now have a single 3.6TB volume called “nas01”.
- Next, open the Services menu and find “iSCSI”.
- Open “Extents” and “Add Extent”. I named mine “extent01”.
- Open “Initiators” and “Add Initiator”. I named mine “init01”.
- Open “Portals” and “Add Portal”. I named mine “portal01”.
- Open “Targets” and “Add Target”. I named mine “target01”. Select your Portal and Initiator and select “none” for both the “Auth method” and “Authorisation Group Number”.
- Open “Targets / Extents” and “Add Target / Extent”. Select your Target and Extent.
- Go to the main Services page and enable the iSCSI service.
Connecting FreeNAS to XenCenter
Now we have the XenServers and the FreeNAS box running, we need to tell the XenServers that the Pool can use the FreeNAS box as a shared storage device. This was actually very painless.
- Choose “Add SR” from the “Storage” menu in XenCenter.
- Select “Software iSCSI”.
- Give the repository a suitable name (eg. “primary-nas”).
- Enter the IP of the FreeNAS box and click the “Discover IQNs” button.
- Select (if not already selected) your IQN.
- Click the “Discover LUNs” button.
- Select (if not already selected) your LUN.
- Click Finish.
You may want to add some more storage for ISO’s and images – but this is also simple to do with FreeNAS. I ended up creating a CIFS share since it is then easy to access from Windows boxes too.
You can download a virtual machine of the Xen Orchestra software from their website. This is a pre-built Debian-based VM with the Xen Orchestra software installed. It can be added to the Pool by simply using the Import menu item (for some reason, this took several hours on my set up).
At this stage, the Xen Orchestra software seems like it needs quite a lot of work to be able to use it in production. It is very easy to connect to the existing Pool (you enter the web-interface, add the IP, username and password of one of the VM’s are you’re done) and you can see that the project has great potential. Its unfortunate that at this early stage, many of the useful features seem to just be a button that leads to a “to do” page. It definitely looks very promising and I hope to be able to contribute to the project.
I’m also going to look in to the XenServer API – since it seems quite straight forward and hope to be able to take a look at both OpenStack and CloudStack soon.