Mind Passthrough

Nicolas Ochem's mumblings on virtualization and network policy

Archipel + Vagrant + Puppet

Archipel is a lightweight cloud management platform built on top of libvirt. While it does not match the feature set of more popular orchestrators, Archipel’s Cappuccino-based UI is way better than every other cloud management platform out there.

I spent some time last year trying to maintain an environment where I could hack on ArchipelAgent, Archipel’s Python backend, and test it in a box.

To test Archipel, I need one virtual machine acting as ejabberd server and central agent, and one or more nested hypervisors. I wrote some scripts to manage the lifecycle of this setup. The features were :

  • ability to start and stop all Archipel daemons
  • deploy the Archipel working tree in all VMs by NFS
  • send commands to the VMs shell by script

I ended up writing mostly unmaintainable code.

Then a few months ago I stumbled upon the Technical Blog of James. James has made puppet-gluster-vagrant, an awesome self-contained hacking environment for glusterfs which turned out very handy to test a gluster setup I am using at work. The interesting part is that it is running in Vagrant with Vagrant-Libvirt plugin. Since Archipel is a pure Libvirt shop, it sounds like a plan, now, does it not ?

Setup

Vagrant is a deployment and prototyping environment perfectly adapted for the use case described above, but it was designed around Virtualbox. Only a few months ago, feature parity with Libvirt was achieved. Still, based on James’ blog post, Libvirt felt like a second-class citizen, at least on Fedora 20. So I decided to give it a try on Ubuntu 14.04 LTS.

First things first, I installed vagrant and the necessary plugin :

1
2
3
4
5

# This will install Vagrant 1.4.1.
# Good, because vagrant-libvirt is currently broken with Vagrant 1.5.
apt-get install vagrant

vagrant plugin install vagrant-libvirt

I added this line to my .bashrc to make Vagrant use libvirt by default :

1

export VAGRANT_DEFAULT_PROVIDER=libvirt # libvirt as default vagrant provider

I then downloaded a centos6.5 box in libvirt format :

1

vagrant box add centos-6 https://download.gluster.org/pub/gluster/purpleidea/vagrant/centos-6.box --provider=libvirt

Then I created a Vagrantfile using vagrant init and created 3 VMs :

1
2
3
4
5
6
7
8
9
10
11

# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = "centos-6"
  config.vm.define :central_server, :primary => true
  config.vm.define :agent_1
  config.vm.define :agent_2
end

Then issuing vagrant up brought my 3 VMs up. I was able to login to them using i.e. vagrant ssh central_server. vagrant destroy sent them into oblivion.

Nested virtualization works out-of-the-box on Ubuntu, provided that you set up the correct options in your VM XML definition. Vagrant-libvirt has a configuration option that will do that for you :

1
2
3
4
5
6
7

Vagrant.configure("2") do |config|
  config.vm.define "agent_1" do |agent|
    agent.vm.provider :libvirt do |domain|
      domain.nested = true
    end
  end
end

The rest of the Vagrantfile is fairly standard Ruby code. The default comments make it really easy to incrementally create the setup you want. Check out the final Vagrantfile.

There were a few hurdles before I reached that state though :

  • for some reason my Ubuntu laptop will not allow NFS exports anything under my home directory. Maybe because it is encrypted ? Anyways, I ended up working as root and using /root/workspace/archipel-puppet-vagrant as working directory. I had to make it world-readable.
  • Ubuntu’s firewall, ufw, does not let me export NFS folders. I had to deactivate it completely by issuing sudo ufw disable. Just shutting it down does not do the trick as it keeps coming back up.

It looks like libvirt is still not a first-class citizen in the Vagrant world, but hopefully the situation will improve.

You can check more advice on how to run vagrant with libvirt on this blog post.

Provisioning

I have 3 empty boxes, and I want to turn them into a running Archipel setup, with one central server and 2 agents running VMs. How do I do that in 2014 ? I use a provisioning solution, like Puppet.

That should be easy because Vagrant supports Puppet out of the box. The vagrant provision command will execute puppet manifests on all nodes. The manifests will also be executed by default if they are present when you do vagrant up.

James’s puppet-gluster-vagrant setup has a standalone puppet server box defined in his Vagrantfile. All the other VMs are using it as puppetmaster. This mimics a production environment in the best possible way, but it is overkill for me. I am only going to write Puppet manifests on my workspace and load them in the VMs using headless Puppet.

I followed the installation instructions for Archipel Agent, Archipel Server and Archipel Central Agent. I turned these instructions into a Puppet module for Archipel. The Vagrantfile is in a subdirectory of this module.

A few caveats apply :

  • the setup is very slow over ssh because of DNS timeouts. Bringing up the setup from scratch takes more time than it should. I have found a lot of suggestions but none seemed to help.
  • there is no DNS support so I had to add commands in the Vagrantfile to set the hostnames of every node manually in /etc/hosts of every node. There is a Vagrant plugin named Landrush that does just that for Virtualbox. Hopefully libvirt support will be coming soon.

Hacking

Archipel repository is checked out as a submodule inside puppet-archipel. This way, it gets NFS-mounted to every node.

ArchipelAgent already has a “developer mode” built in. Running buildAgent -d will set up redirects from the python library folder to the development environment. So I added this step in the Puppet manifest.

It means that editing the source code in my laptop deploys it instantly to all nodes. Testing changes is trivial.

Probably the best part of this setup is the portability of the configuration. In theory, anyone can check out the code, perform vagrant up and deploy the same environment in minutes. The barrier of entry for a potential developer is pushed to the lowest limit, no matter how distributed the setup is. That is a huge asset for cloud management systems.

To start using archipel-vagrant-puppet, check out the README. Enjoy !