NOTE: a rewritten version is available on the SemaphoreCi community page. It contains some more details on how to setup your environment.
In my previous post, I explained how you could easily test your Ansible roles by using Travis CI. One of the down sides of Travis for me is that it’s still running on Ubuntu 12.04. Luckily, there is an alternative: meet Semaphore
Some plus sides of Semaphore are:
- it runs Ubuntu 14.04
- run your tests with 2 processors
- have unlimited private repositories
- access the build environments through SSH
Testing with Semaphore is pretty much the same as with Travis CI.
What to test?
Just like with Travis we’ll be focussing on these 4 points:
- Make sure the role syntax is correct.
- Does your role run through all tasks without failing
- Is your role build in an idempotent way? (a second run cannot create new changes)
- Did the role everything you wanted
But again, point 4 is omissible as I’ve told in my post about Testing with Travis CI.
Test setup
The test setup is completely the same as on Travis. This makes it ideal so we don’t have to repeat any code that is basically there for the same purpose.
So we have our inventory and test.yml file present in the test directory of our role:
# Directory structure:
ansible-role-mysql/
tests/
test.yml
inventory
Inside your inventory file you have your localhost defined. We will eventually use the command –connection=local option to tell Ansible to run the test playbook on the local machine.
Inside the test.yml playbook, we define our play:
- hosts: all
roles:
- { role: ../../ansible-role-mysql, sudo: Yes }
Configuring the test tasks
While Travis CI uses a dot file to configure your tests, you’ll have to define your build settings inside the Semaphore interface. You can do this by going to the Project Settings and then select Build settings.
Inside your build settings, you can start by adding the build command that you need. For each build setting, you can select when a certain command needs to be executed. Semaphore gives you 4 options:
- During setup
- After a thread is finished (post-thread)
- In thread 1
- In thread 2
Thread 1 and 2 make it easy for you to run certain tests in parallel, which can speed up things.
Before you start running tests, you’ll need to setup your build environment first. Use these 3 build commands to setup your environment:
sudo apt-get update -qq
sudo apt-get install -qq python-apt pip install ansible==1.9.1
Here you say that apt-get needs to update its package list, install the python-apt package through apt and then install Ansible version 1.9.1 through pip. You can omit the Ansible version is you like.
The guys at Semaphore update their platform pretty regularly. During an update, I encountered issues install Ansible through pip. This caused me to change my build settings for setup as following:
sudo apt-get update -qq
sudo apt-get install -qq python-apt python-pycurl libyaml-dev libpython2.7-dev sudo easy_install pip sudo /usr/local/bin/pip install ansible==1.9.1
These build settings are almost identical to the ones above. The only difference here is that instead of installing pip through apt-get, I installed it using easy_install. Seemed that the python-pip package got broken, so I needed an alternative.
1. Role syntax
For testing your role syntax, just add the same command as we used for Travis.
ansible-playbook -i tests/inventory tests/test.yml --syntax-check
Assign this command to Thread #1.
2. First role run
Now it’s time to run our role for the first time and see if all tasks get executed properly. For this, we can again use the same command as we did before with Travis:
ansible-playbook -i tests/inventory tests/test.yml --connection=local --sudo
Ansible returns a non-zero exit if the playbook fails, so Semaphore knows whether the command succeeds or not. Assign this build command to Thread #2. By assigning this command to a new thread, we can run the role syntax test and first run in parallel.
3. Role idempotency
So after our first run test finishes successfully, it is now time to test the role idempotency again. And yes, for this we can indeed again use the same command as we did on Travis.
ansible-playbook -i tests/inventory tests/test.yml --connection=local --sudo | tee /tmp/output.txt; grep -q 'changed=0.*failed=0' /tmp/output.txt && (echo 'Idempotence test: pass' && exit 0) || (echo 'Idempotence test: fail' && exit 1)
Assign this test to Thread #2. You don’t want to assign this test on Thread #1, because otherwise your test will fail, due to the fact your first run happens in Thread #2.
4. Role result
As stated before, testing your role result is not something that I’m too keen on, so most of the times, I skip this part. I prefer making sure most elements are present in my roles, to the first run and idempotency test picks those up.
Caveats
Well, Semaphore is pretty neat platform if you ask me, and the fact they run Ubuntu 14.04 is just great. There are some things to keep in mind:
- It runs only Ubuntu 14.04. No other OS’s, distributions and versions are supported.
- The Semaphore builds also come with some pre-installed software packages. So make sure to purge those if you are running for a package that is already pre-installed.
- For some roles it might be hard to test them against idempotency. Take for instance NewRelic. In my NewRelic role, I specifically tell the service to start up. The problem is, if you don’t provide an official license key, the service will automatically stop. Meaning that during your idempotency test, if will again try and start the service, which will make the idempotency test fail.
