Use the source, Luke – what’s coming up in Ansible v2

If you are one of the almost 10k developers who starred ansible github repo, you might have noticed that at some point in time, a mysterious v2 directory appeared in the sources – containing alternative implementations of ansible and ansible-playbook. Is it the beginning of some new product, a complete rewrite, refactoring, or just some experimental branch accidentaly merged into devel? In this article, I’ll4 go through both the source code of ansible, and roam the internet, to put it all together.

What is V2?

Ansible grew very rapidly over the last few years – and such a quick pace of development comes at a cost. Technical debt accumulated, and product devs felt it was the moment to start thinking more seriously about long-term maintainability of the software. So the initial rationale was just to pay interest on accumulated debt, thus enabling sustainable pace in the feature. The aim is to preserve 100% playbook compatibility – modules also do not need any changes – in fact, ansible already split module development into separate repositories, and v2 uses the same git submodule branches as v1 does. Some of the internal API’s might change though – so if you used a custom-made plugin, you might need to change. Generally, this is certainly not a python 2 to 3 conversion, and switching to new version should be seamless for most parties, sans-bugs, which would still be possible if the refactorings weren’t there, wouldn’t they?

Execution strategies

If you ever thought about how a tool like ansible might work internally, you might have come to conclusion that conceptually, it’s not that different than
gnu make – it reads a file in some DSL format, transforms it into a graph where nodes are the commands to execute (e.g.: compile file.c in make, run apt-get update in ansible), and edges reflect dependencies. When you have that graph, you just need to process it in a way that no command is executed before it’s dependencies and you’re good. And really, the main difference between most of the build/automation tools out there is the DSL used to specify the graph. Ansible is a bit different though. First of all – the graph is very simple, it’s just a sequence of tasks, each of them having associated hosts. It’s so simple it does not have an explicit representation in code.

The simplicity is also probably the root of ansible’s success – it just turned out, that when it comes to infrastructure automation, complex dependency relationships are not that much necessary – so instead of writing complex processing engine, developers come up with the simplest possible solution, and focused on modules and convenience of use. That being said, there’s nothing preventing ansible from changing the way tasks are processed, or even introducing a full graph model. The latter is just my speculation, the former
is an actual feature of v2 – it’s named strategies. The best way to explain it is by example, here’s a playbook:

- hosts: all
  gather_facts: no
  strategy: free
  tasks:
  - pause: seconds={{ 10 |random}}
  - debug: msg="msg_1"
  - pause: seconds={{ 10 |random}}
  - debug: msg="msg_2"
  - pause: seconds={{ 10 |random}}
  - debug: msg="msg_3"

You might notice the strategy: free line – this one chooses the execution strategy plugin for the playbook. The other (and the default) strategy is linear, which makes ansible behave the same way it did before, so the output from linear strategy could look like this (I’ve removed the pause command output and collapsed some lines for brevity):

TASK [debug msg=msg_1] **********************************************************  
ok: [host3] => { "msg": "msg_1", "changed": false}  
ok: [host4] => { "msg": "msg_1", "changed": false}  
ok: [host2] => { "msg": "msg_1", "changed": false}  
ok: [host1] => { "msg": "msg_1", "changed": false}

TASK [debug msg=msg_2] **********************************************************  
ok: [host4] => {"msg": "msg_2", "changed": false}  
ok: [host1] => {"msg": "msg_2", "changed": false}  
ok: [host2] => {"msg": "msg_2", "changed": false}  
ok: [host3] => {"msg": "msg_2", "changed": false}

TASK [debug msg=msg_3] **********************************************************  
ok: [host1] => {"msg": "msg_3", "changed": false}  
ok: [host2] => {"msg": "msg_3", "changed": false}  
ok: [host3] => {"msg": "msg_3", "changed": false}  
ok: [host4] => {"msg": "msg_3", "changed": false}

The order of hosts is always random, but none of the “msg_3” task will start until, all msg_2 tasks are completed. With free strategy this is no longer the case:

PLAY [] ******************************************************  
ok: [host3] => {"msg": "msg_1", "changed": false}  
ok: [host4] => {"msg": "msg_1", "changed": false}  
ok: [host2] => {"msg": "msg_1", "changed": false}  
ok: [host4] => {"msg": "msg_2", "changed": false}  
ok: [host2] => {"msg": "msg_2", "changed": false}  
ok: [host4] => {"msg": "msg_3", "changed": false}  
ok: [host1] => {"msg": "msg_1", "changed": false}  
ok: [host2] => {"msg": "msg_3", "changed": false}  
ok: [host3] => {"msg": "msg_2", "changed": false}  
ok: [host3] => {"msg": "msg_3", "changed": false}  
ok: [host1] => {"msg": "msg_2", "changed": false}  
ok: [host1] => {"msg": "msg_3", "changed": false}

This time, the msg_1, msg_2 and msg_3 are not grouped together – that’s because the free strategy treats every host independently – so for example the msg_2 task on host2 has no dependency on msg_1 task of each host, and can execute as fast as possible. This might improve the execution time of some playbooks.

You can also write your own strategy plugin, although at this point in time, the API is far from being stable and documented (I actually had to fix the free plugin just to show you the results) – but who knows, maybe at the end, you will get an API with easy access to playbook data, allowing you to construct your own graphs, and then just send tasks to execution in any way you can conceive.

Blocks

Another new feature is block – try/except/finally for ansible. It looks like this:

- hosts: localhost
  connection: local
  gather_facts: no
  tasks:
  - block:
      - command: /bin/false
      - debug: msg="you shouldn't see me"
    rescue:
      - debug: msg="this is the rescue"
      - command: /bin/false
      - debug: msg="you shouldn't see this either"
    always:
      - debug: msg="this is the always block, it will always be seen"

Playbook like this will produce following output:

PLAY [] ******************************************************

TASK [command] ******************************************************************  
fatal: [localhost]: FAILED! => ...

NO MORE HOSTS LEFT **************************************************************

CLEANUP TASK [debug msg=this is the rescue] *************************************  
ok: [localhost] => {  
    "msg": "this is the rescue", 
    "changed": false
}

CLEANUP TASK [command] **********************************************************  
fatal: [localhost]: FAILED! => ...

CLEANUP TASK [debug msg=this is the always block, it will always be seen] *******  
ok: [localhost] => {  
    "msg": "this is the always block, it will always be seen", 
    "changed": false
}

It’s important to note that, when the block fails – you get to execute the rescue and always segments, but even if those execute correctly, the playbook won’t continue, so in terms of try/except/finally, the catch still re-throws the exception.

Include + with

This is a comeback of a feature that was deprecated in Ansible 1.5. Here’s an example:

main.yml:

- hosts: localhost
  connection: local
  gather_facts: no
  tasks:
    - include: foo.yml some_var={{ item }}
      with_items:
       - a
       - b
       - c

foo.yml:

 - debug: msg={{some_var}}

In v1, you’ll only get an depreciation error message, whereas in v2 you’ll get the expected output – I’ll leave it as an exercise to you what that output should be.

Improved error detection

If you make a mistake in module name, say:

- hosts: all
  gather_facts: no
  tasks:
  - debug: msg="hi"
  - not_a_syntax_error_just_invalid_module: msg="error"
  - debug: msg="bye"

In v1, you’ll get a rather short error message:

ERROR: not_a_syntax_error_just_invalid_module is not a legal parameter in an Ansible task or handler

While in v2, you’ll get a bit more of context, and location of your error.

ERROR! no action detected in task

The error appears to have been in '... .yml': line 5, column 44, but may  
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  - debug: msg="hi"
  - not_a_syntax_error_just_invalid_module: msg="error"
                                           ^ here

It might be not the most important feature ever, but such small conveniences add up, and make work easier for everyone. Certainly easier to spot the cause, than by core-dump analysis :).

Python 3 support?

There is some work started on making ansible v2 engine (as opposed to modules, which are independent from the engine) work on py3, with six library helping with maintaining compatibility between versions – still, trying to run any of the v2 samples on python 3.4 ended up with syntax errors (on except blocks, which use old syntax to specify variable into which exception is being caught) – 2to3 would probably fix those, but as this was a clear sign that code is not regularly run on python 3, I gave up on further exploration.

V2 release date.

Initially, v2 was planned to replace the current execution engine in 1.9 release, but now, it seems like it will be the last major release using the previous one – so most likely the switch will occur on 2.0. 1.9 will happen somewhere in the middle of March 2015, and looking at the past releases timeline, 2.0 should follow in about 2 to 3 months, so my guess is we are looking at June 2015.

Since the sources are there, you can run the v2 by yourself. Be wary though – you’ll probably end up troubleshooting some errors by yourself – I personally had trouble with:

• running every remote command – turned out v2 logs in with root by default, and most boxes do not allow that
• specifying local connection in inventory did not work
• playbooks exploding on not being able to find any filter module (which turned out to be a quirk in filter plugin default configuration) even though no filters were actually being used
• Unable to find lookup plugins and thus informing about error in playbooks – had to copy plugins from v1 and fix them, so they use api from v2 (functions moved into different modules mostly)

Conclusion

To sum up, I must say, that at it’s current stage, v2 is still very much a work in progress, running it locally requires some debugging to figure out all the quirks, but the overall direction is good, the tool remains simple at it’s core (which imo is the foundation of it’s widespread success) and possibly the new plugin architecture might open a very nice area for extensions. Personally I hope for some clearer graph API, which could even enable alternative DSL’s for declaring playbooks – that might be a topic for v3 though.

Hope you enjoyed the read, please let us know what you think.