Ansible filters, learn how to control your repository

guide with examples to build ansible filters

In this article we bring you an introduction to the Ansible tool and Ansible filters.

Ansible is a configuration and orchestration management tool developed in Python bought some time ago by the company RedHat.

This is an introductory article, but you’re supposed to know the concepts of playbook and role.

If you don’t, in this article I leave you with a section of links to learn more, or if you prefer you can contact the Dare Planet Technology’s team and we will try to help you.

Table of Contents

The goal of working with Ansible filters

Let’s see how Ansible filters can help us get our ansible repository under control, avoiding unnecessary repetitions.

For this, let’s put an example of variable organization in our ansible repository.

Let us not forget that our automation must be as descriptive and clear as possible, remembering that ‘with great power comes great responsibility’.

Work methodology with Ansible

Abraham Lincoln once said:
“If I had eight hours to cut down a tree, I’d spend six hours sharpening the axe.”

Let’s follow that philosophy, first we define our final state, and then we’ll take action. In that way, if things get complicated, we’ll always be clear about what our goals are.

To make the process more understandable, which is really the interesting part, we are going to do little iterations (baby steps). Shall we start?

Starting point

When we develop any role, we have to do it in the most decoupled way possible from future integration. In this way, our role can evolve independently.

Let’s give an example, let’s start with a role of a fairly respected person within the ansible community, his name is Jeff Geerling. The role in particular is his docker role.

Among its variables, it asks for a list of usernames that will be added within the docker group so that they can use docker:

				
					docker_users:
  - user1
  - user2

In the context of the role and its name space, the variable name is perfect.

Also, let’s suppose that we have another role with the name ‘sudo‘ that and offer us two other variables where we can put our user that we want to use sudo (with or without password):

				
					sudo_with_password_users:
  - user1
  - user2
sudo_without_password_users:
  - user1
  - user2

Of course, we have a role or playbook that the users of our environments create for us, we can have something like:

				
					common_users: 
  - name: user1 
    state: present 
    group: group1  

  - name: user2 

  - name: user3 
    group: group2

Finally, we have the role of our application that is going to retire us from success, and you need to know the user that will be used to launch the application.

Watch out, it’s not a list, it’s a text with the user’s name, something like:

				
					app_username: user3

Application in our environment

With all this in the abstract, we want to apply the following in our case:

Users to be defined:

  • john
  • peter
  • anthony
  • margaret
  • july
  • mary

Users who can use docker are:

  • margaret
  • july

Users who can use ‘sudo‘ without a password are:

  • john
  • peter

Users who can use ‘sudo‘ with a password are:

  • anthony

The user of our application is mary, and there can only be one.

Software required to continue

If you want to follow the article and do it little by little you will need Ansible and jmespath.

The installation can be done for example by pip:

				
					pip install ansible jmespath

JMESPath is a query language for JSON that allows us to use the JSON filter within ansible. It’s actually quite powerful.

Developing the exercise with Ansible

Are you all ready? We’ll proceed to develop the Ansible filters.

Test definition in Ansible

Before doing anything, and as we talked about at the beginning we are going to define a simple playbook that contains our tests, and that of course will fail in a resounding way.

But what I want is to have in a descriptive and primitive way the final result, the result is in 00_playbook.yml

				
					---
- hosts: localhost
  connection: local
  gather_facts: no
  vars:
    dockers_users: []
    sudo_with_password_users: []
    sudo_without_password_users: []
    common_users: []
    app_username: ''

  tasks:
    - name: Checking common_users
      assert:
        that:
          - common_users is defined
          - common_users | length == 6
        fail_msg: Not all the users are included
      ignore_errors: yes

    - name: Checking docker users
      assert:
        that: 
          - docker_users is defined
          - docker_users | length == 2
          - '"margaret" in docker_users'
          - '"july" in docker_users'
        fail_msg: Margaret and July only should be in docker_users
      ignore_errors: yes

    - name: Checking sudo user without password
      assert:
        that: 
          - sudo_without_password_users is defined
          - sudo_without_password_users | length == 2
          - '"john" in sudo_without_password_users'
          - '"peter" in sudo_without_password_users'
        fail_msg: John and Peter only should be in sudo_without_password_users
      ignore_errors: yes

    - name: Checking sudo user with password
      assert:
        that: 
          - sudo_with_password_users is defined
          - sudo_with_password_users | length == 1
          - '"anthony" in sudo_with_password_users'
        fail_msg: Anthony only should be in sudo_with_password_users
      ignore_errors: yes

    - name: Checking that Mary is the app user in the hosts
      assert:
        that:
          - '"mary" == app_username'
        fail_msg: Mary should be the user for the app
      ignore_errors: yes

If we run it, obviously all the tests will fail:

				
					$ ansible-playbook 00_playbook.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'

PLAY [localhost] ****************************************************************************************

TASK [Checking common_users] ****************************************************************************
fatal: [localhost]: FAILED! => {
    "assertion": "common_users | length == 6",
    "changed": false,
    "evaluated_to": false,
    "msg": "Not all the users are included"
}
...ignoring

TASK [Checking docker users] ****************************************************************************
fatal: [localhost]: FAILED! => {
    "assertion": "docker_users is defined",
    "changed": false,
    "evaluated_to": false,
    "msg": "Margaret and July only should be in docker_users"
}
...ignoring

TASK [Checking sudo user without password] **************************************************************
fatal: [localhost]: FAILED! => {
    "assertion": "sudo_without_password_users | length == 2",
    "changed": false,
    "evaluated_to": false,
    "msg": "John and Peter only should be in sudo_without_password_users"
}
...ignoring

TASK [Checking sudo user with password] *****************************************************************
fatal: [localhost]: FAILED! => {
    "assertion": "sudo_with_password_users | length == 1",
    "changed": false,
    "evaluated_to": false,
    "msg": "Anthony only should be in sudo_with_password_users"
}
...ignoring

TASK [Checking that Mary is the app user in the hosts] **************************************************
fatal: [localhost]: FAILED! => {
    "assertion": "\"mary\" == app_username",
    "changed": false,
    "evaluated_to": false,
    "msg": "Mary should be the user for the app"
}
...ignoring

PLAY RECAP **********************************************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=5

As much as it seems like a waste of time, we’re already pretty advanced.

First iteration (without Ansible filters)

Now we’re going to make it work only in a very rough and primitive way, it works but it won’t be our final version.

This version is exactly the same as the previous one, but we’ve just filled in the variables, now it look like this:

				
					vars:
   common_users:
     - name: margaret
     - name: july
     - name: john
     - name: peter
     - name: anthony
     - name: mary
   docker_users:
     - margaret
     - july
   sudo_without_password_users:
     - john
     - peter
   sudo_with_password_users:
     - anthony
   app_username: mary

The file for this step is 01_playbook.yml

Conclusions of this step

There is a lot of repetition of names, and in the future it is not very maintainable and prone to errors.

I could for example write margaret in common_users and margarit in docker_users, starting to have those errors that we don’t like so much.

Also, there are times when they are obvious and others when they are not.

Complete execution of this step

				
					ansible-playbook 01_playbook.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'

PLAY [localhost] ****************************************************************************************

TASK [Checking common_users] ****************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking docker users] ****************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking sudo user without password] **************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking sudo user with password] *****************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking that Mary is the app user in the hosts] **************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY RECAP **********************************************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Second iteration

Now that we have it working I wonder if all this part that refers to users, I could not in a single site and then go exploit that information.

Because I want to take small steps I will go from the simplest to the most complicated.

In this case, the most affordable way is to create a users structure by copying me as a user and then referencing it.

Although it sounds a bit weird, you’ll see that it’s very easy.

				
					users:
  - name: margaret
  - name: july
  - name: john
  - name: peter
  - name: anthony
  - name: mary

common_users: "{{ users }}"

As always if you want to see the whole file you have it in 02_playbook.yml.

Conclusions of this step

Although this small change seems to be nothing, we have already created our structure that will contain the rest. We’re on the right track.

Complete execution of this step

				
					$ ansible-playbook 02_playbook.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'

PLAY [localhost] ****************************************************************************************

TASK [Checking common_users] ****************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking docker users] ****************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking sudo user without password] **************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking sudo user with password] *****************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking that Mary is the app user in the hosts] **************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY RECAP **********************************************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Third iteration.

This is really where we start using Ansible filters, we’re going to try to extract the users from our user structure ‘users’, for that we’re going to add the following logic.

I’ll add an optional field, which will be with a ‘docker’ key and a boolean value.

If you do not put the field, it is understood that you will not be a user who can run the docker.

That is, we can have our share of users like this:

				
					users:
  - name: margaret
    docker: true
  - name: july
    docker: true
  - name: john
    docker: false
  - name: peter
  - name: anthony
  - name: mary

This step we are going to do with Ansible filters, and we can do it in two different ways with and without jmespath (json_query) that allows us more complex queries.

Ansible filter without JMESPATH

				
					docker_users:  "{{ users |
                   selectattr('docker', 'defined') |
                   selectattr('docker', 'equalto', True) |
                   map(attribute='name') |
                   list }}"

The docker_users Ansible filters basically does the following:

It collects the list of users, check each element that has the docker key defined, and then check that it is equal to true.

After that, it only keeps the value of the ‘name’ key for each element and returns a list.

Ansible filter with JMESPATH

				
					jmespath_docker_users: "{{ users | json_query('[?docker].name') }}"

TEST adaptation

In this case, we see how it has become clearer with jmespath, but the important thing is that the result is the same.

As you can see what I extract with jmespath I put the prefix jmespath, so now I must modify the tests to check both variables.

The test would look like this:

				
					- name: Checking docker users
  assert:
    that: 
      - docker_users is defined
      - docker_users | length == 2
      - '"margaret" in docker_users'
      - '"july" in docker_users'
      - docker_users == jmespath_docker_users
    fail_msg: Margaret and July only should be in docker_users

In this way I check that both contain the same thing in the last statement.

The entire file of this iteration is 03_playbook.yml.

Conclusions of this step

It looks like this is starting to take shape, and we’re getting things organized.

Complete execution of this step

				
					$ ansible-playbook 04_playbook.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'

PLAY [localhost] ****************************************************************************************

TASK [Checking common_users] ****************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking docker users] ****************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking sudo user without password] **************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking sudo user with password] *****************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking that Mary is the app user in the hosts] **************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY RECAP **********************************************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Fourth iteration.

Here we’re going to run ansible’s debug module for a moment, because I think we’re putting data that doesn’t interest us or wasn’t the initials in our common_users variables.

What we would have to execute would be something like this in the playbook (inside the tasks section):

				
					- name: Show common_users var
  debug:
    msg: "common_users: {{ common_users }}"

We run our playbook and see the following:

				
					TASK [Show common_users var] ****************************************************************************
TASK [Show common_users var] ****************************************************************************
ok: [localhost] => {
    "msg": "common_users: [{'name': 'margaret', 'docker': True}, {'name': 'july', 'docker': True}, {'name': 'john', 'docker': False}, {'name': 'peter'}, {'name': 'anthony'}, {'name': 'mary'}]"
}
....

We’re passing all the users fields, when in our example, we only want to pass ‘name’, it doesn’t matter in your case, but we’re going to correct ‘common_users‘ so that it only shows the ‘name‘ attribute as it was originally.

Ansible filter without JMESPATH

				
					common_users: "{{ users | map(attribute='name') | list }}"

Ansible filter with JMESPATH

As a reference we will put the Ansible filter to do the same with jmespath.

				
					jmespath_common_users: "{{ users | json_query('[].name') }}"

Checks

We run again and see that we have our problem fixed:

				
					TASK [Show common_users var] ****************************************************************************
ok: [localhost] => {
    "msg": "common_users: ['margaret', 'july', 'john', 'peter', 'anthony', 'mary']"
}

I will extend the tests to check that common_users and jmespath_common_users.

				
					- name: Checking common_users
  assert:
    that:
      - common_users is defined
      - common_users | length == 6
      - common_users == jmespath_common_users
    fail_msg: Not all the users are included

What is not done, and I leave it as an exercise of the reader, is to make a test that checks that “users” only contains the keys that should have.

The file for this step is 04_playbook.yml.

Conclusion of this step

Although the tests help us, we must be careful, because we may not have all the cases covered.

Fifth iteration

We have learned how to make an Ansible filter for boolean variables, and we have noticed some fault or feature, which we have adjusted on the fly.

On this occasion, we are going to approach the sudo issue, in this case, we are going to assume the following logic.

We create a key in our user structure called ‘sudo’, which can have the following values ‘with_password’, ‘without_password’. If it contains something else or doesn’t have the key we’ll assume that you won’t be using sudo.

Our users variable could look like this:

				
					vars:
  users:
    - name: margaret
      docker: true
    - name: july
      docker: true
    - name: john
      docker: false
      sudo: without_password
    - name: peter
      sudo: without_password
    - name: anthony
      sudo: with_password
    - name: mary

As in the third iteration, we’re going to do it with and without jmespath.

Ansible filter without JMESPATH

Let’s see how the version without jmespath would look, for the two groups we have to create:

				
					sudo_without_password_users: "{{ users |
                                 selectattr('sudo', 'defined') |
                                 selectattr('sudo', 'equalto', 'without_password') |
                                 map(attribute='name') |
                                 list }}"
sudo_with_password_users:    "{{ users |
                                 selectattr('sudo', 'defined') |
                                 selectattr('sudo', 'equalto', 'with_password') |
                                 map(attribute='name') |
                                 list }}"

Ansible filter with JMESPATH

Now let’s look at the example with jmespath:

				
					jmespath_sudo_without_password_users: "{{ users |
                                          json_query(\"[?sudo=='without_password'].name\") }}"

jmespath_sudo_with_password_users:    "{{ users |
                                          json_query(\"[?sudo=='with_password'].name\") }}"

TEST adaptation

Now all that remains is to modify the test a little, so that it checks that the two variabels contain the same thing:

				
					- name: Checking sudo user without password
  assert:
    that: 
      - sudo_without_password_users is defined
      - sudo_without_password_users | length == 2
      - '"john" in sudo_without_password_users'
      - '"peter" in sudo_without_password_users'
      - sudo_without_password_users == jmespath_sudo_without_password_users
    fail_msg: John and Peter only should be in sudo_without_password_users

- name: Checking sudo user with password
  assert:
    that: 
      - sudo_with_password_users is defined
      - sudo_with_password_users | length == 1
      - '"anthony" in sudo_with_password_users'
      - sudo_with_password_users == jmespath_sudo_with_password_users
    fail_msg: Anthony only should be in sudo_with_password_users

The file for this step is available at 05_playbook.yml.

Conclusion of this step

This is getting more organized, and we hardly have to repeat anything. Cheer up, we’re almost done.

Full execution of this step

				
					ansible-playbook 05_playbook.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'

PLAY [localhost] ****************************************************************************************

TASK [Show common_users var] ****************************************************************************
ok: [localhost] => {
    "msg": "common_users: ['margaret', 'july', 'john', 'peter', 'anthony', 'mary']"
}

TASK [Checking common_users] ****************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking docker users] ****************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking sudo user without password] **************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking sudo user with password] *****************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking that Mary is the app user in the hosts] **************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY RECAP **********************************************************************************************
localhost                  : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Sixth iteration

In this last execution, we are going to deal with the user case that will be used to start our application.

Now what we want is a value, not a list of elements, for that we are going to use an Ansible filter that will not end in a list, and we will take the first positive value that we find.

Our user part would look like this:

				
					vars:
 users:
   - name: margaret
     docker: true
   - name: july
     docker: true
   - name: john
     docker: false
     sudo: without_password
   - name: peter
     sudo: without_password
   - name: anthony
     sudo: with_password
   - name: mary
     app_user: true

Let’s see the versions of our Ansible filter, as always, with and without jmespath.

We’ve decided that the key to this our value will be app_user and it will have a boolean to true value.

Ansible filter without JMESPATH

				
					app_username: "{{ users |
                  selectattr('app_user', 'defined') |
                  selectattr('app_user', 'equalto', True) |
                  map(attribute='name') |
                  first }}"

Ansible filter with JMESPATH

				
					jmespath_app_username: "{{ users | json_query(\"[?app_user].name | [0] \") }}"

Finally, as always, we modify the tests to see that both variables are the same:

				
					- name: Checking that Mary is the app user in the hosts
  assert:
    that:
      - '"mary" == app_username'
      - app_username == jmespath_app_username
    fail_msg: Mary should be the user for the app

Are we really finished? Last iteration

We launch our test, and everything comes out in green, but what would happen if by mistake I also set john’s app_user attribute to true? That is, it would have this in users:

				
					vars:
  users:
    - name: margaret
      docker: true
    - name: july
      docker: true
    - name: john
      docker: false
      app_user: true
      sudo: without_password
    - name: peter
      sudo: without_password
    - name: anthony
      sudo: with_password
    - name: mary
      app_user: true

Finally, as always, we modified the tests to see that both variables are the same:

				
					$ ansible-playbook 06_playbook.yml
...

TASK [Checking that Mary is the app user in the hosts] **************************************************
fatal: [localhost]: FAILED! => {
    "assertion": "\"mary\" == app_username",
    "changed": false,
    "evaluated_to": false,
    "msg": "Mary should be the user for the app"
}

PLAY RECAP **********************************************************************************************
localhost                  : ok=6    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

We can go crazy looking for where the bug is, because mary is marked as app_user.

To correct this case, we can put one more check in the tests that proves that only one user has that property as true, let’s see how we would do it:

				
					- name: Checking that Mary is the app user in the hosts
  vars:
    number_of_users_with_app_username: "{{ users |
                                           json_query(\"[?app_user].name\") |
                                           length }}"
  assert:
    that:
      - number_of_users_with_app_username == '1'
      - '"mary" == app_username'
      - app_username == jmespath_app_username
    fail_msg: Mary should be the user for the app

Now when the test fails it does so because it has found several users with that property, which gives us more clues and is easier to debug.

				
					$ ansible-playbook 06_playbook.yml
...
TASK [Checking that Mary is the app user in the hosts] **************************************************
fatal: [localhost]: FAILED! => {
    "assertion": "number_of_users_with_app_username == '1'",
    "changed": false,
    "evaluated_to": false,
    "msg": "Mary should be the user for the app"
}

PLAY RECAP **********************************************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

We correct the fault and everything works out perfectly. The file for this step is 06_playbook.yml.

Conclusions of this step

This last execution has left our variables quite clean, more maintainable and manageable in the future.

Our variables could end up as follows:

				
					vars:
  users:
    - name: margaret
      docker: true
    - name: july
      docker: true
    - name: john
      docker: false
      sudo: without_password
    - name: peter
      sudo: without_password
    - name: anthony
      sudo: with_password
    - name: mary
      app_user: true
  
  common_users: "{{ users | json_query('[].name') }}"

  docker_users: "{{ users | json_query('[?docker].name') }}"

  sudo_without_password_users: "{{ users | json_query(\"[?sudo=='without_password'].name\") }}"

  sudo_with_password_users: "{{ users | json_query(\"[?sudo=='with_password'].name\") }}"

  app_username: "{{ users | json_query(\"[?app_user].name | [0] \") }}"

Complete execution of this step

				
					ansible-playbook 06_playbook.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'

PLAY [localhost] ****************************************************************************************

TASK [Show common_users var] ****************************************************************************
ok: [localhost] => {
    "msg": "common_users: ['margaret', 'july', 'john', 'peter', 'anthony', 'mary']"
}

TASK [Checking common_users] ****************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking docker users] ****************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking sudo user without password] **************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking sudo user with password] *****************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Checking that Mary is the app user in the hosts] **************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY RECAP **********************************************************************************************
localhost                  : ok=6    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

What did you think?

We have seen in a practical way how we can apply Ansible filters with which we can manage the information in Ansible in a much cleaner, maintainable and manageable way in the future.

If you have any questions you can contact us through the following form.

Do you want to tell us something?

CONTACT US!

Did you like it? Then share it!

LinkedIn
Facebook
Twitter
Email