First, thanks to Patrick Moffitt and David Fidler for their help on this topic!
Aria Automation includes a configuration management solution based on SaltStack technology – called Aria Automation Config. It’s integrated into the design canvas and is part of a full provisioning process from VM OS up to application level. SaltStack consists of a master component (server side) and minions (client side). As part of the provisioning process Aria Automation takes care of the initial minion installation on the targets. However as of today it does not manage the update process of the minions. While updates can be done manually for a smaller number of clients, it’s challenging in a high scale environment.
The main challenge is that when doing a minion update, the minion service is restarted. This means that the related job will not get a response and you’ll see orphaned job entries in the SaltStack management server.
This blog describes a potential way to overcome this and do mass minion update using the SaltStack minion component itself. In this example a Windows-based SaltStack minion is used.
State file for minion update
As first step you’ll need a state file that handles the minion update itself. It’s basically using commands to download a newer version of the salt minion into a temporary directory and afterwards executes a silent installation process. If an older minion version is installed, it’s automatically updated by this process. Just be aware that internet connectivity is required if the files are downloaded from the public page. Alternatively, you can point the state file to a local web server where the file has been preloaded.
get_salt_installer:
file.managed:
- name: C:\temp\Salt-Minion-3006.3-Py3-AMD64-Setup.exe
- source: https://repo.saltproject.io/salt/py3/windows/latest/Salt-Minion-3006.3-Py3-AMD64-Setup.exe
- skip_verify: True
- makedirs: True
salt_installed:
cmd.run:
- name: C:\temp\Salt-Minion-3006.3-Py3-AMD64-Setup.exe /S
- bg: true
In general, you can store this file in any structure on the file server you like provided the reference from other files works.
In my case I used this structure:
Orchestrator state file
As mentioned in the intro the main challenge is the minion restart during the setup. To solve this, the idea is to wrap the installation process in an orchestration workflow and successfully end the job once the salt minion has been restarted. SaltStack does support orchestration workflows which work in similar way to normal state files, and it has an event-based mechanism that can be used to identify when a salt minion restart has run.
Apart from that there needs to be a way to hand over the minions this job should be executed on to the orchestration file. There’s no simple method to just provide the executing minion name to the job. In this example we expect that all relevant minion names are populated in a pillar as json structure. Alternatively, you could also store a grain on the minion that marks the system as to-be-updated (e.g. minionupgrade:1) and clear the grain after the last execution step.
Here’s a related orchestration state file which uses the pillar mechanism. It basically takes the pillar data and repeats the state file code for each entry (minon name m).
{% set minions = pillar.get('minions', '') %}
{% if minions %}
{% for m in minions %}
do_upgrade_{{ m }}:
salt.state:
- tgt: {{ m }}
- sls:
- minion-update.initbackground
wait_for_start_{{ m }}:
salt.runner:
- name: state.event
- tagmatch: salt/minion/{{ m }}/start
- count: 1
check_minion_{{ m }}:
salt.function:
- tgt: {{ m }}
- name: test.ping
{% endfor %}
{% else %}
no_minions_selected:
salt.function:
- name: test.false
- tgt: raas
{% endif %}
It is stored in similar way as the other file on the file server:
Create update job
Now you must create an update job that is executing the SaltStack orchestration file. Make sure you provide the pillar “minion” in some way. For a simple first test you could just populate the job “pillar override” field (see screenshot below) with a json list of the minions to be updated.
Testing the procedure
First make sure you have a Windows system with an older minion version installed (in my case saltminion07).
Select the minion in Aria Automation Config and run the previously created job (make sure you have the minion pillar data correct).
Watch the job activities. Once finished you should see 4 succeeded jobs:
In the job details you should see the executions that happened. You won’t see an output of the installation job directly as the salt minion is restarted in the middle of it. However a successful execution means that after the installation has run the salt minion was started successfully which is checked by catching the salt minion start event as well as doing a test.ping command which at least shows the salt minion is operational. Optionally there might be some ways as well to verify the minion version after the update, but this is not covered in this blog.
As a manual check you can go to your Windows client and verify the resulting salt minion version installed.
Hope, this blog is some help for you!
Have fun!
- Approval with Jira change ticket - 4. January 2024
- Automated SaltStack Minion Updates through winrepo - 9. November 2023
- Automated SaltStack Minion Updates - 27. October 2023