Problem: 'node' concurrency_mode not working as described

[greenseeker]

Issue

I'm using ansible to configure VMs immediately after provisioning. These VMs are provisioned on-demand and our users are waiting for them, so I need to be as quick as possible. During a typical day, as many as 15 of these VMs could be provisioning in parallel.

This works fine with vanilla ansible, but with semaphore, if I try to run the same task template multiple times in parallel, they all queue up. I found the concurrency_mode option and set it to 'node'. Docs says this should allow them to run in parallel if they are targeting different hosts, which they are, but they still queue up.

Depending on the needs of the VM, it can take anywhere from 5-30 minutes for the playbook to complete. Today we had 141 VMs created. If we restrict the playbook to running sequentially, that would take between 12 and 70 hours.

My censored config.json is below. Have I misconfigured something? Have I misunderstood node concurrency?

Impact

Ansible (task execution)

Installation method

Package

Database

MySQL

Browser

No response

Semaphore Version

2.10.22-e44910d-1721658299

Ansible Version

ansible [core 2.16.3]
  config file = /opt/staging/playbook/ansible.cfg
  configured module search path = ['/home/staging/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/staging/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.12.3 (main, Jul 31 2024, 17:43:48) [GCC 13.2.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

Logs & errors

No response

Manual installation - system information

$ uname -a
Linux xxxxxxx 6.8.0-1013-azure #15-Ubuntu SMP Thu Aug  8 18:40:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Configuration

{
    "mysql": {
        "host": "127.0.0.1:3306",
        "user": "xxxxxxx",
        "pass": "xxxxxxx",
        "name": "semaphore_db",
        "options": null
    },
    "dialect": "mysql",
    "port": "80",
    "concurrency_mode": "node",
    "tmp_path": "/var/tmp/semaphore",
    "cookie_hash": "xxxxxxx",
    "cookie_encryption": "xxxxxxx",
    "access_key_encryption": "xxxxxxx",
    "email_alert": true,
    "email_sender": "semaphore@xxxxxxx.com",
    "email_host": "mail.xxxxxxx.com",
    "email_port": "25"
}

Additional information

No response

[greenseeker]

I think I understand why this is happening. Even though tasks are executed against a single host listed in the playbook...

---
- hosts: "{{ target_vm }}"
  roles:
...

...where target_vm is passed as an extra var, they still share the same inventory. The node concurrency check seems to be checking either if the tasks share the same inventory, or if their target hosts are found in each other's inventory. In either case, this would look like a collision in my case, even though it's not.

I'll need to find some way to isolate inventories that can be automatically maintained, and isn't too convoluted.