-
Notifications
You must be signed in to change notification settings - Fork 574
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Config Sync - failed reloads due to uncomplete syncs #7742
Comments
Please also share the output of |
The attached file is from one of our "middle" Satellites (from the "aws-frankfurt-satellite" zone). I hope is enough and helps. Let me know if you need the output from all 4 affected satellites. (anonymizing is always a little bit difficult) |
We were facing the same issue: https://community.icinga.com/t/global-configuration-zone-missing-check-commands/2976/6 Should you need more debugging data, we would be happy to switch our config sync back to Icinga2 and send you logfiles. |
Note: The example error message implies an unexpectedly changed FS tree, but we definitively lock |
Hello @Clasko and thank you for reporting! independent of this issue you should upgrade to v2.11.3 not to have a lot of other trouble. Best, |
@Clasko Please could you test v2.11.3 + #7917: https://git.icinga.com/packaging/rpm-icinga2/-/jobs/45459 / "Job artifacts" / "Download" |
I'm on vacation the next 2 weeks. I will see if a colleague can do the testing. |
If they can't and the artifacts disappear – just let me know once you'll be going to do the tests and I'll re-create the artifacts. |
Did you upgrade all of the nodes to the same version? If no, please share the Icinga 2 versions of all nodes in both the zone of the affected node and all parent zones. Also please share the output of |
Also: Which zones do you have config for and in which dir on the affected node?
|
I've upgrades all (from my point of view) affected nodes. Which means: Master: The issue currently occurs only on this 4 (2 HA zones) Satellites. We have other Satellites als childs of our master which a not affected by this issue. These satellites are on version Output of the find command on our config master. The output on the affected satellites is empty:
I can not share an unanonymizing output of our zone names as it contains customer names on GitHub. Sorry if my answer are a bit confusing due to anonymizing my outputs. I can provide raw output if you can provide me a nextcloud filedrop link or via netways ticket #663455 if this helps. |
|
to 1) I know and follow this rule on stable releases but i'm a bit careful with Snapshot or even RPMs directly from the master branch in a production enviroment if not absolutly necessary. I will try to reproduce the issue in our test enviroment but i had no luck in the past. I will reconsider when my next attempts fails again. |
3: Fine. One point of failure fewer. 1: Snapshots are RPMs directly from the master branch. But my RPMs are neither of those. I know customers' stability requirements and you can fully trust me: If I say "This packages contain version X + PR Y", the packages won't any line of code more. |
I've upgrades our two masters to |
ref/NC/663455 |
NoteThe have been three problems:
|
Describe the bug
We're facing an issue with failed config reloads due to uncomplete syncs of our global-templates zone.
But this only happens in a zone with a second satellite hierarchy (Master -> Satellite -> Satellite). Satellites without Childs are not affected by this.
We can only fix this by purging
/var/lib/icinga2/api/zones
and/var/lib/icinga2/api/zones-stage
on the satellites.This only happens on object creation or deletion. Changes on already existing objects does not trigger this issue.
Error Message Example:
Error: Function call 'opendir' for file '/var/lib/icinga2/api/zones-stage//global-templates/_etc/credentials' failed with error code 2, 'No such file or directory'
To Reproduce
Expected behavior
Working Config sync and reload on all Icinga nodes
Your Environment
Include as many relevant details about the environment you experienced the problem in
icinga2 --version
):icinga2 feature list
):icinga2 daemon -C
):zones.conf
file (oricinga2 object list --type Endpoint
andicinga2 object list --type Zone
) from all affected nodes.Additional context
N/A
The text was updated successfully, but these errors were encountered: