Create a check that always returns a warning

Oct 13, 2025 by wbrum.org 73 views

Introduction

In this article, we will discuss a critical issue that can occur in the Checkable::ProcessCheckResult function in the lib/icinga/checkable-check.cpp file. This function is responsible for processing check results and updating the object state accordingly. However, due to a race-condition, it can happen that both update the object before they dispatch the events to the IDO and notifications, resulting in the second state being dispatched multiple times.

Describe the bug

The Checkable::ProcessCheckResult function in the lib/icinga/checkable-check.cpp file locks and unlocks the Mutex on the Objects several times. If two check results come in at the same time (e.g. through a running check and the /v1/actions/process-check-result endpoint) it can happen that both update the object before they dispatch the events to the IDO and notifications, resulting in the second state being dispatched multiple times.

To Reproduce

This issue is used in production for a countdown to notify if no event came in in the last few minutes. To reproduce this issue, follow these steps:

Create a check that always returns a warning: This check will be used to simulate a warning state.
Create a service that runs that check every 30 seconds (or less, for more events): This service will be used to simulate a frequent check.
Send repeatedly check results for that service to the /v1/actions/process-check-result endpoint: This will simulate multiple check results coming in at the same time.
After some time, there should occur this exact scenario: This is where the issue will occur, resulting in the second state being dispatched multiple times.
The chances can be improved with multiple services: Having multiple services will increase the chances of this issue occurring.
The chances are improved after a restart, as multiple services need to rerun the check, while a retry logic for the checkresult endpoint will send the new state: After a restart, the issue will occur more frequently due to the multiple services and retry logic.
The occurrence can be detected by reading out the icinga:history:stream:state redis stream from icingadb, checking if an event was sent twice: This is how the issue can be detected.

Expected behavior

The expected behavior is that all state changes will trigger the appropriate events, even if they happened at the same time.

Screenshots

Your Environment

The issue was experienced in the following environment:

Version used (icinga2 --version): r2.14.3-1
Operating System and version: Red Hat Enterprise Linux 8.10

Conclusion

In conclusion, the Checkable::ProcessCheckResult function in the lib/icinga/checkable-check.cpp file has a critical issue that can occur due to a race-condition. This issue can result in the second state being dispatched multiple times, leading to incorrect behavior. To reproduce this issue, follow the steps outlined above. The expected behavior is that all state changes will trigger the appropriate events, even if they happened at the same time.

Recommendations

To fix this issue, the following recommendations can be made:

Improve the locking mechanism: The locking mechanism in the Checkable::ProcessCheckResult function should be improved to prevent multiple updates to the object state at the same time.
Use a more robust synchronization mechanism: A more robust synchronization mechanism, such as a semaphore or a mutex with a higher priority, can be used to prevent multiple updates to the object state at the same time.
Implement a retry logic: A retry logic can be implemented to retry the check result processing if it fails due to a race-condition.
Monitor the issue: The issue should be monitored to detect its occurrence and to identify the root cause.

Introduction

In our previous article, we discussed a critical issue that can occur in the Checkable::ProcessCheckResult function in the lib/icinga/checkable-check.cpp file. This issue is caused by a race-condition that can result in the second state being dispatched multiple times, leading to incorrect behavior. In this article, we will answer some frequently asked questions about this issue.

Q: What is a race-condition?

A: A race-condition is a situation where two or more threads or processes are accessing shared resources simultaneously, leading to unpredictable behavior. In the case of the Checkable::ProcessCheckResult function, the race-condition occurs when two check results come in at the same time, causing both updates to the object state to be processed simultaneously.

Q: What are the symptoms of this issue?

A: The symptoms of this issue include:

Multiple events being dispatched for the same state change
Incorrect behavior due to the second state being dispatched multiple times
Difficulty in reproducing the issue due to its intermittent nature

Q: How can I reproduce this issue?

A: To reproduce this issue, follow these steps:

Create a check that always returns a warning
Create a service that runs that check every 30 seconds (or less, for more events)
Send repeatedly check results for that service to the /v1/actions/process-check-result endpoint
After some time, there should occur this exact scenario

Q: What are the chances of this issue occurring?

A: The chances of this issue occurring can be improved with multiple services and a retry logic for the checkresult endpoint. After a restart, the issue will occur more frequently due to the multiple services and retry logic.

Q: How can I detect this issue?

A: This issue can be detected by reading out the icinga:history:stream:state redis stream from icingadb, checking if an event was sent twice.

Q: What are the expected behavior and actual behavior?

A: The expected behavior is that all state changes will trigger the appropriate events, even if they happened at the same time. The actual behavior is that the second state is dispatched multiple times, leading to incorrect behavior.

Q: How can I fix this issue?

A: To fix this issue, the following recommendations can be made:

Improve the locking mechanism
Use a more robust synchronization mechanism
Implement a retry logic
Monitor the issue

Q: What are the benefits of fixing this issue?

A: Fixing this issue will ensure that the object state is updated correctly, even if multiple check results come in at the same time. This will prevent incorrect behavior and ensure that the system operates as expected.

Conclusion

In conclusion, the Checkable::ProcessCheckResult function in the lib/icinga/checkable-check.cpp file has a critical issue that can occur due to a race-condition. This issue can result in the second state being dispatched multiple times, leading to incorrect behavior. By following the recommendations outlined in this article, the issue can be fixed, and the object state will be updated correctly, even if multiple check results come in at the same time.