Implement a retry mechanism with a backoff strategy

Oct 13, 2025 by wbrum.org 134 views

Client Overwhelmed by Large Stack Traces: A Critical Issue in Infinispan

Infinispan, a popular in-memory data grid, is designed to handle large amounts of data and provide high-performance caching capabilities. However, a recent bug has been identified that can cause the client to be overwhelmed by large stack traces, leading to a significant decrease in performance and potentially causing the application to crash. In this article, we will delve into the details of this issue, its expected behavior, and the actual behavior observed.

The bug in question is related to the handling of a RejectedExecutionException in Infinispan. This exception is thrown when the client is unable to execute a task due to a lack of available resources. In the past, this exception was ignored, but a recent commit (https://github.com/infinispan/infinispan/commit/e1958a6df75c5b885cea71200aa5dafdeaa974ed) has changed the behavior to propagate this exception on the calling thread. This can lead to a loop where the client continues to retry the exception, causing the stack trace to grow exponentially.

The expected behavior in this scenario is for the client to simply not respond to the request, indicating that the operation has failed. This would prevent the client from entering an infinite loop and would allow the application to recover from the failure.

Unfortunately, the actual behavior observed is that the client continues to retry the exception, leading to a large stack trace that can overwhelm the client and cause the application to crash. This behavior is not only unexpected but also detrimental to the performance and reliability of the application.

To reproduce this issue, you can follow these steps:

Create a test case that simulates a RejectedExecutionException being thrown by the client.
Run the test case and observe the behavior of the client.
Verify that the client continues to retry the exception and grows the stack trace exponentially.

No output is provided for this section, as it is not relevant to the issue at hand.

No additional information is provided for this section, as it is not relevant to the issue at hand.

To remedy this issue, the client should not retry an exception if it failed with no server key. Instead, the publisher should be marked as being in a failure state. This would prevent the client from entering an infinite loop and would allow the application to recover from the failure.

In conclusion, the issue of large stack traces overwhelming the client is a critical problem in Infinispan. By understanding the expected behavior, actual behavior, and how to reproduce the issue, we can work towards a solution that prevents the client from entering an infinite loop and allows the application to recover from the failure. By marking the publisher as being in a failure state, we can prevent the client from retrying the exception and growing the stack trace exponentially. This solution would not only improve the performance and reliability of the application but also prevent the client from being overwhelmed by large stack traces.

Based on the analysis of this issue, the following recommendations are made:

Implement a retry mechanism with a backoff strategy: This would allow the client to retry the exception after a certain period of time, preventing the client from entering an infinite loop.
Mark the publisher as being in a failure state: This would prevent the client from retrying the exception and growing the stack trace exponentially.
Monitor the client's behavior: This would allow the application to detect when the client is experiencing issues and take corrective action to prevent the client from being overwhelmed by large stack traces.

By implementing these recommendations, we can prevent the client from being overwhelmed by large stack traces and improve the performance and reliability of the application.
Q&A: Client Overwhelmed by Large Stack Traces in Infinispan

In our previous article, we discussed the issue of large stack traces overwhelming the client in Infinispan. In this article, we will provide a Q&A section to address some of the common questions and concerns related to this issue.

A: The root cause of this issue is the propagation of a RejectedExecutionException on the calling thread, which can lead to a loop where the client continues to retry the exception, causing the stack trace to grow exponentially.

A: This issue is not being ignored like before because the recent commit (https://github.com/infinispan/infinispan/commit/e1958a6df75c5b885cea71200aa5dafdeaa974ed) has changed the behavior to propagate this exception on the calling thread, making it more noticeable and problematic.

A: The expected behavior in this scenario is for the client to simply not respond to the request, indicating that the operation has failed. This would prevent the client from entering an infinite loop and would allow the application to recover from the failure.

A: The actual behavior observed is that the client continues to retry the exception, leading to a large stack trace that can overwhelm the client and cause the application to crash.

A: To reproduce this issue, you can follow these steps:

Create a test case that simulates a RejectedExecutionException being thrown by the client.
Run the test case and observe the behavior of the client.
Verify that the client continues to retry the exception and grows the stack trace exponentially.

A: The solution to this issue is to mark the publisher as being in a failure state, preventing the client from retrying the exception and growing the stack trace exponentially.

A: To implement this solution, you can follow these steps:

Modify the client to mark the publisher as being in a failure state when a RejectedExecutionException is thrown.
Implement a retry mechanism with a backoff strategy to allow the client to retry the exception after a certain period of time.
Monitor the client's behavior to detect when the client is experiencing issues and take corrective action to prevent the client from being overwhelmed by large stack traces.

A: The benefits of implementing this solution include:

Improved performance: By preventing the client from entering an infinite loop, the application can recover from the failure and continue to operate.
Improved reliability: By marking the publisher as being in a failure state, the client can detect when the publisher is not available and take corrective action.
Reduced risk of application crashes: By preventing the client from being overwhelmed by large stack traces, the application is less likely to crash due to this issue.

In conclusion, the issue of large stack traces overwhelming the client in Infinispan is a critical problem that can cause the application to crash. By understanding the root cause of this issue, the expected behavior, and the actual behavior observed, we can work towards a solution that prevents the client from entering an infinite loop and allows the application to recover from the failure. By marking the publisher as being in a failure state, we can prevent the client from retrying the exception and growing the stack trace exponentially. This solution would not only improve the performance and reliability of the application but also prevent the client from being overwhelmed by large stack traces.