Deploy Sleeper with the PersistentEMRBulkImportStack

Oct 13, 2025 by wbrum.org 76 views

Description

EMR (Elastic MapReduce) clusters have a crucial limit of 256 running or pending steps. This limit is essential to prevent the cluster from becoming overwhelmed with too many jobs, which can lead to performance issues and errors. When the rate of submission of jobs to the cluster exceeds the rate at which the cluster can process the messages, an exception is thrown, indicating that there are too many steps.

Steps to Reproduce

To reproduce this issue, follow these steps:

Deploy Sleeper with the PersistentEMRBulkImportStack: This step involves setting up the necessary infrastructure for the Sleeper application, including the EMR cluster and the necessary dependencies.
Send lots of import jobs via the Python API: Once the infrastructure is set up, send a large number of import jobs to the cluster using the Python API. This will simulate a high volume of job submissions.
See that some messages will be put on the dead letter queue because the request to add the step failed: As the cluster reaches its limit of 256 running or pending steps, some of the job submissions will fail, and the corresponding messages will be moved to the dead letter queue.

Expected Behaviour

When the EMR cluster reaches its limit of 256 running or pending steps, the expected behaviour is to catch the exception thrown by the cluster and put the message back on the queue with a delay before it becomes visible again. This delay should be set to 1 minute by default, allowing the cluster to process the existing jobs before attempting to add new ones.

Background

It's essential to note that this issue is different from issue #4381, which deals with a rate limit on how frequently steps can be added to the cluster. In contrast, this issue focuses on what happens when the set of steps on the cluster is already full. While it's not sustainable for the rate of submission of jobs to exceed the rate at which they can be processed, it's crucial to allow users to submit large numbers of jobs to the queue and have them processed eventually.

To address this issue, the fix should be designed to handle the scenario where the number of steps on the cluster is 256, indicating that it could take minutes for a job to be finished. In this case, the job should be put back on the queue with a delay before it becomes visible again. This approach ensures that the cluster can process the existing jobs without becoming overwhelmed and that users can continue to submit jobs to the queue.

Proposed Solution

To address this issue, we propose the following solution:

Catch the exception thrown by the EMR cluster: When the cluster reaches its limit of 256 running or pending steps, catch the exception thrown by the cluster and put the message back on the queue.
Add a delay before the message becomes visible again: Set a delay of 1 minute by default before the message becomes visible again on the queue. This allows the cluster to process the existing jobs before attempting to add new ones.
Implement a retry mechanism: Implement a retry mechanism to allow the message to be retried after the delay. This ensures that the message is not lost and that the user can continue to submit jobs to the queue.

Benefits of the Proposed Solution

The proposed solution offers several benefits, including:

Improved cluster performance: By preventing the cluster from becoming overwhelmed with too many jobs, the proposed solution improves cluster performance and reduces the likelihood of errors.
Enhanced user experience: By allowing users to submit large numbers of jobs to the queue and have them processed eventually, the proposed solution enhances the user experience and provides a more reliable service.
Increased scalability: The proposed solution enables the cluster to scale more efficiently, handling a higher volume of job submissions without becoming overwhelmed.

Conclusion

Q: What is the limit of running or pending steps in an EMR cluster?

A: The limit of running or pending steps in an EMR cluster is 256.

Q: What happens when the EMR cluster reaches its limit of 256 running or pending steps?

A: When the EMR cluster reaches its limit of 256 running or pending steps, an exception is thrown, indicating that there are too many steps.

Q: What is the expected behaviour when the EMR cluster reaches its limit of 256 running or pending steps?

A: The expected behaviour is to catch the exception thrown by the cluster and put the message back on the queue with a delay before it becomes visible again.

Q: What is the default delay before the message becomes visible again?

A: The default delay is 1 minute.

Q: Why is it not sustainable for the rate of submission of jobs to exceed the rate at which they can be processed?

A: It's not sustainable for the rate of submission of jobs to exceed the rate at which they can be processed because it can lead to performance issues and errors.

Q: What is the proposed solution to address this issue?

A: The proposed solution involves catching the exception thrown by the EMR cluster, adding a delay before the message becomes visible again, and implementing a retry mechanism.

Q: What are the benefits of the proposed solution?

A: The benefits of the proposed solution include improved cluster performance, enhanced user experience, and increased scalability.

Q: How does the proposed solution handle the scenario where the number of steps on the cluster is 256?

A: The proposed solution handles the scenario where the number of steps on the cluster is 256 by putting the job back on the queue with a delay before it becomes visible again.

Q: Why is it essential to allow users to submit large numbers of jobs to the queue and have them processed eventually?

A: It's essential to allow users to submit large numbers of jobs to the queue and have them processed eventually because it provides a more reliable service and enhances the user experience.

Q: What is the retry mechanism in the proposed solution?

A: The retry mechanism in the proposed solution allows the message to be retried after the delay, ensuring that the message is not lost and that the user can continue to submit jobs to the queue.

Q: How does the proposed solution improve cluster performance?

A: The proposed solution improves cluster performance by preventing the cluster from becoming overwhelmed with too many jobs, reducing the likelihood of errors.

Q: How does the proposed solution enhance user experience?

A: The proposed solution enhances user experience by allowing users to submit large numbers of jobs to the queue and have them processed eventually, providing a more reliable service.

Q: How does the proposed solution increase scalability?

A: The proposed solution increases scalability by enabling the cluster to handle a higher volume of job submissions without becoming overwhelmed.

Q: What is the next step in implementing the proposed solution?

A: The next step in implementing the proposed solution is to catch the exception thrown by the EMR cluster, add a delay before the message becomes visible again, and implement a retry mechanism.

Q: Who should implement the proposed solution?

A: The proposed solution should be implemented by the development team responsible for maintaining the EMR cluster and the associated application.

Q: What are the potential risks of not implementing the proposed solution?

A: The potential risks of not implementing the proposed solution include performance issues, errors, and a poor user experience.