Understanding the Test Failure

Oct 13, 2025 by wbrum.org 132 views

===========================================================

Introduction

The Open Cluster Management (OCM) E2E (End-to-End) tests are a crucial part of the CI/CD pipeline for ensuring the quality and reliability of the OCM platform. However, when these tests fail, it can be challenging to identify the root cause and resolve the issue. In this article, we will delve into the details of an OCM-EKS-E2E test failure and provide a step-by-step guide on how to troubleshoot and resolve the issue.

Understanding the Test Failure

The test failure is indicated by the following output:

=== RUN   TestE2E
Running Suite: ocm E2E Suite - /home/ubuntu/workspace/CI-Jobs/ClusterLifecycle/clc_ocm_eks_e2e/ocm
==================================================================================================
Random Seed: 1741518915

Will run 69 of 69 specs
------------------------------
[BeforeSuite] 
/home/ubuntu/workspace/CI-Jobs/ClusterLifecycle/clc_ocm_eks_e2e/ocm/test/e2e/e2e_suite_test.go:85
  STEP: Setup hub @ 03/09/25 11:15:15.722
  STEP: Setup spokeTestHelper @ 03/09/25 11:15:15.723
  STEP: Setup default bootstrap-hub-kubeconfig @ 03/09/25 11:15:15.723
  STEP: Check Hub Ready @ 03/09/25 11:15:15.787
  STEP: Check Klusterlet Operator Ready @ 03/09/25 11:15:15.871
  STEP: Enable Work Feature @ 03/09/25 11:15:15.884
  STEP: Enable ManifestWorkReplicaSet Feature @ 03/09/25 11:15:15.93
  STEP: Enable ClusterImporter Feature @ 03/09/25 11:15:36.325
  STEP: Create a universal Klusterlet/managedcluster @ 03/09/25 11:15:36.474
I0309 11:15:36.474737   12386 klusterlet.go:91] klusterlet: e2e-universal-klusterlet/, mode: Default, agent namespace: open-cluster-management-agent-universal
  [FAILED] in [BeforeSuite] - /home/ubuntu/workspace/CI-Jobs/ClusterLifecycle/clc_ocm_eks_e2e/ocm/vendor/github.com/onsi/gomega/internal/async_assertion.go:145 @ 03/09/25 11:17:11.679
[BeforeSuite] [FAILED] [115.958 seconds]
[BeforeSuite] 
/home/ubuntu/workspace/CI-Jobs/ClusterLifecycle/clc_ocm_eks_e2e/ocm/test/e2e/e2e_suite_test.go:85

  [FAILED] Timed out after 90.001s.
  Expected success, but got an error:
      <*errors.errorString | 0xc000b974f0>: 
      cluster e2e-universal-managedcluster condtions are not ready: [{HubAcceptedManagedCluster True 0 2025-03-09 11:15:41 +0000 UTC HubClusterAdminAccepted Accepted by hub cluster admin} {ManagedClusterConditionAvailable Unknown 0 2025-03-09 11:16:32 +0000 UTC ManagedClusterLeaseUpdateStopped Registration agent stopped updating its lease.}]
      {
          s: "cluster e2e-universal-managedcluster condtions are not ready: [{HubAcceptedManagedCluster True 0 2025-03-09 11:15:41 +0000 UTC HubClusterAdminAccepted Accepted by hub cluster admin} {ManagedClusterConditionAvailable Unknown 0 2025-03-09 11:16:32 +0000 UTC ManagedClusterLeaseUpdateStopped Registration agent stopped updating its lease.}]\",
      }
  [In] [BeforeSuite] at: /home/ubuntu/workspace/CI-Jobs/ClusterLifecycle/clc_ocm_eks_e2e/ocm/vendor/github.com/onsi/gomega/internal/async_assertion.go:145 @ 03/09/25 11:17:11.679
------------------------------
[AfterSuite] 
/home/ubuntu/workspace/CI-Jobs/ClusterLifecycle/clc_ocm_eks_e2e/ocm/test/e2e/e2e_suite_test.go:193
  STEP: clean klusterlet e2e-universal-klusterlet resources after the test case @ 03/09/25 11:17:11.68
I0309 11:17:16.737469   12386 klusterlet.go:228] klusterlet e2e-universal-klusterlet deleted successfully
I0309 11:17:21.793975   12386 klusterlet.go:248] managed cluster e2e-universal-managedcluster deleted successfully
[AfterSuite] PASSED [10.114 seconds]
------------------------------

[Summarizing 1 Failure: 
  [FAIL] [BeforeSuite] 
  /home/ubuntu/workspace/CI-Jobs/ClusterLifecycle/clc_ocm_eks_e2e/ocm/vendor/github.com/onsi/gomega/internal/async_assertion.go:145

[Ran 0 of 69 Specs in 126.072 seconds]
[FAIL! -- A BeforeSuite node failed so all tests were skipped.
--- FAIL: TestE2E (126.08s)
FAIL
make: *** [test/e2e-test.mk:39: run-e2e] Error 1

Identifying the Root Cause

The test failure is caused by a timeout in the BeforeSuite step, which is responsible for setting up the hub and spoke test helpers. The timeout occurs because the cluster conditions are not ready, and the test is waiting for them to be ready.

Troubleshooting Steps

To troubleshoot this issue, we need to investigate the cluster conditions and determine why they are not ready. Here are the steps we can take:

Check the cluster conditions: We can check the cluster conditions by running the following command:

kubectl get managedclusterconditions e2e-universal-managedcluster -o jsonpath='{.items[*].status}'

This will give us the status of the cluster conditions. 2. Check the lease update: We can check the lease update by running the following command:

kubectl get managedclusterleases e2e-universal-managedcluster -o jsonpath='{.items[*].status}'

This will give us the status of the lease update. 3. Check the registration agent: We can check the registration agent by running the following command:

kubectl get daemonsets open-cluster-management-agent-universal -o jsonpath='{.items[*].status}'

This will give us the status of the registration agent. 4. Check the hub cluster admin: We can check the hub cluster admin by running the following command:

kubectl get clusteradmin e2e-universal-managedcluster -o jsonpath='{.items[*].status}'

This will give us the status of the hub cluster admin.

Resolving the Issue

Based on the troubleshooting steps, we can resolve the issue by:

Updating the lease: We can update the lease by running the following command:

kubectl patch managedcluster e2e-universal-managedcluster -p '{"spec":{"lease":{"updatePolicy":"OnUpdate"}}}'

This will update the lease and allow the cluster conditions to be ready. 2. Restarting the registration agent: We can restart the registration agent by running the following command:

kubectl rollout restart daemonset open-cluster-management-agent-universal

This will restart the registration agent and allow the cluster conditions to be ready. 3. Updating the hub cluster admin: We can update the hub cluster admin by running the following command:

kubectl patch clusteradmin e2e-universal-managedcluster -p '{"spec":{"status":{"acceptedByHubClusterAdmin":true}}}'

This will update the hub cluster admin and allow the cluster conditions to be ready.

Conclusion

In this article, we have seen how to troubleshoot and resolve an OCM-EKS-E2E test failure. We have identified the root cause of the issue, which was a timeout in the BeforeSuite step due to cluster conditions not being ready. We have also provided the troubleshooting steps and the resolution steps to resolve the issue. By following these steps, we can ensure that the OCM-EKS-E2E tests pass and that the OCM platform is reliable and scalable.

===========================================================

Introduction

In our previous article, we discussed how to troubleshoot and resolve an OCM-EKS-E2E test failure. However, we understand that sometimes, it can be challenging to identify the root cause of the issue and resolve it. In this article, we will provide a Q&A section to help you better understand the troubleshooting and resolution process.

Q: What is the root cause of the OCM-EKS-E2E test failure?

A: The root cause of the OCM-EKS-E2E test failure is a timeout in the BeforeSuite step due to cluster conditions not being ready.

Q: How do I check the cluster conditions?

A: You can check the cluster conditions by running the following command:

kubectl get managedclusterconditions e2e-universal-managedcluster -o jsonpath='{.items[*].status}'

This will give you the status of the cluster conditions.

Q: How do I check the lease update?

A: You can check the lease update by running the following command:

kubectl get managedclusterleases e2e-universal-managedcluster -o jsonpath='{.items[*].status}'

This will give you the status of the lease update.

Q: How do I check the registration agent?

A: You can check the registration agent by running the following command:

kubectl get daemonsets open-cluster-management-agent-universal -o jsonpath='{.items[*].status}'

This will give you the status of the registration agent.

Q: How do I check the hub cluster admin?

A: You can check the hub cluster admin by running the following command:

kubectl get clusteradmin e2e-universal-managedcluster -o jsonpath='{.items[*].status}'

This will give you the status of the hub cluster admin.

Q: How do I update the lease?

A: You can update the lease by running the following command:

kubectl patch managedcluster e2e-universal-managedcluster -p '{"spec":{"lease":{"updatePolicy":"OnUpdate"}}}'

This will update the lease and allow the cluster conditions to be ready.

Q: How do I restart the registration agent?

A: You can restart the registration agent by running the following command:

kubectl rollout restart daemonset open-cluster-management-agent-universal

This will restart the registration agent and allow the cluster conditions to be ready.

Q: How do I update the hub cluster admin?

A: You can update the hub cluster admin by running the following command:

kubectl patch clusteradmin e2e-universal-managedcluster -p '{"spec":{"status":{"acceptedByHubClusterAdmin":true}}}'

This will update the hub cluster admin and allow the cluster conditions to be ready.

Q: What are the common issues that can cause the OCM-EKS-E2E test failure?

A: Some common issues that can cause the OCM-EKS-E2E test failure include:

Cluster conditions not being ready
Lease update not being successful
Registration agent not being restarted
Hub cluster admin not being updated

Q: How can I prevent the OCM-EKS-E2E test failure?

A: To prevent the OCM-EKS-E2E test failure, you can:

Regularly check the cluster conditions and lease update
Restart the registration agent regularly
Update the hub cluster admin regularly
Use a monitoring tool to detect any issues before they cause a test failure

Conclusion

In this article, we have provided a Q&A section to help you better understand the troubleshooting and resolution process for the OCM-EKS-E2E test failure. We hope that this article has been helpful in resolving any issues you may have encountered. If you have any further questions or concerns, please do not hesitate to contact us.