Troubleshooting Clusters
This page covers the most common problems seen while upgrading, reconciling, and removing clusters managed by the KubeDB Platform. Each section follows the same pattern: Symptom → Check → Fix.
Terms used below:
- Hub cluster — the cluster running the Open Cluster Management (OCM) hub. It holds one
ManagedClusterobject per spoke. - Spoke cluster — a cluster joined to the hub. It runs the
klusterletagent, installed by thecluster-manager-spokeHelm chart. - HelmRelease (HR) — the FluxCD resource through which every platform feature is installed (see Feature Reference).
Before You Start
Collect the current state first. Run these against the cluster that shows the problem.
# All HelmReleases and their Ready status
kubectl get helmreleases -A
# Only the ones that are not Ready
kubectl get helmreleases -A | grep -v True
# Flux controllers must be running
kubectl get pods -n flux-system
On the hub cluster, check the spoke’s connection state:
kubectl get managedclusters
NAME HUB ACCEPTED MANAGED CLUSTER URLS JOINED AVAILABLE AGE
monitoring-1 true https://10.2.1.87:6443 True Unknown 12d
AVAILABLE=Unknown means the spoke’s agent stopped reporting to the hub (the cluster is
inactive). See Removing an Inactive Cluster.
Through the API, the same information is available from
GET /clustersv2/{owner}/{cluster}/status.
Stuck Reconciliation
Symptom
A HelmRelease stays Ready=Unknown with the message Reconciliation in progress, or the
version shown in the UI does not change after an upgrade.
Check
kubectl describe helmrelease <name> -n <namespace>
Look at the Conditions and Events sections. The common causes:
| What you see | Cause |
|---|---|
spec.suspend: true | The HelmRelease is suspended and will not reconcile. |
HelmChart ... is not ready / source not found | The chart source (HelmRepository / HelmChart) is failing. |
dependency ... is not ready | A HelmRelease listed in spec.dependsOn is not Ready yet. |
| No new events for a long time | The Flux controllers are not running or are crash-looping. |
Fix
- Suspended — resume it:
kubectl patch helmrelease <name> -n <namespace> --type merge -p '{"spec":{"suspend":false}}' - Source not ready — check the source object and the source-controller logs:
kubectl get helmrepositories,helmcharts -A | grep -v True kubectl logs -n flux-system deploy/source-controller --tail=100 - Dependency not ready — fix the HelmRelease it depends on first; this one follows.
- Controllers not running — check
kubectl get pods -n flux-systemand thehelm-controllerlogs:kubectl logs -n flux-system deploy/helm-controller --tail=100 - Force a reconcile once the cause is fixed:From the API,
kubectl annotate helmrelease <name> -n <namespace> \ reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwriteGET /clustersv2/{owner}/{cluster}/reconcile?features=<feature>does the same for the HelmReleases of the given features.
HelmRelease Progress Issues
Symptom
A HelmRelease shows Ready=False with a reason such as InstallFailed, UpgradeFailed, or
RetriesExceeded, and the upgrade does not move forward.
Check
kubectl get helmrelease <name> -n <namespace> \
-o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
# Helm's own history of the release
helm history <release-name> -n <release-namespace>
Fix
| Situation | Fix |
|---|---|
| A pod in the release does not become Ready (image pull, resources, probe) | Fix the workload (kubectl describe pod, kubectl logs). Then force a reconcile (see above). |
RetriesExceeded | Fix the cause, then suspend and resume the HelmRelease to reset the retry counter. |
helm history shows the last revision as pending-install, pending-upgrade, or pending-rollback | A previous Helm operation was interrupted and holds the release lock. Roll back to the last deployed revision with helm rollback <release-name> <revision> -n <release-namespace>, then force a reconcile. |
Suspend and resume:
kubectl patch helmrelease <name> -n <namespace> --type merge -p '{"spec":{"suspend":true}}'
kubectl patch helmrelease <name> -n <namespace> --type merge -p '{"spec":{"suspend":false}}'
If the upgrade job reported a release as failed but the HelmRelease is already on the target
chart version with Ready=True, a later reconciliation succeeded. No action is needed.
Spoke Version Does Not Change
Symptom
After upgrading a spoke from the Hub UI, CURRENT VERSION in
kubectl get managedclusterprofilebinding -A shows the target version, but the HelmReleases on
the spoke stay on the old version.
Check
- On the hub cluster, check that the spoke is connected:The spoke must show
kubectl get managedclustersJOINED=TrueandAVAILABLE=True. - On the spoke cluster, check the HelmReleases:
kubectl get helmreleases -A | grep -v True
Fix
AVAILABLE=Unknown— the hub cannot deliver the new version to the spoke. Restore the spoke’s connectivity, or remove it (see Removing an Inactive Cluster).- Spoke connected, HelmReleases not Ready — follow Stuck Reconciliation and HelmRelease Progress Issues on the spoke.
ClusterSet Stays Unaligned
Symptom
After Update Cluster Set, the ClusterSet still shows the old Current version or a cluster labelled unaligned.
Check
On the hub cluster, find the spoke that is not on the target version:
kubectl get managedclusterprofilebinding -A
Fix
- Upgrade that spoke (see Upgrade ACE Clusters), or remove it if it is no longer in use (see Removing a Cluster).
- Click Update Cluster Set again.
Removing a Cluster
Removing through the API
POST /clustersv2/{owner}/{cluster}/remove
is asynchronous: 200 means the removal task was accepted, not that it finished.
- Call it once.
- Poll
GET /clustersv2/{owner}/{cluster}/statusto follow progress. The cluster is removed when it no longer appears in the cluster list. - Do not call
removeagain while the first removal is running. A second call returns409 Conflictwithcluster removal already in progress.
Removing an Inactive Cluster
When a spoke is inactive (AVAILABLE=Unknown on the hub), the platform cannot reach it, so
removal skips the spoke-side cleanup. The spoke’s ManagedCluster object stays on the hub
and must be removed manually.
On the hub cluster:
- Delete the
ManagedCluster. This is the only hub object you need to remove; OCM cleans up the cluster’s namespace, ManifestWorks, and add-ons.kubectl delete managedcluster <cluster-name> --wait=false kubectl get managedcluster <cluster-name> - If it stays in
Terminating, the finalizers are waiting for an agent that no longer responds. Remove them:kubectl patch managedcluster <cluster-name> --type merge -p '{"metadata":{"finalizers":null}}'
On the spoke cluster (if you can still reach it):
- Check whether the OCM agent is still installed:
kubectl get klusterlet helm ls -A --all | grep cluster-manager-spoke - If either returns a result, the spoke still thinks it belongs to the hub. Uninstall the
chart using the namespace shown by
helm ls:helm uninstall cluster-manager-spoke -n <namespace> - If the
klusterletobject stays inTerminating, remove its finalizers:kubectl patch klusterlet klusterlet --type merge -p '{"metadata":{"finalizers":null}}'
See also Remove / Delete Cluster for removing a cluster from the UI.
































