Troubleshooting Your Installation¶
Symptoms you may see during or after installation, what usually causes them, and what to do. The commands in this document assume the release home-cloud-control-plane in the namespace home-cloud-control-plane.
Before You Start¶
Start every investigation with the following:
NS=home-cloud-control-plane
kubectl get pods -n $NS # what is not Running or not Ready
kubectl get events -n $NS --sort-by=.lastTimestamp | tail -30
kubectl describe pod <pod> -n $NS # why a pod is Pending, Init or ContainerCreating
The control plane logs one JSON object per line. To see only errors, with the service that logged each one:
kubectl logs -n $NS deploy/home-cloud-control-plane-app -c home-cloud-control-plane \
| grep '^{' | jq -r 'select(.level=="ERROR") | "\(.hostedFleetService): \(.message)"'
Add --previous to read the log of a container that has already restarted.
The Control Plane Does Not Start¶
| Pod state | Likely cause | What to do |
|---|---|---|
Pending |
Not enough CPU or memory on any node, or a volume cannot be provisioned | kubectl describe pod names the reason. Add capacity, or check that the StorageClass exists and can provision volumes. |
ContainerCreating for a long time |
A Secret the pod mounts does not exist, such as a CA Secret named in composite.trustedCA |
The FailedMount event names the Secret; create it |
Init:… |
An initialization step is waiting or failing | Read the step's log, as below |
CrashLoopBackOff |
The control plane exits at startup | Read its log with --previous, as below |
Running but not ready, for up to 20 minutes |
First start; services start a few at a time | Wait. If it is still not ready after 20 minutes, the pod is restarted; see A Service Reports Unknown Health. |
OOMKilled in kubectl describe |
The memory limit is too low | Raise composite.resources; see Compute Resources |
The initialization steps run in this order. Read a step's log with kubectl logs -n $NS deploy/home-cloud-control-plane-app -c <step>:
| Step | Does | Typical failure |
|---|---|---|
wait-for-data-layer |
Waits until every data store accepts connections | It keeps waiting: the store is not running, or its host or port is wrong |
build-trusted-cacerts |
Adds your CAs to the trusted certificates; present only when composite.trustedCA is set |
A file in the CA Secret holds no certificate, or one that cannot be read |
broker-bootstrap |
Prepares the in-cluster event broker | 401 from the broker: it kept an older password from a previous installation; see Uninstalling |
postgres-bootstrap |
Creates the databases | The PostgreSQL account lacks CREATEDB, or the password is wrong |
object-store-bootstrap |
Creates the bucket and the access identity | The bundled object store is not ready yet |
Startup errors in the control plane's own log:
| Message contains | Cause | What to do |
|---|---|---|
maas.vault.local-key must be 64 hex characters |
VAULT_LOCAL_KEY is missing or malformed |
Set it to 64 hexadecimal characters (openssl rand -hex 32). On an existing installation, use the original key. |
database "maas_…" does not exist |
The database was not created | Read the postgres-bootstrap log. With your own PostgreSQL, give the account CREATEDB or create the databases. |
password authentication failed |
The PostgreSQL password does not match the server | Bundled PostgreSQL keeps the password its volume was created with; changing it in your values later does not change it on the server |
cannot decrypt a value this mode did not produce |
The encryption key changed under existing data | Restore the original VAULT_LOCAL_KEY. Data encrypted under a lost key cannot be recovered. |
PKIX path building failed |
A data store or broker presents a certificate from a CA the control plane does not trust | Add the CA; see Trust Your Private CA |
A Service Reports Unknown Health¶
The control plane is ready only when every one of its services has started and none reports DOWN. List each service's state:
kubectl exec -n $NS deploy/home-cloud-control-plane-app -c home-cloud-control-plane -- \
sh -c 'curl -s -H "Authorization: Bearer $HOSTED_FLEET_CONTROL_TOKEN" http://127.0.0.1:$COMPOSITE_PORT/_composite/services' \
| jq -r '.[] | "\(.name) \(.status)"'
Then read that service's errors: filter the log on hostedFleetService, as in Before You Start. Most failures are a data store or the event broker the service cannot reach.
A Data Store Fails at Boot¶
- OpenSearch. The control plane connects without credentials. An endpoint that requires authentication is not supported; restrict access by network instead.
- Cache. Cluster mode and username-and-password authentication are not supported. Use a standalone endpoint, or Amazon ElastiCache with IAM authentication.
- Object store. With your own store, the bucket must already exist and the identity needs the permissions in Before You Begin.
- Any store, after a reinstall. Kept volumes keep their old data and passwords; see Uninstalling.
Pods Cannot Pull Images¶
ImagePullBackOff or ErrImagePull. kubectl describe pod shows the image reference that failed.
| Cause | What to do |
|---|---|
| The pull secret is missing in the namespace, or has the wrong credentials | Create or correct it. Credentials that expire break pulls again later. |
| The pull secret is for another registry | Its server must be the registry in the failing reference |
| Only some images point at your registry | Set all five registry values together; see Installing in an Air-Gapped Environment |
| The image is not in your registry | Usually an event broker version, the event broker's monitoring sidecar, or busybox. Mirror it into the registry the datacenter pulls from. |
The Datacenter Agent Does Not Start¶
| Symptom | Cause | What to do |
|---|---|---|
The agent stays at 0/0 |
Registration has not finished. The agent starts only after the registration job succeeds. | Follow kubectl logs -n $NS job/home-cloud-control-plane-datacenter-register -f |
| The registration job failed | Usually the control plane was not ready, or a sign-in failed (401) |
Read the job's log. A failed job is kept for that reason. Fix the cause, then run helm upgrade with the same values; it runs registration again. |
helm install reports a timeout, but pods keep starting |
The client stopped waiting; the registration job carries on | Follow the job's log. Use --timeout 30m. |
The agent is in ContainerCreating |
A broker certificate Secret named in the values does not exist, or a key name is wrong | Create it; see Broker Certificates. The agent does not fall back to self-signed certificates. |
The agent restarts, and its log contains 9021 |
The broker CA was refused: expired, too close to expiry, not a CA, or not RSA | The log names the check that failed; replace the CA |
A sign-in fails with 401 and you manage the Secret yourself |
The Secret lacks a key | Check it has every key in Provide Secrets |
Event Broker Service Creation or Upgrade Fails¶
| Symptom | Cause | What to do |
|---|---|---|
| Creation fails on a NodePort datacenter | serviceHostname does not resolve or is not reachable from inside the cluster |
Use a node address, or a name your cluster's DNS resolves; see NodePort |
The event broker service pod is in ImagePullBackOff |
That event broker version's image is not in the registry | Load the image; see Event Broker Versions and Images |
Pre-upgrade checks fail on Kubernetes resource access, and a check pod is in ImagePullBackOff |
The busybox image the checks use is not in the registry |
Load busybox:latest into the registry the datacenter pulls from |
| An event broker behind a LoadBalancer never gets an address | No load balancer provider in the cluster | Install one, or use NodePort or ClusterIP for new event broker services |
| Clients reject the event broker service's certificate name | The client used an address that is not on the certificate | Use the address shown on the service's Connect tab, or your own CA for ClusterIP and LoadBalancer |
The datacenter agent's log records each step of every creation and upgrade:
kubectl logs -n <datacenter namespace> deploy/<datacenter id> -c solace-cloud-ca
The UI Does Not Load¶
| Symptom | Cause | What to do |
|---|---|---|
| The hostname does not resolve | No DNS record, or the record points elsewhere | Point it at your ingress controller |
| The browser warns about the certificate | The certificate does not cover the hostname, or is from a CA the browser does not trust | Use a certificate covering every hostname in ingress.tls |
| On OpenShift, the hostname leads nowhere | The ingress entry has no secretName, so no Route was created |
Set secretName in ingress.tls |
| The page loads but stays blank, or every action fails | The browser used an address other than the configured hostnames | Open the UI at exactly the ingress.consoleHost or ingress.opsHost hostname |
| Sign-in fails | Wrong username or password | Read the passwords from the Secret, as in Verify the Deployment |
Collect Event Broker Service Diagnostics¶
Solace support may ask for an event broker service's diagnostics. Run the event broker service's own gather-diagnostics command in its CLI, inside the event broker service pod:
kubectl exec -it -n <datacenter namespace> <broker pod> -c solace -- cli
At the CLI prompt:
enable
admin
gather-diagnostics
The command prints the name of the file it writes. Copy that file out of the pod:
kubectl cp -n <datacenter namespace> -c solace <broker pod>:<file path> ./<file name>
An event broker service pod's name starts with the service's name. The event broker's documentation describes gather-diagnostics and its options.
Get Support¶
When you contact Solace, include:
- the chart version:
helm list -n $NS - the pod list and recent events:
kubectl get pods -n $NS -o wideandkubectl get events -n $NS --sort-by=.lastTimestamp - the control plane's log, and the datacenter agent's log for event broker service problems
- the registration job's log, if it failed
- your values files with every password, key and token removed