Skip to content

Monitoring Your Deployment

Sovereign Home Cloud runs as ordinary Kubernetes workloads, so monitor it with the tools you already use for your cluster: your Kubernetes monitoring agent, your log pipeline, and your alerting. This document covers what to watch, the health checks the control plane provides, and how to read its logs.

Note: Sovereign Home Cloud or the event broker services you deploy do not send metrics or logs outside your cluster.

What to Watch

Workload Kind Healthy when
home-cloud-control-plane-app Deployment 1/1 ready, no restarts
home-cloud-control-plane-postgresql, -opensearch, -valkey, -seaweedfs StatefulSets, when bundled 1/1 ready
home-cloud-control-plane-broker StatefulSet, when the in-cluster event broker is used 1/1 ready
home-dc, and each added datacenter's agent Deployment 1/1 ready
Each event broker StatefulSet in its datacenter's namespace, named after the service Ready
Persistent volumes — Below about 80% full; nothing deletes old data automatically

Alert when a pod is not ready for more than a few minutes, restarts repeatedly, or a volume fills up. The audit log in OpenSearch and the object store grow without a limit, so watch their volumes in particular.

Health Endpoints

The control plane runs all its services in one pod, and reports on them through a small health interface. Kubernetes already uses it:

Probe Checks Effect when it fails
Startup The API answers. Allows up to 20 minutes for the first start. The pod is restarted
Readiness Every service has started and none reports DOWN The pod is taken out of service until it recovers
Liveness The API port accepts connections The pod is restarted

To see each service's state yourself, ask from inside the pod. The interface listens only on the pod's loopback address and requires the pod's control token:

kubectl exec -n home-cloud-control-plane deploy/home-cloud-control-plane-app -c home-cloud-control-plane -- \
  sh -c 'curl -s -H "Authorization: Bearer $HOSTED_FLEET_CONTROL_TOKEN" http://127.0.0.1:$COMPOSITE_PORT/_composite/services'
Path Returns
/_composite/ready 200 when every service has started and is up, 503 otherwise
/_composite/health Overall health: 200 when UP, 503 otherwise
/_composite/services Each service with its status
/_composite/services/<service>/health One service's health

Logs

The control plane writes one JSON object per line to standard output, so any Kubernetes log collector picks it up without configuration.

Field Meaning
@timestamp When the event happened
level ERROR, WARN, INFO, DEBUG
hostedFleetService Which service wrote the line, or host for the control plane itself
logger_name, thread_name Where in the service
message The message
stack_trace Present when an error carries one
X-B3-TraceId Present on lines that belong to a request; the same value across every service the request touched

A few lines, such as Java runtime warnings, are plain text. Filter on lines that start with { before parsing:

kubectl logs -n home-cloud-control-plane deploy/home-cloud-control-plane-app -c home-cloud-control-plane \
  | grep '^{' | jq -r 'select(.level=="ERROR") | "\(.hostedFleetService) \(.message)"'

Other logs worth collecting:

  • The datacenter agent: kubectl logs -n <namespace> deploy/<datacenter id> -c solace-cloud-ca. It records every event broker service creation, upgrade, and deletion.
  • The registration job: kubectl logs -n home-cloud-control-plane job/home-cloud-control-plane-datacenter-register. It runs on installation and upgrade, and deletes itself when it succeeds. A failed run is kept so that its log stays available.
  • Initialization steps of the control plane pod, which prepare the data stores at each start: kubectl logs -n home-cloud-control-plane deploy/home-cloud-control-plane-app -c <step>, where the step is postgres-bootstrap, object-store-bootstrap or broker-bootstrap.

Event Broker Services

Each event broker service is a Solace event broker and can be monitored like one: through its own management interface, Broker Manager, from the Service Details page for the event broker service in the Cloud Console, or through its SEMP API, together with your Kubernetes monitoring of its pods and volumes.

First-Week Checklist

  1. All workloads in What to Watch are ready, and your monitoring alerts when one is not.
  2. Control plane logs reach your log pipeline as JSON, and you can filter them by hostedFleetService and level.
  3. Volume usage is tracked, with an alert before a volume fills.
  4. A scheduled backup has run, and you have tried a restore; see Backup and Restore.
  5. Certificate expiry dates are tracked: the ingress certificate, and the event broker CA or certificate if you provided one.

Next Steps