Observability Runbook

Offline K8s Metrics Baseline

The Go services expose Prometheus text metrics on /metrics. The offline kind baseline deploys Prometheus inside the hoctapaz-go namespace and scrapes every service listed in scripts/dev/services.sh.

Before going offline, make sure the pinned observability image exists locally:

docker pull prom/prometheus:v2.53.0

After building service images, load both service and observability images:

make kind-load-images KIND_CLUSTER=hoctapaz-go IMAGE_TAG=local

Deploy the stack:

make k8s-deploy IMAGE_TAG=local

Inspect targets:

kubectl -n hoctapaz-go port-forward svc/prometheus 9090:9090

Open http://localhost:9090/targets and check the hoctapaz-go-services job.

Coverage Check

Before deploying, verify that the Prometheus scrape config includes every local Go service:

scripts/test/k8s-observability-coverage.sh

The check fails if a service in scripts/dev/services.sh is missing from the Prometheus target list or if the manifest does not enforce local image usage.