07 — Cluster Dashboard

Architecture

Browser → OVH nginx → proxy_pass http://10.99.101.206
    │
    ▼
MetalLB 10.99.101.206 (externalTrafficPolicy: Cluster)
    │
    ▼
Service: dashboard-nginx (cluster-dashboard namespace)
    │
    ▼
Pod: nginx:1.27-alpine
    serves /data/index.html + /data/status.json
    │
    ▼
NFS PV: 192.168.0.165:/srv/nfs/dashboard
    ▲
    │ writes status.json every ~30s
CronJob: cluster-probe (bitnami/kubectl:latest)
    └── ServiceAccount: dashboard-probe
        ClusterRole: read-only (get/list nodes,pods,deployments...)

Lessons learned — quick reference

Issue	Wrong	Correct
Storage	hostPath on one node	NFS — any node can read/write
Pod placement	nodeSelector: kube2	No nodeSelector
MetalLB annotation	metallb.universe.tf/loadBalancerIPs	metallb.io/loadBalancerIPs
MetalLB traffic policy	Local (IP never bound)	Cluster
Probe image	bitnami/kubectl:1.29 (pull failed)	bitnami/kubectl:latest
status.json path	/status.json	/cluster/status.json
Directory perms	777 (lazy)	chown 1001:0 chmod 755
CronJob stuck	concurrencyPolicy: Forbid + bad image	Delete the bad job manually
CSF firewall	BOGON blocks MetalLB IPs	LF_BOGON_SKIP = "wg0"
WireGuard route	scope link fails in scripts	No scope link in rc.local

RBAC — why it matters

The probe pod needs to call the Kubernetes API to collect cluster state. Instead of using the default ServiceAccount (which has no permissions) or admin credentials (which is dangerous), we create a minimal read-only ClusterRole:

# What the probe CAN do
kubectl auth can-i list pods \
  --as=system:serviceaccount:cluster-dashboard:dashboard-probe
# yes

# What the probe CANNOT do
kubectl auth can-i delete pods \
  --as=system:serviceaccount:cluster-dashboard:dashboard-probe
# no

Principle: The probe only needs get and list on nodes, pods, deployments, namespaces, services, statefulsets, cronjobs. Never create, delete, or patch. Least privilege — if the probe pod is compromised, the blast radius is read-only.

NFS vs hostPath — why NFS wins

The first version used a hostPath PersistentVolume pointing at /opt/cluster-dashboard-data on kube2, with a nodeSelector forcing both the nginx pod and the CronJob to land on kube2.

⚠ Problem: hostPath means data lives on ONE node's disk. If the pod moves to kube3, it finds an empty directory. That's no better than the old single-VPS setup — one node failure takes everything down.

✓ Solution: NFS PV at 192.168.0.165:/srv/nfs/dashboard. Any node can mount it simultaneously (ReadWriteMany). Pod schedules freely between kube2 and kube3. No nodeSelector needed.

# NFS setup on 192.168.0.165
mkdir -p /srv/nfs/dashboard
chown 1001:0 /srv/nfs/dashboard   # probe runs as uid 1001
chmod 755 /srv/nfs/dashboard
echo "/srv/nfs/dashboard 192.168.0.0/24(rw,sync,no_subtree_check,no_root_squash)" \
  >> /etc/exports
exportfs -ra

storageClassName: "" — when creating a static NFS PV and binding it directly by volumeName, set storageClassName: "" on both PV and PVC. Without this, Kubernetes tries to match StorageClasses and fails with "storageClassName does not match" if multiple classes exist (local-path, longhorn, nfs-client).

CronJob — 30 second refresh trick

Kubernetes CronJob minimum schedule is 1 minute (* * * * *). To get ~30 second data freshness, the probe script runs twice per Job with a sleep in between:

# In the CronJob container args:
- |
  /scripts/probe.sh
  sleep 30
  /scripts/probe.sh

This means each Job runs for ~64 seconds total, and a new Job fires every 60 seconds — giving an effective refresh of ~30 seconds. The concurrencyPolicy: Forbid ensures only one Job runs at a time.

⚠ Gotcha — stuck CronJob: If a Job gets stuck (e.g. ImagePullBackOff), concurrencyPolicy: Forbid blocks all future Jobs. Fix: kubectl delete job <stuck-job> -n cluster-dashboard. The next scheduled Job fires within 60 seconds.

MetalLB — IP not announced after creation

After creating the LoadBalancer Service, the MetalLB speaker logs showed nothing for the new IP. The IP was assigned by the controller but never announced via ARP — making it completely unreachable.

⚠ Root cause: externalTrafficPolicy: Local caused MetalLB healthcheck failures. With Local policy, MetalLB only announces from nodes that are running the pod AND pass the healthcheck nodeport. The healthcheck was failing, so no announcement was made.

# Fix — patch to Cluster policy
kubectl patch svc dashboard-nginx -n cluster-dashboard \
  -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'

# If IP still not announced after that — delete and recreate the Service
kubectl delete svc dashboard-nginx -n cluster-dashboard
kubectl apply -f 07-service.yaml

# Verify announcement in speaker logs
kubectl logs -n metallb-system -l component=speaker | grep "10.99.101.206"

Operational commands

# Check everything in the namespace
kubectl get all -n cluster-dashboard

# Watch pods in real time
kubectl get pods -n cluster-dashboard -w

# Check which node the pod is on
kubectl get pods -n cluster-dashboard -o wide

# Force an immediate probe run
kubectl create job --from=cronjob/cluster-probe \
  manual-$(date +%s) -n cluster-dashboard

# Tail probe logs
kubectl logs -n cluster-dashboard \
  -l app.kubernetes.io/component=probe -f

# Check status.json is being written
ls -la /srv/nfs/dashboard/status.json

# Peek at the data
cat /srv/nfs/dashboard/status.json | jq .summary

# Test endpoint directly from OVH
curl -s http://10.99.101.206/status.json | jq .summary

# Restart nginx pod
kubectl rollout restart deployment/dashboard-nginx \
  -n cluster-dashboard