Build phase 2Building

Mesh + Observability

Istio for mTLS, traffic management, and mesh telemetry; Prometheus and Grafana for the monitoring foundation everything after this must plug into.

This is a deliberate complexity layer. A service mesh on a single-developer cluster is arguably overkill, which is exactly the point. It closes a real skill gap in most DevOps portfolios and signals Staff-level systems thinking, provided the reasoning is documented.

Service mesh: Istio

Istio installed via Argo CD (Helm chart or operator).
STRICT mTLS enforced across all namespaces, verifiable via Kiali or istioctl.
A canary / traffic-split demo over two tagged versions of crashdummy, driven by a VirtualService + DestinationRule.
Kiali deployed for mesh visualization.

Observability stack

Prometheus via kube-prometheus-stack: Operator, Alertmanager, and default dashboards, with persistent storage on Civo volumes.
Grafana dashboards: cluster health, Argo CD sync status, and Istio mesh telemetry.
Alerting rules for node health, pod crash loops, and resource saturation, each fired on demand via crashdummy fault injection.
ServiceMonitor CRDs for every deployed component.

Observable from here on

Phase 2 sets the rule for everything that follows: if a component is deployed, it exposes metrics and lands on a dashboard. The AI agents in later phases inherit this: their /metrics endpoints are scraped the moment they exist.

Tasks

MESH-001MESH-002MESH-003MESH-004MESH-005MESH-006OBS-001OBS-002OBS-003OBS-004OBS-005OBS-006OBS-007OBS-008

Write-ups

“Prometheus + Grafana + Istio on K3s: full observability for a portfolio cluster.”
“Why I added Istio to a single-developer project (and what it taught me about service mesh).”