Codified Observability: Making Monitoring Unavoidable
This series is based on a talk I gave at DevOpsCon San Diego 2026. The submitted title was a mouthful — Simplifying and Scaling Observability for Non-Enterprise Teams — but if I only had two words for it, they would be codified observability. The goal is simple: no click-ops, no manual configuration in your pipelines, and observability as the default for every component in the environment.
The full lab that backs the talk lives in devopscon-2026. Parts 2 through 5 walk each layer with that code. This post is the why and the map.
The Problem with Reactive Observability
A pattern I have seen in previous roles and still see today: observability is not part of infrastructure provisioning or service deployment. Infrastructure and services ship, then ownership is handed off to an operations or reliability team to “add monitoring later.”
What that usually looks like in practice:
- Monitoring is configured manually — after go-live
- Agents are installed by hand and drift by environment
- Observability is optional. Until it isn’t.
When observability finally comes into focus, it is usually after an incident. Teams reactively chase the root cause, then bolt monitoring onto the thing that just burned them. That is the opposite of making monitoring unavoidable.
What If Observability Was Unavoidable?
No checklists. No runbook step that someone has to remember. Observability as a property of the system itself — baked into images, managed as code for the platform, co-deployed with services, and kept dynamic at runtime.
In earlier talks I have called this observability by default: pipelines and standardization so that any resource, especially anything in production, is centrally monitored without a separate project to “turn on o11y.”
Four Layers
There are four layers you can implement to get there. You do not have to do all four at once — more on that in part 6 — but each one removes a whole class of human error.
| Layer | Focus | Idea |
|---|---|---|
| 01 Images | Agents baked in at build time | Every VM is observable the moment it boots |
| 02 Platforms | The observability stack managed as code | The tooling that watches everything is itself repeatable |
| 03 Provisioning | Monitoring and service shipped together | Dashboards and alerts live in the same repo as the service |
| 04 Orchestration | Observability at runtime | Agents stay dynamic — no fleet-wide manual updates |
The rest of the series is one post per layer, then a wrap-up.
Non-Enterprise on Purpose
Everything in this talk can be run without a product trial, without an email signup for a SaaS tier, and on your own hardware. That constraint matters for small teams and for anyone who wants to prove the pattern in a lab before proposing it at work.
The Toolset
The demos lean on the Hashi stack, but the ideas map to most clouds and schedulers:
- GitLab — CI/CD (and in production, local runners)
- Grafana LGTM + Alloy — logs, metrics, traces, and the agent
- Packer — image building
- Terraform — infrastructure and Grafana resources as code
- Nomad — workload scheduling
- Vault — centralized secrets
One honest gap between the slides and the lab: the deck shows GitLab CI and Vault at every layer. The runnable repo is Multipass-local — no GitLab pipeline wired up, no Vault instance required. In each post I will show the working demo code first, then call out the Vault/GitLab snippets from the talk as the place those pieces plug in.
The Lab
If you want to follow along as you read, the repo is ordered to match the layers:
0-multipass-setup/ # Ubuntu VM with Nomad, Docker, Alloy
1-images/ # Packer Debian 13 base image
2-LGTM/ # Traefik + Loki + Tempo + Mimir + Grafana + Alloy
3-terraform/ # Grafana dashboards + demo log generator
From the repo root:
./run-all.sh
Or step through individually. Step 0 bootstraps a Multipass VM named devopscon with Nomad (server + client), Docker, and Grafana Alloy installed but not left as the long-term control plane for the agent — that detail becomes important in part 5.
cd 0-multipass-setup
./run.sh
multipass shell devopscon
What’s Next
Part 2 starts at Layer 1: baking the agent into the base image so there is no dark launch — and no undocumented install step after the VM is already in service.
Series
- Codified Observability: Making Monitoring Unavoidable (this post)
- Layer 1 — Images
- Layer 2 — Platforms
- Layer 3 — Provisioning
- Layer 4 — Orchestration
- Key Takeaways