Observability on Lykins Blog

Codified Observability — Key Takeaways

Sat, 06 Jun 2026 00:00:00 +0000

This is the final part of a series based on my DevOpsCon San Diego 2026 talk. The earlier posts walked each layer with the devopscon-2026 lab. This post is the map folded back up: one strategy, what it removes, and how to start without doing everything at once.

Four Layers as a Unified Strategy

Layer	Outcome
Images	Agent and config strategy baked in — observable at first boot
Platforms	Observability stack managed as code — repeatable and auditable
Provisioning	Dashboards and alerts co-deployed with services
Orchestration	Runtime observability — no manual steps for day-2 agent changes

None of these is a silver bullet alone. Images without a managed platform still leave you rebuilding Grafana by hand. A beautiful LGTM deploy without co-provisioned dashboards still leaves services dark until someone clicks around. Orchestration without an image that already has the binary still starts from “install the agent.” The value compounds.

Codified Observability — Layer 4: Orchestration

Fri, 05 Jun 2026 00:00:00 +0000

This is part 5 of a series based on my DevOpsCon San Diego 2026 talk. Part 4 co-deployed dashboards with Terraform. This post is Layer 4 — Orchestration: keeping agents dynamic on day 2 instead of treating them like frozen packages on disk.

The Alloy job lives in 2-LGTM/jobs/lgtm/alloy.nomad.hcl. It is the payoff for Layer 1 installing Alloy but leaving systemd disabled.

Day 2 Is Where TOIL Lives

Agents manually deployed and manually updated. Agent configuration static. Secrets hardcoded on the host. Fleet-wide version bumps become a project. Missed updates become tomorrow’s incident.

Codified Observability — Layer 3: Provisioning

Thu, 04 Jun 2026 00:00:00 +0000

This is part 4 of a series based on my DevOpsCon San Diego 2026 talk. Part 3 put the LGTM stack on Nomad. This post is Layer 3 — Provisioning: dashboards and alerts that ship with the service, not weeks later.

The code lives in 3-terraform/ of the devopscon-2026 lab.

The Secondary Step

Infrastructure deploys. The dashboard gets added later. Alert thresholds are set by gut feel and never reviewed. The dashboard only lives in Grafana — not in git. Observability config has no ownership and no history.

Codified Observability — Layer 2: Platforms

Wed, 03 Jun 2026 00:00:00 +0000

This is part 3 of a series based on my DevOpsCon San Diego 2026 talk. Part 2 baked Alloy into the base image. This post is Layer 2 — Platforms: the observability stack itself, managed as code.

The code lives in 2-LGTM/ of the devopscon-2026 lab.

Is Your Observability Stack Observable?

When the stack was not built with an IaC tool, the infrastructure under it is hard to track. Changes live in tickets and tribal knowledge, not git history. After an incident that forces a rebuild, you reconstruct from documentation — if the docs are still right.

Codified Observability — Layer 1: Images

Tue, 02 Jun 2026 00:00:00 +0000

This is part 2 of a series based on my DevOpsCon San Diego 2026 talk. Part 1 covered why reactive observability fails and laid out the four layers. This post is Layer 1 — Images.

The code lives in 1-images/ of the devopscon-2026 lab.

The Dark Launch

New VM deployed — no agent installed. Someone installs and configures it by hand after provisioning. The config is undocumented or unrepeatable. The service is live before anyone can see it.

Codified Observability: Making Monitoring Unavoidable

Mon, 01 Jun 2026 00:00:00 +0000

This series is based on a talk I gave at DevOpsCon San Diego 2026. The submitted title was a mouthful — Simplifying and Scaling Observability for Non-Enterprise Teams — but if I only had two words for it, they would be codified observability. The goal is simple: no click-ops, no manual configuration in your pipelines, and observability as the default for every component in the environment.

The full lab that backs the talk lives in devopscon-2026. Parts 2 through 5 walk each layer with that code. This post is the why and the map.