Codified Observability — Layer 2: Platforms

This is part 3 of a series based on my DevOpsCon San Diego 2026 talk. Part 2 baked Alloy into the base image. This post is Layer 2 — Platforms: the observability stack itself, managed as code.

The code lives in 2-LGTM/ of the devopscon-2026 lab.

Is Your Observability Stack Observable?

When the stack was not built with an IaC tool, the infrastructure under it is hard to track. Changes live in tickets and tribal knowledge, not git history. After an incident that forces a rebuild, you reconstruct from documentation — if the docs are still right.

Put differently: the tooling that watches everything is itself unwatched. Provisioned by hand. One-off changes. Recreated from scratch when something breaks.

Goal: Codify the O11y Platform

• Platform defined as Terraform resources and Nomad jobspecs
• Changes reviewed before apply
• Fast recovery and simpler failover — the same definition redeploys prod and DR
• State tracked so drift is visible
• Same workflow as application infrastructure
• Lock down what Terraform (or Nomad) manages so click-ops cannot silently diverge

Nomad Task Drivers

Without going deep on Nomad: it runs containerized and non-containerized workloads through task drivers.

The lab mixes both. Grafana Alloy runs with raw_exec so it can see host and Docker logs on the node. Loki, Tempo, Mimir, Grafana, and Traefik run as Docker tasks. That split matters when you care about host metrics and /var/lib/docker/containers logs without stuffing privileged containers everywhere.

Stack Architecture

Dedicated or shared Nomad clients run the stack. In production you would spread instances across availability zones. Traefik sits in front with Nomad service discovery. Agents push logs, traces, and metrics through that front door; users hit Grafana the same way.

Provisioning the Platform

In the talk, platform changes follow the same merge-request path as everything else:

Merge request → review → merge → Nomad pulls secrets and runs the Loki, Grafana, Mimir, and Tempo jobs on Nomad clients.

The lab short-circuits CI for the demo and deploys jobspecs with 2-LGTM/run.sh:

cd 2-LGTM
./run.sh

Order matters: Traefik first so host-based routes exist, then Loki, Tempo, Mimir, Grafana, Alloy.

Traefik Discovers Nomad Services

Traefik is configured with the Nomad provider so registered services become routes from tags:

task "traefik" {
  driver = "docker"

  config {
    image = "traefik:v3.1"
    ports = ["http", "dashboard"]
    args = [
      "--api.dashboard=true",
      "--api.insecure=true",
      "--log.level=INFO",
      "--entrypoints.web.address=:80",
      "--providers.nomad=true",
      "--providers.nomad.endpoint.address=http://${attr.unique.network.ip-address}:4646",
      "--providers.nomad.refreshInterval=30s",
    ]
  }
}

Each LGTM job registers with Nomad and tags itself for Traefik. Loki, for example:

service {
  name     = "loki"
  port     = "http"
  provider = "nomad"

  tags = [
    "traefik.enable=true",
    "traefik.http.routers.loki.rule=Host(`loki.localhost`)",
    "traefik.http.services.loki.loadbalancer.server.port=3100",
  ]
}

No separate ingress YAML to keep in sync with the job — the job is the routing declaration.

Config as Nomad Templates

Loki’s config is not a hand-edited file on disk. It is a template block in the jobspec, rendered into the allocation:

template {
  destination = "local/loki.yaml"
  data = <<-EOF
    auth_enabled: false

    server:
      http_listen_port: 3100
      grpc_listen_port: 9096

    common:
      path_prefix: /alloc/data/loki
      replication_factor: 1
      ring:
        kvstore:
          store: inmemory
    # ... schema_config, storage_config, limits_config, compactor ...
  EOF
}

Grafana’s datasources are the same pattern — provisioned YAML templated with Nomad’s allocation IP so Loki, Mimir, and Tempo resolve on the single-node lab:

template {
  destination = "local/provisioning/datasources.yaml"
  data = <<-EOF
    apiVersion: 1
    datasources:
      - name: Loki
        uid: loki
        type: loki
        access: proxy
        url: http://{{ env "NOMAD_IP_http" }}:3100
        isDefault: true
      - name: Mimir
        uid: mimir
        type: prometheus
        access: proxy
        url: http://{{ env "NOMAD_IP_http" }}:9009/prometheus
      - name: Tempo
        uid: tempo
        type: tempo
        access: proxy
        url: http://{{ env "NOMAD_IP_http" }}:3200
  EOF
}

Same workflow as application jobs: change the jobspec, review it, redeploy. The platform stops being a snowflake.

Vault Around the Platform

Again, the Multipass lab does not require Vault. In production-shaped talk material, the same Vault instance shows up at every layer:

• GitLab ↔ Vault — workload identity for CI
• Nomad ↔ Vault — workload identity for jobs
• Terraform — reads and creates secrets during plan/apply
• Traefik — authentication layer in front of LGTM
• LGTM ↔ Vault PKI — certificate management and rotation

Workload identity means auth is about what is running and where, not a shared long-lived token copied into six jobspecs.

Conclusion

Layer 2 makes the platform as boring (in a good way) as the apps it watches:

1. Define Traefik and LGTM as Nomad jobs, not click-installed VMs.
2. Use the right task driver — raw_exec for host-visible Alloy, Docker for the rest of the stack.
3. Put routing tags and config templates in the jobspec so git is the source of truth.
4. Plan for Vault workload identity and PKI even if your first lab skips them.

Next: Layer 3 — Provisioning, where dashboards and alerts ship in the same Terraform apply as the service.

Series

1. Codified Observability: Making Monitoring Unavoidable
2. Layer 1 — Images
3. Layer 2 — Platforms (this post)
4. Layer 3 — Provisioning
5. Layer 4 — Orchestration
6. Key Takeaways

Lab: https://gitlab.com/lykins/devopscon-2026