Codified Observability — Layer 4: Orchestration
This is part 5 of a series based on my DevOpsCon San Diego 2026 talk. Part 4 co-deployed dashboards with Terraform. This post is Layer 4 — Orchestration: keeping agents dynamic on day 2 instead of treating them like frozen packages on disk.
The Alloy job lives in 2-LGTM/jobs/lgtm/alloy.nomad.hcl. It is the payoff for Layer 1 installing Alloy but leaving systemd disabled.
Day 2 Is Where TOIL Lives
Agents manually deployed and manually updated. Agent configuration static. Secrets hardcoded on the host. Fleet-wide version bumps become a project. Missed updates become tomorrow’s incident.
Bump Alloy’s version or config across hundreds or thousands of instances and even a scripted Ansible pass is still heavy. Static host config means every change is human-shaped. Secrets baked into files mean rotation is redeploy-the-world.
Goal: Dynamic Configuration
- Maintain agents with Nomad, not systemd, as the control plane
- Update configuration from service or secret changes without a manual fleet push
- Use Nomad Packs or Terraform for templated workloads when you need reuse
- One control plane for observability agents across the fleet
Alloy as a Nomad System Job
type = "system" places the job on every Nomad client. Combined with raw_exec, Alloy runs on the host and can read host and Docker log paths:
job "alloy" {
datacenters = ["dc1"]
type = "system"
group "alloy" {
count = 1
network {
port "http" {
static = 12345
to = 12345
}
}
task "alloy" {
driver = "raw_exec"
config {
command = "bash"
args = [
"-lc",
"/usr/bin/alloy run ${NOMAD_TASK_DIR}/config.alloy --server.http.listen-addr=0.0.0.0:${NOMAD_PORT_http}",
]
}
}
}
}
The binary came from the image in Layer 1. Nomad owns whether it is running, which config file it loads, and how updates roll out. On Nomad 1.11+, system job deployments give you canaries and rollbacks for destructive updates — the same deployment vocabulary you already use for service jobs.
Config Lives in a Template Block
The Alloy config is not a snowflake under /etc/alloy. It is rendered by Nomad:
template {
destination = "local/config.alloy"
data = <<-EOF
logging {
level = "info"
format = "logfmt"
}
discovery.relabel "local_logs" {
targets = [{
__path__ = "/var/log/*.log",
job = "host-logs",
host = env("node_name"),
}]
}
discovery.relabel "docker_logs" {
targets = [{
__path__ = "/var/lib/docker/containers/*/*-json.log",
job = "docker-logs",
host = env("node_name"),
}]
}
local.file_match "logs" {
path_targets = concat(
discovery.relabel.local_logs.output,
discovery.relabel.docker_logs.output,
)
}
loki.source.file "logs" {
targets = local.file_match.logs.targets
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://{{ env "NOMAD_IP_http" }}:3100/loki/api/v1/push"
}
}
EOF
}
Change the jobspec, redeploy, Nomad restarts or rolls the allocation. Need to roll back? Redeploy the previous jobspec. That is a different operational posture than editing units on every host.
Metrics Alongside Logs
The same template scrapes node metrics and the platform services from Layer 2, then remote-writes to Mimir:
prometheus.exporter.unix "node" {
include_exporter_metrics = true
}
prometheus.scrape "node" {
targets = prometheus.exporter.unix.node.targets
forward_to = [prometheus.remote_write.default.receiver]
}
prometheus.scrape "platform_services" {
targets = [
{ __address__ = "{{ env "NOMAD_IP_http" }}:3000", job = "grafana" },
{ __address__ = "{{ env "NOMAD_IP_http" }}:3100", job = "loki" },
{ __address__ = "{{ env "NOMAD_IP_http" }}:3200", job = "tempo" },
{ __address__ = "{{ env "NOMAD_IP_http" }}:9009", job = "mimir" },
]
forward_to = [prometheus.remote_write.default.receiver]
}
prometheus.remote_write "default" {
endpoint {
url = "http://{{ env "NOMAD_IP_http" }}:9009/api/v1/push"
}
}
The stack that watches the apps also watches itself — and the scrape list is reviewed in git with everything else.
Secrets Without Redeploying by Hand
Vault shows up again at the orchestration layer. Nomad integrates natively; Alloy can pull from Vault directly.
In Alloy:
# Within Grafana Alloy configuration, secrets can be pulled from Vault.
remote.vault "<LABEL>" {
server = "<VAULT_SERVER>"
path = "<VAULT_PATH>"
key = "<VAULT_KEY>"
}
In Nomad, a template that reads a secret can restart the task when the value changes (the default behavior):
# Nomad has multiple behaviors for secret changes.
# Default: restart the task with the new secret value.
template {
data = <<EOF
KEY={{ with secret "secret/data/aws/" }}{{ .Data.data.key }}{{ end }}
EOF
}
Same Vault instance as Layers 1–3. Workload identity for auth. Certificates and secrets rotate without a human SSHing to every node.
The Multipass lab does not stand up Vault — the snippets above are the production-shaped pieces from the talk. The runnable part is still the system job and the template-driven config.alloy.
Deploying Alloy in the Lab
cd 2-LGTM
./run.sh
# or just:
multipass exec devopscon -- nomad job run - < jobs/lgtm/alloy.nomad.hcl
run.sh deploys Alloy last, after Loki and Mimir exist as push targets. Pair that with the Layer 3 logger job and the demo dashboard lights up.
Conclusion
Layer 4 is where “baked into the image” stops meaning “frozen forever”:
- Install the agent in the image; run it under Nomad as a system job.
- Keep Alloy config in a Nomad
templateso changes are reviewed and rolled. - Scrape the platform itself — do not leave LGTM as a blind spot.
- Use Vault + Nomad templates (or Alloy
remote.vault) so secret rotation is not a fleet exercise.
Next: Key Takeaways — the four layers as one strategy, what this eliminates, and how to adopt without boiling the ocean.
Series
- Codified Observability: Making Monitoring Unavoidable
- Layer 1 — Images
- Layer 2 — Platforms
- Layer 3 — Provisioning
- Layer 4 — Orchestration (this post)
- Key Takeaways