Codified Observability — Layer 1: Images
This is part 2 of a series based on my DevOpsCon San Diego 2026 talk. Part 1 covered why reactive observability fails and laid out the four layers. This post is Layer 1 — Images.
The code lives in 1-images/ of the devopscon-2026 lab.
The Dark Launch
New VM deployed — no agent installed. Someone installs and configures it by hand after provisioning. The config is undocumented or unrepeatable. The service is live before anyone can see it.
That pattern shows up in small shops and large ones. A common failure mode is boring: a disk that was never monitored fills up and takes the instance down. The machine came from a default image or ISO, got handed to another team for “further setup,” and the agent install was one more manual step that did not always happen.
Goal: Bake Observability into Base Images
Agents and an initial config belong in the image, not in the deploy checklist.
- Image builds are stored in git; changes are reviewed before merge
- A pipeline builds on approved merges
- Every VM is observable the moment it boots
- Specialized images can layer on top of the base
Image Build Architecture
In a fuller setup than the Multipass lab, the flow looks like this:
Changes land in GitLab. A GitLab runner scheduled on Nomad pulls the job and runs Packer. Packer builds from an ISO in a sandbox (QEMU in this talk). When the build finishes, the image artifact — and extras like an SBOM — upload back to GitLab. Vault sits beside the runner so secrets for the build are pulled centrally instead of living in CI variables forever.
Image Factory
A base image is not the end of the story. Treat it as the root of a factory:
Start from an ISO. Branch into hardened images for FedRAMP or PCI when you need them. Publish platform-specific bases into Azure, AWS, Google, or a private cloud. From those bases, build role-specific images — containers, web servers, and so on. At each layer you can tighten what gets monitored for that class of instance.
The lab keeps this to one layer: a Debian 13 base with Alloy installed. The factory diagram is the shape you grow into.
The Packer Template
The template in 1-images/base/debian13.pkr.hcl builds with the QEMU plugin from a Debian netinst ISO and a preseed:
source "qemu" "debian13" {
accelerator = "tcg"
boot_wait = "5s"
communicator = "ssh"
cpus = var.cpus
disk_interface = "virtio"
disk_size = "8192"
format = "qcow2"
http_directory = "http"
iso_checksum = var.iso_checksum
iso_url = var.iso_url
memory = var.memory
net_device = "virtio-net"
output_directory = "output/${var.vm_name}"
shutdown_command = "echo '${var.ssh_password}' | sudo -S shutdown -P now"
ssh_password = var.ssh_password
ssh_timeout = "120m"
ssh_username = var.ssh_username
vm_name = "${var.vm_name}.qcow2"
boot_command = [
"<wait5><esc><wait>",
"install auto=true auto-install/enable=true priority=critical ",
"preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg ",
"debconf/frontend=noninteractive quiet --- <enter>"
]
}
Unattended install via http/preseed.cfg, SSH in as packer, then provision.
Installing Alloy (and Leaving systemd Alone)
The first provisioner runs scripts/alloy.sh:
#!/usr/bin/env bash
set -euxo pipefail
sudo DEBIAN_FRONTEND=noninteractive apt-get -y install gpg
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get -y install alloy
# Do not enable the service — we want to avoid starting it during provisioning.
That last comment is load-bearing. Alloy is on disk in the image, but the systemd unit is not the long-term control plane. Part 5 runs Alloy as a Nomad system job so config and lifecycle stay under the orchestrator. Installing at image build time still means the binary is present at first boot — you just do not commit to a static unit file forever.
SBOM and Manifest
The same build also produces a CycloneDX SBOM with Mondoo’s cnquery, downloads it out of the guest, and writes a Packer manifest:
provisioner "shell" {
execute_command = "echo '${var.ssh_password}' | {{ .Vars }} sudo -S -E bash '{{ .Path }}'"
inline = [
"apt-get update -y",
"apt-get install -y curl gpg",
"bash -c \"$(curl -sSL https://install.mondoo.com/sh)\"",
"cnquery sbom --output cyclonedx-json --output-target /tmp/sbom_cyclonedx.json",
"chmod 644 /tmp/sbom_cyclonedx.json",
]
}
provisioner "file" {
source = "/tmp/sbom_cyclonedx.json"
destination = "output/sbom_cyclonedx.json"
direction = "download"
}
provisioner "shell" {
script = "scripts/cleanup.sh"
}
post-processor "manifest" {
output = "output/manifest.json"
}
Observable at boot is the headline. Knowing what was in the image when it was built is the quiet companion — especially once you are shipping many layers through the factory.
Build It
cd 1-images
./run.sh
That is a thin wrapper around packer init and packer build in 1-images/base/. Output lands under 1-images/base/output/.
Vault in the Image Build
The Multipass lab does not stand up Vault. In the talk, Packer’s built-in vault function is the reminder that build-time secrets should not live in a dozen CI variables:
# Leverages Vault as a single source for all secrets.
# Tip: not many people know Packer has a direct function to pull secrets from Vault.
locals {
api_key = vault("/shared/data/o11y", "token")
}
Same idea as the architecture diagram: the runner executes Packer, Vault supplies credentials, artifacts go back to GitLab.
Conclusion
Layer 1 removes the dark launch:
- Put the agent (and a sane default config strategy) in the base image.
- Review image changes in git the same way you review application code.
- Leave room for an image factory — base, compliance, cloud, role — instead of one golden AMI forever.
- Install the binary early; decide later whether systemd or the orchestrator owns the process. The lab chooses Nomad in Layer 4.
Next: Layer 2 — Platforms, where the LGTM stack itself becomes code on Nomad.