First Look Nomad 1.11.x - Client Node Introduction and Identity

Nomad 1.11.x was recently released and I wanted to highlight some of the new features in a series of posts.

Client node introduction and identity
Artifact secrets
System job deployments

For this first post, I want to focus on the client node introduction and identity. This will require reading through the release note documentation, updated pages, CLI and API references, as a lot has been added around this feature and I will not cover in detail.

Client Node Introduction#

This feature adds in an additional layer of security to prevent unauthorized client from joining a Nomad cluster. Prior to this the first measure of security was valid TLS certificates between the client and server. With this new feature, when strictly enforced, clients must have a valid token to join the cluster.

Think of this as multifactor authentication, but for your Nomad clusters.

Networking - can the client reach the server
TLS - does the client have valid certificates for the cluster
Client Introduction Token - does the client have a valid token to join the cluster

Although I do not cover it in detail, this gives an added benefit in additional control over misconfigured clients trying to join the cluster. You can specify node names, node pools, and TTLs for the tokens you generate.

Nomad Version#

This feature requires 1.11.x on both the client and server side to function.

nomad -v
Nomad v1.11.0
BuildDate 2025-11-11T16:18:19Z
Revision 9103d938133311b2da905858801f0e111a2df0a1

Server Configuration#

On the server side, a new configuration block has been added to the server configuration. This block is called client_introduction with configurable options.

On my setup, it is going to be very simple as this is a lab environment - so a bit of precaution it might not be up to production standards.

I set my enforcement to strict so that any client that does not have a valid token will be rejected from joining the cluster, instead of a warning.

data_dir = "/opt/nomad/"

acl {
  enabled = true
}

server {
  enabled          = true
  bootstrap_expect = 1
  client_introduction {
    enforcement          = "strict" #Default = "warn"
    default_identity_ttl = "5m"     #Default = "5m"
    max_identity_ttl     = "30m"    #Default = "30m"
  }
}

tls {
  http = true
  rpc  = true
  ca_file   = "/opt/nomad/tls/nomad-agent-ca.pem"
  cert_file = "/opt/nomad/tls/global-server-nomad.pem"
  key_file  = "/opt/nomad/tls/global-server-nomad-key.pem"
}

Starting up my Nomad server with this configuration:

==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 192.168.39.125:4646; RPC: 192.168.39.125:4647; Serf: 192.168.39.125:4648
            Bind Addrs: HTTP: [0.0.0.0:4646]; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
                Client: false
             Log Level: INFO
               Node Id: cfc63efc-0f2d-6e60-c27a-4124a9cf8137
                Region: global (DC: dc1)
                Server: true
               Version: 1.11.0

==> Nomad agent started! Log data will stream in below:
...
    2025-11-18T18:32:03.142Z [INFO]  nomad.raft: entering leader state: leader="Node at 192.168.39.125:4647 [Leader]"
    2025-11-18T18:32:03.144Z [INFO]  nomad: cluster leadership acquired
    2025-11-18T18:32:03.162Z [INFO]  nomad.core: established cluster id: cluster_id=bcdaffcb-a79a-d57c-03e4-6cf32357dee5 create_time=1763490723160485342
    2025-11-18T18:32:03.162Z [INFO]  nomad: eval broker status modified: paused=false
    2025-11-18T18:32:03.162Z [INFO]  nomad: blocked evals status modified: paused=false
    2025-11-18T18:32:03.237Z [INFO]  nomad.keyring: initialized keyring: id=9cd82611-3df2-1c73-d010-05be16d750e8

Pretty standard at this point, nothing new outside of the new configuration block. After starting up I bootstrapped the ACL system and pulled the initial management token.

Client Configuration#

On the client side, config will stay the same, no additional configuration is needed.Lets set up the Nomad client configuration. Which we should be good to join since we have valid certificates. Once we have the client config set up, we can start the Nomad agent.

data_dir = "/opt/nomad/"

 client {
   enabled = true
   servers = ["192.168.39.125"]
 }

tls {
  http = true
  rpc  = true
  ca_file   = "/opt/nomad/tls/nomad-agent-ca.pem"
  cert_file = "/opt/nomad/tls/global-client-nomad.pem"
  key_file  = "/opt/nomad/tls/global-client-nomad-key.pem"
}

Starting up the Nomad client agent:

==> Loaded configuration from /etc/nomad.d/nomad.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 192.168.39.133:4646
            Bind Addrs: HTTP: [0.0.0.0:4646]
                Client: true
             Log Level: INFO
                Region: global (DC: dc1)
                Server: false
               Version: 1.11.0

==> Nomad agent started! Log data will stream in below:
...
    2025-11-18T19:14:17.242Z [INFO]  client: started client: node_id=01f47f9a-7281-c450-6db0-3220fdc016a0
    2025-11-18T19:14:17.250Z [ERROR] client.rpc: error performing RPC to server: error="rpc error: Permission denied" rpc=Node.Register server=192.168.39.125:4647
    2025-11-18T19:14:17.251Z [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: Permission denied" rpc=Node.Register server=192.168.39.125:4647
    2025-11-18T19:14:17.253Z [ERROR] client: error registering: error="rpc error: Permission denied"

Hmm. That did not seem to work.

Monitoring Client Join Failures#

Before we dive into how to get the client to join, as platform owners we should be aware of when client nodes are failing to join, so lets go to the server logs.

2025-11-18T19:14:24.952Z [ERROR] nomad.client: node registration without introduction token: enforcement_level=strict node_id=01f47f9a-7281-c450-6db0-3220fdc016a0 node_pool=default node_name=client

From what we can see here, a node attempted to register without an introduction token. If the enforcement was warn, it would join, but it would log as a warning. Since we set it to strict, it is rejected from joining the cluster. For production enviornments, I would recommend after upgrading to 1.11.x to set enforcement to warn first, monitor logs for any unauthorized clients trying to join, and then once you have a repeatble workflow in place, switch to strict enforcement.

Also, if you have telemetry enabled, as you probably should, there are new metrics around client introductions.

How to Generate Client Introduction Tokens#

Before we generate our client introduction token, I just want to cover what I have set up for this lab environment:

Got a server
Got a client
Both running 1.11.x
Both with valid TLS certificates - not necessary for client introductions but I guess I’ve gotten use to always having TLS enabled.
ACL Enabled - please just always make this a habit on all clusters.
- node:write policy is required to generate tokens.
Server has client_introduction block configured.

So our next step is to generate a client introduction token. This can be done via the CLI or API. I will show both ways.

To do this I’m going to walk thorugh creating a policy, role, and then generating a token from that role. We will use that token to request a client introduction token when starting up the Nomad client agent.

Create ACL Policy#

# client-introduction.hcl
node {
  policy = "write"
}

nomad acl policy apply client-introduction client-introduction.hcl

Create ACL Role from Policy#

This is optional as you can generate tokens directly from policies, but I use roles to help manage my policies better.

nomad acl role create --tls-skip-verify -name="client-introduction" -policy="client-introduction"
ID           = cf0b4a43-b00f-cc30-b656-b34d66151b04
Name         = client-introduction
Description  = <none>
Policies     = client-introduction
Create Index = 117
Modify Index = 117

Generate Nomad Token via CLI#

Now we can generate a Nomad token from the role we just created. This token will be used to request client introduction tokens. Ecosystem wise I would recommend Vault to generate and manage these tokens, but for this lab we will just generate it via the CLI.

root@server:~# nomad acl token create -name="client-intro-token-1" -role-name="client-introduction" -type=client -ttl=8h
Accessor ID  = 8c22a7c0-44f5-044d-ef84-bfa06118faf4
Secret ID    = d99d678d-426c-330e-74f3-de53a868e2f9
Name         = client-intro-token-1
Type         = client
Global       = false
Create Time  = 2025-11-18 21:01:44.62075624 +0000 UTC
Expiry Time  = 2025-11-19 05:01:44.62075624 +0000 UTC
Create Index = 128
Modify Index = 128
Policies     = []

Roles
ID                                    Name
cf0b4a43-b00f-cc30-b656-b34d66151b04  client-introduction

Generate Nomad Client Introduction Token via API#

This feature introduces a new API endpoint to generate client introduction tokens.

Optional: You can create a JSON file to hold the request body, but in this case we are not any additional parameters so we can just send an empty JSON object.

Example payload file, just to show you can send additional parameters for added control such as NodeName, NodePool, and TTL.:

{
  "NodeName": "node-338ef6e9",
  "NodePool": "platform",
  "TTL": "15m"
}

Curl request to generate client introduction token - I am sending an empty JSON object as the body.:

curl \
    --request POST \
    --header "X-Nomad-Token: d99d678d-426c-330e-74f3-de53a868e2f9" \
    --data {} \
    https://localhost:4646/v1/acl/identity/client-introduction-token \
    >> intro_token.jwt

Response:

{
  "JWT": "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgyNjExLTNkZjItMWM3My1kMDEwLTA1YmUxNmQ3NTBlOCIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJub21hZHByb2plY3QuaW8iLCJleHAiOjE3NjM1MDA5MTgsImlhdCI6MTc2MzUwMDYxOCwianRpIjoiMzBkYjU0OGUtNzQ1OS05MGVlLWYwYmItMjk1MGIzNDJjMTQ2IiwibmJmIjoxNzYzNTAwNjE4LCJub21hZF9ub2RlX25hbWUiOiIiLCJub21hZF9ub2RlX3Bvb2wiOiJkZWZhdWx0Iiwic3ViIjoibm9kZS1pbnRyb2R1Y3Rpb246Z2xvYmFsOmRlZmF1bHQ6OmRlZmF1bHQifQ.acAuFxXshidUF_61l3HFgI0W_eW9ag5J9jptklNb5was2go5E8IHtvueWeYOMcsZakegLX8GNMy8CmBhtSRmmvzWxCoXf5CDHo9jqyxTrFc8yWFozbCucieTWBsyyKMqgkgf52QCm5owO0Kw40AzDK0sqkYrmzh4FrXCruomk0Zmmw9nHfactLRkT4PfYE_RpvMp7ptNkXNlEuEypZ-oVGxOK1maXZ51F0A9xrrMqxBroIfpusuGj8OI4j8VCjDnTKlI1wPWPvNVdirpt3vWp3FdZ-OCTPIGD7wmTzzhdec-cADWrQOxuczmJPoH0U4iF7AdNLswJGhhIKyN4ktVbA"
}

Generate Nomad Client Introduction Token via CLI#

In addition to generating the token via API, you can also generate it via the CLI.

Similar to the API, you can pass in additional parameters if needed, but in this case we will just generate a basic token.

nomad node intro create  > intro_token.jwt

The content of the intro_token.jwt file will be similar to this:

 "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgyNjExLTNkZjItMWM3My1kMDEwLTA1YmUxNmQ3NTBlOCIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJub21hZHByb2plY3QuaW8iLCJleHAiOjE3NjM1MDMwNzgsImlhdCI6MTc2MzUwMjc3OCwianRpIjoiZDlhZmRmZjEtMTcxYy1lYWJjLTU4MmEtZjc3NmJjNzY2N2I0IiwibmJmIjoxNzYzNTAyNzc4LCJub21hZF9ub2RlX25hbWUiOiIiLCJub21hZF9ub2RlX3Bvb2wiOiJkZWZhdWx0Iiwic3ViIjoibm9kZS1pbnRyb2R1Y3Rpb246Z2xvYmFsOmRlZmF1bHQ6OmRlZmF1bHQifQ.P9xDbK9dlD8g7XvZ8Wq87mxddCTniaHP0qsR5ovGw9H6KtG6-rHpkYYzKgHgQBVXDUnQjo29yUBLDbZFsa2woub4bX_Qv3LH1TqNq1eAJ3W3W0sHBBj_NZx9GKwW_5MmdlXWe9_tgQVpTM19x1f0Tu8qGRZsQAvZWWhMJZQr9Ad32OP492CT5r_8aZafYCyZfJJqr8puFBFe5_PsEQ5uL-BXT-L8owE41OV9kUZ8zUF3YG75hWCzIcZp-nZHuqzOzqIElXA83V-u4GObrOZKIltkazIbH-JZ20XTA83b8YSh477XQ7OtI1wBD_fKZOXb5ehl7gjFocheSN9LgSdHFg"

With that, we are ready to use this token when starting up our Nomad client agent. In both of these we only want the value of the JWT field and not the surrounding JSON structure.

Using the Client Introduction Token#

To use the client introduction token, we have a couple of options.

Set the -client-intro-token flag when starting up the Nomad agent.
Place the token in a file named intro_token.jwt within the client’s state directory (default is <data_dir>/<client_state_dir>/).
Set the NOMAD_CLIENT_INTRO_TOKEN environment variable.

For this, I set the token with the environment variable and start up the Nomad client agent again:

nomad agent -config /etc/nomad.d/nomad.hcl
==> Loaded configuration from /etc/nomad.d/nomad.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 192.168.39.133:4646
            Bind Addrs: HTTP: [0.0.0.0:4646]
                Client: true
             Log Level: INFO
                Region: global (DC: dc1)
                Server: false
               Version: 1.11.0

==> Nomad agent started! Log data will stream in below:
 ...
    2025-11-19T02:38:56.724Z [INFO]  client: setting node identity token
    2025-11-19T02:38:56.724Z [INFO]  client: started client: node_id=01f47f9a-7281-c450-6db0-3220fdc016a0
    2025-11-19T02:38:56.728Z [INFO]  client: setting node identity token
    2025-11-19T02:38:56.735Z [INFO]  client: node registration complete
    2025-11-19T02:39:03.193Z [INFO]  client: node registration complete

And there we have it! The client has successfully joined the Nomad cluster using the client introduction token.

Conclusion#

The client node introduction feature in Nomad 1.11.x adds an important layer of security to your Nomad clusters. By requiring clients to present a valid introduction token, you can better control which nodes are allowed to join your cluster and node pools. This is especially useful in environments where security is a top priority.