Nomad client new tasks goes into pending state is server restarts (mac).

[raghav155]

Nomad clients running on macOS do not resume normal task execution after the Nomad server is restarted, even though they reconnect and appear ready. In contrast, Ubuntu clients reconnect and continue accepting jobs as expected.

Nomad version

Nomad v1.6.2
BuildDate 2023-09-13T16:47:25Z
Revision 73e372a

Operating system and Environment details

ProductName: macOS
ProductVersion: 14.5
BuildVersion: 23F79

Issue

I'm running nomad server as a statefulset. The clients are running on a pool of machines (macos servers).
Nomad clients running on macOS do not resume normal task execution after the Nomad server is restarted, even though they reconnect and appear ready. In contrast, Ubuntu clients reconnect and continue accepting jobs as expected.

Reproduction steps

1. Restart the nomad server
2. Schedule a new job on the server
3. The clients will connect to the server but the new allocations which go into pending state.

Expected Result

1. When the Nomad server restarts, all connected Nomad clients (Linux/macOS) should reconnect.
2. Jobs submitted after server restart should be accepted and placed on any eligible client.
3. Client nodes in ready state should be able to run new tasks.

Actual Result

1. After restarting the Nomad server, the macOS Nomad client:
2. Reconnects to the server
3. Shows up as ready and eligible
4. But jobs submitted afterward remain in pending state when placed on the macOS client
5. The same job runs fine when targeted to a Linux (Ubuntu) client.

Restarting the Nomad client process on macOS immediately fixes the issue — jobs get placed and run correctly.

Logs:
nomad node status on macOS shows:

Status: ready
Eligibility: eligible
Allocated Resources: 0
Driver Status: raw_exec

Allocation remains pending (nomad job status destroy_job_tafbw3ybdhpwzfaq1aep):

ID            = destroy_job_tafbw3ybdhpwzfaq1aep
Name          = 2cZmM4V7xTTQcugQlT1c
Submit Date   = 2025-06-19T23:08:51-07:00
Type          = batch
Priority      = 50
Datacenters   = dc1
Namespace     = default
Node Pool     = <none>
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group                              Queued  Starting  Running  Failed  Complete  Lost  Unknown
delete_task_group_tafbw3ybdhpwzfaq1aep  0       1         0        0       0         0     0

Allocations
ID        Node ID   Task Group                              Version  Desired  Status   Created   Modified
cb09f4cb  5e777097  delete_task_group_tafbw3ybdhpwzfaq1aep  4        run      pending  1h2m ago  49m51s ago

nomad alloc status cb09f4cb

ID                  = cb09f4cb-f01d-c1ce-becc-711b5ace0b6d
Eval ID             = df69462a
Name                = destroy_job_tafbw3ybdhpwzfaq1aep.delete_task_group_tafbw3ybdhpwzfaq1aep[0]
Node ID             = 5e777097
Node Name           = 67604.local
Job ID              = destroy_job_tafbw3ybdhpwzfaq1aep
Job Version         = 4
Client Status       = pending
Client Description  = <none>
Desired Status      = run
Desired Description = <none>
Created             = 1h3m ago
Modified            = 50m33s ago

Couldn't retrieve stats: Unexpected response code: 404 (rpc error: Unknown allocation "cb09f4cb-f01d-c1ce-becc-711b5ace0b6d")

[Juanadelacuesta]

Hi @raghav155 thank you for the very complete description of your problem, it makes our job a lot easier :)
Do you mind sharing you jobspec? What task driver are you using for this job?

[raghav155]

Hey @Juanadelacuesta , sharing the full job spec below. I'm using raw_exec driver.

{
    "Job": {
        "Affinities": null,
        "AllAtOnce": false,
        "Constraints": [
            {
                "LTarget": "${node.unique.id}",
                "Operand": "=",
                "RTarget": "446bc363-cc4d-fee9-990f-20b2dabaec5d"
            }
        ],
        "ConsulNamespace": "",
        "ConsulToken": "",
        "CreateIndex": 376912,
        "Datacenters": [
            "dc1"
        ],
        "DispatchIdempotencyToken": "",
        "Dispatched": false,
        "ID": "destroy_job_zyatqqphtlclpxtnlhca",
        "JobModifyIndex": 376912,
        "Meta": null,
        "Migrate": null,
        "ModifyIndex": 376925,
        "Multiregion": null,
        "Name": "ES6oSsfPeXX0U4Bb7wbT",
        "Namespace": "default",
        "NodePool": null,
        "NomadTokenID": "",
        "ParameterizedJob": null,
        "ParentID": "",
        "Payload": null,
        "Periodic": null,
        "Priority": 50,
        "Region": "global",
        "Reschedule": null,
        "Spreads": null,
        "Stable": false,
        "Status": "running",
        "StatusDescription": "",
        "Stop": false,
        "SubmitTime": 1749821089312363331,
        "TaskGroups": [
            {
                "Affinities": null,
                "Constraints": null,
                "Consul": {
                    "Namespace": ""
                },
                "Count": 1,
                "EphemeralDisk": {
                    "Migrate": false,
                    "SizeMB": 300,
                    "Sticky": false
                },
                "MaxClientDisconnect": null,
                "Meta": null,
                "Migrate": null,
                "Name": "delete_task_group_zyatqqphtlclpxtnlhca",
                "Networks": null,
                "ReschedulePolicy": {
                    "Attempts": 1,
                    "Delay": 5000000000,
                    "DelayFunction": "constant",
                    "Interval": 86400000000000,
                    "MaxDelay": 0,
                    "Unlimited": false
                },
                "RestartPolicy": {
                    "Attempts": 1,
                    "Delay": 15000000000,
                    "Interval": 86400000000000,
                    "Mode": "fail",
                    "RenderTemplates": null
                },
                "Scaling": null,
                "Services": null,
                "ShutdownDelay": null,
                "Spreads": null,
                "StopAfterClientDisconnect": 240000000000,
                "Tasks": [
                    {
                        "Affinities": null,
                        "Artifacts": null,
                        "Config": {
                            "args": [
                                "-c",
                                "\n\t    /usr/local/bin/ignite stop zyatqqphtlclpxtnlhca; /usr/local/bin/ignite rm zyatqqphtlclpxtnlhca\n\t\tif [ $? -ne 0 ]; then\n\t\t  /usr/local/bin/ignite stop -f zyatqqphtlclpxtnlhca; /usr/local/bin/ignite rm -f zyatqqphtlclpxtnlhca\n\t\tfi\n\t"
                            ],
                            "command": "/usr/bin/su"
                        },
                        "Constraints": null,
                        "DispatchPayload": null,
                        "Driver": "raw_exec",
                        "Env": null,
                        "Identity": null,
                        "KillSignal": "",
                        "KillTimeout": 5000000000,
                        "Kind": "",
                        "Leader": false,
                        "Lifecycle": null,
                        "LogConfig": {
                            "Disabled": null,
                            "Enabled": null,
                            "MaxFileSizeMB": 10,
                            "MaxFiles": 10
                        },
                        "Meta": null,
                        "Name": "ignite_stop_and_rm",
                        "Resources": {
                            "CPU": 40,
                            "Cores": 0,
                            "Devices": null,
                            "DiskMB": 0,
                            "IOPS": 0,
                            "MemoryMB": 20,
                            "MemoryMaxMB": 0,
                            "Networks": null
                        },
                        "RestartPolicy": {
                            "Attempts": 1,
                            "Delay": 15000000000,
                            "Interval": 86400000000000,
                            "Mode": "fail",
                            "RenderTemplates": null
                        },
                        "ScalingPolicies": null,
                        "Services": null,
                        "ShutdownDelay": 0,
                        "Templates": null,
                        "User": "",
                        "Vault": null,
                        "VolumeMounts": null
                    }
                ],
                "Update": null,
                "Volumes": null
            }
        ],
        "Type": "batch",
        "Update": {
            "AutoPromote": false,
            "AutoRevert": false,
            "Canary": 0,
            "HealthCheck": "",
            "HealthyDeadline": 0,
            "MaxParallel": 0,
            "MinHealthyTime": 0,
            "ProgressDeadline": 0,
            "Stagger": 0
        },
        "VaultNamespace": "",
        "VaultToken": "",
        "Version": 0
    }
}

[raghav155]

Also sharing the client.hcl file for better debugging.

data_dir  = "/opt/nomad/data"
bind_addr = "127.0.0.1"

plugin "raw_exec" {
  config {
    enabled = true
  }
}

plugin "generic-device" {
  config {
    device {
      type = "vm"
      model = "large"
    }
  }
}

advertise {
  http = "{HOST_IP}",
  rpc = "{HOST_IP}",
  serf = "{HOST_IP}"
}

tls {
  http = true
  rpc  = true

  ca_file   = "/etc/tls/ca.pem"
  cert_file = "/etc/tls/nomad.pem"
  key_file  = "/etc/tls/nomad-key.pem"

  verify_server_hostname = true
}

client {
  gc_interval="10m"
  gc_max_allocs=50
  enabled = true
  servers = ["{SERVER_IP}"]
}