1.10.2 Regression: Task CPU/Memory usage zero / not reported by clients

[konnextv]

Nomad version

Nomad v1.10.2
BuildDate 2025-06-09T22:00:49Z
Revision df4c764

Operating system and Environment details

24.04.2 LTS (GNU/Linux 6.8.0-51-generic aarch64)

Issue

Task CPU/Memory usage is zero when a client's version is 1.10.2. Observed with exec/raw_exec drivers.
Downgrading (only the client!) to 1.10.1 instantly brings back the actual measured values.

Reproduction steps

On a client (v1.10.2) run a job with the exec or raw_exec driver (could only test with Ubuntu/aarch64 OS, maybe someone reports the same for other arch/OS). Observe the CPU/memory usage of tasks on that client (e.g. on the client page).

Expected Result

Actual CPU/memory usages should be reported.

Actual Result

CPU/memory usages stay at 0 on clients running v1.10.2.

[tgross]

Hi @konnextv! I did a quick check and wasn't able to confirm this. For example, with the following simple rawexec job:

jobspec

job "example" {

  group "web" {

    network {
      mode = "host"
    }

    task "http" {

      driver = "raw_exec"

      config {
        command = "/usr/sbin/busybox"
        args    = ["httpd", "-v", "-f", "-p", "8001", "-h", "${NOMAD_TASK_DIR}"]
      }

      resources {
        cpu    = 100
        memory = 100
      }

    }
  }
}

I run that and curl its endpoint in a tight loop to run up its CPU usage. After 30s or so I check its stats as follows and see both CPU and memory collected. (I also seem them in the web UI.)

$ nomad operator api "/v1/client/allocation/776d4b34-4d78-2953-c453-512dc76b1a91/stats" | jq .

stats

{
  "ResourceUsage": {
    "CpuStats": {
      "Measured": [
        "System Mode",
        "User Mode",
        "Percent"
      ],
      "Percent": 1.798492638716874,
      "SystemMode": 1.7984926800475654,
      "ThrottledPeriods": 0,
      "ThrottledTime": 0,
      "TotalTicks": 20.764415010640274,
      "UserMode": 0.0
    },
    "DeviceStats": [],
    "MemoryStats": {
      "Cache": 0,
      "KernelMaxUsage": 0,
      "KernelUsage": 0,
      "MappedFile": 0,
      "MaxUsage": 0,
      "Measured": [
        "RSS",
        "Swap"
      ],
      "RSS": 671744,
      "Swap": 0,
      "Usage": 0
    }
  },
  "Tasks": {
    "http": {
      "Pids": {
        "78513": {
          "CpuStats": {
            "Measured": [
              "System Mode",
              "User Mode",
              "Percent"
            ],
            "Percent": 1.798492638716874,
            "SystemMode": 1.7984926800475654,
            "ThrottledPeriods": 0,
            "ThrottledTime": 0,
            "TotalTicks": 0.0,
            "UserMode": 0.0
          },
          "DeviceStats": null,
          "MemoryStats": {
            "Cache": 0,
            "KernelMaxUsage": 0,
            "KernelUsage": 0,
            "MappedFile": 0,
            "MaxUsage": 0,
            "Measured": [
              "RSS",
              "Swap"
            ],
            "RSS": 671744,
            "Swap": 0,
            "Usage": 0
          }
        }
      },
      "ResourceUsage": {
        "CpuStats": {
          "Measured": [
            "System Mode",
            "User Mode",
            "Percent"
          ],
          "Percent": 1.798492638716874,
          "SystemMode": 1.7984926800475654,
          "ThrottledPeriods": 0,
          "ThrottledTime": 0,
          "TotalTicks": 20.764415010640274,
          "UserMode": 0.0
        },
        "DeviceStats": null,
        "MemoryStats": {
          "Cache": 0,
          "KernelMaxUsage": 0,
          "KernelUsage": 0,
          "MappedFile": 0,
          "MaxUsage": 0,
          "Measured": [
            "RSS",
            "Swap"
          ],
          "RSS": 671744,
          "Swap": 0,
          "Usage": 0
        }
      },
      "Timestamp": 1749670927008620719
    }
  },
  "Timestamp": 1749670927008620719
}

Can you verify that telemetry.publish_allocation_metrics = true in your agent configuration? In #25870 we fixed a bug where this value wasn't getting respected correctly, so it may be that you were previously collecting metrics that shouldn't have been allowed by configuration. If that's not the problem, then there may be something platform-specific we'll need to investigate.

[konnextv]

Bingo, I did not include that setting in the config because it was working without from day 1 :D

After enabling the mentioned setting, everything works again as it used to.

While we are here, I also did not set publish_node_metrics = true but could still see the CPU/RAM usage of clients. What does that setting actually affect?

[tgross]

While we are here, I also did not set publish_node_metrics = true but could still see the CPU/RAM usage of clients. What does that setting actually affect?

It should only impact the publishing of those metrics on the metrics endpoint, not the values reported in the node API you see in the UI. The docs for this configuration could probably use a little improvement here.