Wrong RAM consumtion on poststart sidecar tasks

[iGorebach]

Nomad version

1.8.3

Operating system and Environment details

Ubuntu 22.04, kernel version 6.5.0-35-generic

Issue

Wrong memory utilization is displayed in UI and in Nomad metrics for the post-start sidecar tasks, which work with the filesystem.

Total job memory usage

Poststart sidecar task memory usage

Real memory usage

docker stats a9481fb2f57d --no-stream
CONTAINER ID   NAME                                          CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O    PIDS
a9481fb2f57d   backup-9d2f286d-a117-d31a-0e41-67c997fe08d4   0.00%     15.22MiB / 1GiB     1.49%     0B / 0B   0B / 952MB   4

similar issue

Reproduction steps

Create the poststart sidecar container, working in cmd loop with fs write operations or connect to task manually and create a file on /alloc mount point.

Expected Result

No memory consumption affected

Actual Result

Every file created increases RAM utilization by its size, real utilization is different.

[iGorebach]

The same behavior occurred when CSI volumes were used.

[qk4l]

This bug affects not just post/pre-start sidecar tasks but any task within the group.

For instance, consider the following scenario: I have two tasks, nginx and app. If app writes its logs to /alloc/logs, the size of these logs is incorrectly attributed as RAM usage for the nginx task.

This behavior can lead to misleading resource metrics and potential issues in resource management.

[Juanadelacuesta]

Hello @iGorebach, thank you for reporting this issue, I will forwarding it to @philrenaud who will hopefully look at it soon.

[tgross]

Internal ref: https://hashicorp.atlassian.net/browse/NET-12151