Memory usage discrepancy: cgroup memory.usage_in_bytes vs. RSS inside docker container

by

I don’t know if you already find your answer or not but let me give you some information that may help.

  • cAdvisor extract many memory-related metrics. We will focus on:

    1. container_memory_usage_bytes = value in /sys/fs/cgroup/memory/memory.usage_in_bytes file. (Usage of the memory)

    2. container_memory_working_set_bytes = container_memory_usage_bytes – total_inactive_file (from /sys/fs/cgroup/memory/memory.stat), this is calculated in cAdvisor and is <= container_memory_usage_bytes

    3. container_memory_rss = total_rss value from /sys/fs/cgroup/memory/memory.stat

  • Now you know how those metrics are gathered, you need to know that when you use the kubectl top pods command, you get the value of container_memory_working_set_bytes not container_memory_usage_bytes metric.

    so from your values:

    5039Mi “working set fro kubectl command” ~= 5064 “from memory.usage file” – 28 “total_inactive_file from Memory section from docker’s container stats API”

  • It is also worth to mention that when the value of container_memory_usage_bytes reaches to the limits, your pod will NOT get oom-killed. BUT if container_memory_working_set_bytes or container_memory_rss reached to the limits, the pod will be killed.

Leave a Comment