Identify the faulty hardware

Oct 13, 2025 by wbrum.org 51 views

Debugging OOMed Linux with Low Load, Very Low Commit%, and Low Slab: A Step-by-Step Guide

Linux systems are designed to handle a wide range of workloads, from simple file servers to complex enterprise applications. However, even with the most robust systems, memory-related issues can arise, leading to Out-Of-Memory (OOM) conditions. In this article, we will explore the steps to debug a Linux system that has experienced an OOM condition, despite having a low load, very low commit%, and low slab.

The scenario described involves a Rocky9 (RHEL-like) server with an NFS workload, where the memory usage has increased steadily from 1% to 99%. The key symptoms of this issue are:

Low load: The system load is low, indicating that there are no resource-intensive processes running.
Very low commit%: The commit percentage is very low, suggesting that the system is not experiencing any significant memory pressure.
Low slab: The slab memory usage is low, indicating that the kernel's memory allocation system is not experiencing any issues.

To debug this issue, we need to gather as much information as possible about the system's memory usage and behavior. Here are some key tools and commands to use:

1. vmstat

The vmstat command provides information about the system's memory usage, including the number of pages swapped in and out, the number of pages free, and the number of pages allocated to the kernel.

vmstat 1 10

This command will display the memory statistics every second for 10 seconds.

2. free

The free command displays the amount of free and used memory on the system, as well as the amount of memory allocated to the kernel.

free -m

This command will display the memory statistics in megabytes.

3. slabtop

The slabtop command displays information about the kernel's memory allocation system, including the number of objects allocated and the amount of memory used.

slabtop -S -o name,active,objects,pages

This command will display the slab statistics, sorted by the number of active objects.

4. dmesg

The dmesg command displays the kernel's log messages, which can provide valuable information about the system's behavior.

dmesg | grep oom

This command will display the kernel log messages related to OOM conditions.

5. /proc/meminfo

The /proc/meminfo file provides detailed information about the system's memory usage, including the amount of free and used memory, the amount of memory allocated to the kernel, and the number of pages swapped in and out.

cat /proc/meminfo

This command will display the memory statistics in kilobytes.

Once we have gathered the necessary information, we need to analyze it to identify the root cause of the issue. Here are some key things to look for:

Memory leaks: Check for any memory leaks in the system's applications or services. A memory leak occurs when a program allocates memory but fails to release it, leading to a gradual increase in memory usage.
Kernel bugs: Check for any kernel bugs or issues that may be causing the memory usage to increase. The kernel's log messages can provide valuable information about any issues that may be occurring.
Hardware issues: Check for any hardware issues that may be causing the memory usage to increase. For example, a faulty RAM module can cause the system to use more memory than necessary.

Once we have identified the root cause of the issue, we can begin debugging it. Here are some key steps to follow:

1. Identify the memory leak

If we suspect that a memory leak is the cause of the issue, we need to identify the program or service responsible for the leak. We can use tools like valgrind or memleak to detect memory leaks.

2. Fix the memory leak

Once we have identified the program or service responsible for the memory leak, we need to fix the issue. This may involve updating the program or service to a newer version, or modifying the code to release the allocated memory.

3. Fix kernel bugs

If we suspect that a kernel bug is the cause of the issue, we need to fix the bug. This may involve updating the kernel to a newer version, or applying a patch to fix the issue.

4. Replace faulty hardware

If we suspect that a hardware issue is the cause of the issue, we need to replace the faulty hardware. This may involve replacing a faulty RAM module, or replacing a faulty hard drive.

Debugging an OOMed Linux system with low load, very low commit%, and low slab can be a challenging task. However, by gathering information, analyzing the data, and debugging the issue, we can identify and fix the root cause of the problem. Remember to use the right tools and commands to gather information, and to analyze the data carefully to identify the root cause of the issue. With the right approach and tools, we can debug even the most complex memory-related issues.
Debugging OOMed Linux with Low Load, Very Low Commit%, and Low Slab: A Q&A Guide

In our previous article, we explored the steps to debug a Linux system that has experienced an Out-Of-Memory (OOM) condition, despite having a low load, very low commit%, and low slab. In this article, we will provide a Q&A guide to help you understand the concepts and tools involved in debugging OOMed Linux systems.

A: An OOM condition occurs when a Linux system runs out of memory, causing the system to terminate processes or crash.

A: The symptoms of an OOM condition can vary depending on the system and the workload. However, common symptoms include:

High memory usage: The system's memory usage is high, causing the system to run out of memory.
Low load: The system load is low, indicating that there are no resource-intensive processes running.
Very low commit%: The commit percentage is very low, suggesting that the system is not experiencing any significant memory pressure.
Low slab: The slab memory usage is low, indicating that the kernel's memory allocation system is not experiencing any issues.

A: There are several tools that you can use to debug an OOM condition, including:

vmstat: The vmstat command provides information about the system's memory usage, including the number of pages swapped in and out, the number of pages free, and the number of pages allocated to the kernel.
free: The free command displays the amount of free and used memory on the system, as well as the amount of memory allocated to the kernel.
slabtop: The slabtop command displays information about the kernel's memory allocation system, including the number of objects allocated and the amount of memory used.
dmesg: The dmesg command displays the kernel's log messages, which can provide valuable information about the system's behavior.
/proc/meminfo: The /proc/meminfo file provides detailed information about the system's memory usage, including the amount of free and used memory, the amount of memory allocated to the kernel, and the number of pages swapped in and out.

A: Analyzing the data from these tools requires a good understanding of the system's memory usage and behavior. Here are some key things to look for:

Memory leaks: Check for any memory leaks in the system's applications or services. A memory leak occurs when a program allocates memory but fails to release it, leading to a gradual increase in memory usage.
Kernel bugs: Check for any kernel bugs or issues that may be causing the memory usage to increase. The kernel's log messages can provide valuable information about any issues that may be occurring.
Hardware issues: Check for any hardware issues that may be causing the memory usage to increase. For example, a faulty RAM module can cause the system to use more memory than necessary.

A: Fixing a memory leak requires identifying the program or service responsible for the leak and modifying the code to release the allocated memory. Here are some key steps to follow:

Identify the memory leak: Use tools like valgrind or memleak to detect memory leaks.
Fix the memory leak: Modify the code to release the allocated memory.
Test the fix: Test the fix to ensure that the memory leak is resolved.

A: Fixing a kernel bug requires identifying the issue and applying a patch or updating the kernel to a newer version. Here are some key steps to follow:

Identify the kernel bug: Use the kernel's log messages to identify the issue.
Apply a patch: Apply a patch to fix the issue.
Update the kernel: Update the kernel to a newer version.

A: Replacing faulty hardware requires identifying the faulty component and replacing it with a new one. Here are some key steps to follow:

Identify the faulty hardware: Use tools like dmidecode or lspci to identify the faulty component.
Replace the faulty hardware: Replace the faulty component with a new one.
Test the fix: Test the fix to ensure that the issue is resolved.

Debugging an OOMed Linux system with low load, very low commit%, and low slab can be a challenging task. However, by using the right tools and following the steps outlined in this article, you can identify and fix the root cause of the issue. Remember to analyze the data carefully and to test the fix to ensure that the issue is resolved.