Use a different checkpointing mechanism

Oct 13, 2025 by wbrum.org 87 views

Introduction

In this article, we will explore the issue of checkpointing and restoring background processes using runsc. We will delve into the steps to reproduce the problem, analyze the issue, and provide potential solutions.

Background

runsc is a command-line tool for running and managing containers. It provides a way to checkpoint and restore containers, allowing for efficient use of resources and improved reliability. However, we have encountered an issue where background processes are not properly restored after checkpointing and restoring a container.

Steps to Reproduce

To reproduce the issue, follow these steps:

Configure CNI

First, we need to configure CNI (Container Network Interface) to create a network namespace for our container. We will use the cni plugin to create a bridge network.

sudo mkdir -p /opt/cni/bin
wget https://github.com/containernetworking/plugins/releases/download/v0.8.3/cni-plugins-linux-amd64-v0.8.3.tgz
sudo tar -xvf cni-plugins-linux-amd64-v0.8.3.tgz -C /opt/cni/bin/
sudo mkdir -p /etc/cni/net.d

sudo sh -c 'cat > /etc/cni/net.d/10-bridge.conf << EOF
{
  "cniVersion": "0.3.1",
  "name": "mynet",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "subnet": "10.22.0.0/16",
    "routes": [
      { "dst": "0.0.0.0/0" }
    ]
  }
}
EOF'

sudo sh -c 'cat > /etc/cni/net.d/99-loopback.conf << EOF
{
  "cniVersion": "0.3.1",
  "name": "lo",
  "type": "loopback"
}
EOF'

Create and Configure Namespace for Container

Next, we need to create a network namespace for our container using the ip command.

export CNI_PATH=/opt/cni/bin
export CNI_CONTAINERID=$(printf '%x%x%x%x' $RANDOM $RANDOM $RANDOM $RANDOM)
export CNI_COMMAND=ADD
export CNI_NETNS=/var/run/netns/${CNI_CONTAINERID}

sudo ip netns add ${CNI_CONTAINERID}
export CNI_IFNAME="eth0"
sudo -E /opt/cni/bin/bridge < /etc/cni/net.d/10-bridge.conf
export CNI_IFNAME="lo"
sudo -E /opt/cni/bin/loopback < /etc/cni/net.d/99-loopback.conf

Create and Configure Container

Now, we need to create a container using the runsc command.

sudo mkdir -p bundle
cd bundle
sudo mkdir rootfs
sudo docker export $(docker create python) | sudo tar --same-owner -pxf - -C rootfs
sudo mkdir -p rootfs/var/www/html
sudo sh -c 'echo "Hello World!" > rootfs/var/www/html/index.html'
sudo runsc spec \
    --cwd /var/www/html \
    --netns /var/run/netns/${CNI_CONTAINERID} \
    -- python -m http.server

Run Container and Make Sure It's Working

Next, we need to run the container using the runsc command.

sudo runsc run -detach ${CNI_CONTAINERID}
# Confirm it's running:
sudo runsc list
# See that we can contact the server from inside the container
sudo runsc exec ${CNI_CONTAINERID} curl http://localhost:8000
# Launch a background server
sudo runsc exec ${CNI_CONTAINERID} bash -c "python3 -m http.server 8001 >/dev/null 2>&1 &"
# Check that it's running
sudo runsc exec ${CNI_CONTAINERID} curl http://localhost:8001
# Outputs:
# Hello World!

Take a Checkpoint and Try to Restore It

Now, we need to take a checkpoint of the container using the runsc command.

# Take a checkpoint into a directory checkpoint
sudo runsc --debug-log=/tmp/runsc/ --debug --strace checkpoint --image-path=checkpoint ${CNI_CONTAINERID}
# Confirm checkpoint exists
ls checkpoint/
# Outputs:
# checkpoint.img
# Confirm container is stopped
sudo runsc list
# Outputs:
# ID                 PID         STATUS      BUNDLE                  CREATED                          OWNER
# 2e372ef02c024ef4   -1          stopped     /home/ec2-user/bundle   2025-03-12T14:27:04.906888314Z   root

# Restore and start new container from checkpoint image
sudo runsc --debug-log=/tmp/runsc-restore/ --debug --strace restore --image-path=checkpoint restored

# Check it's running
sudo runsc list
# Outputs: 
# ID                 PID         STATUS      BUNDLE                  CREATED                          OWNER
# 2e372ef02c024ef4   -1          stopped     /home/ec2-user/bundle   2025-03-12T14:27:04.906888314Z   root
# restored           703710      running     /home/ec2-user/bundle   2025-03-12T14:37:49.662967206Z   root

# Check running processes. Main entrypoint was successfully restored, 
# but the background process is restored as defunct
sudo runsc exec restored ps -fea
# Outputs:
# UID        PID  PPID  C STIME TTY          TIME CMD
# root         1     0  0 14:27 ?        00:00:00 python -m http.server
# root         6     1  0 14:32 ?        00:00:00 [python3] <defunct>
# root        11     0  0 14:40 ?        00:00:00 ps -fea

# Entrypoint working fine
sudo runsc exec restored curl http://localhost:8000
# Outputs:
# Hello World!

Analysis

After reproducing the issue, we can see that the background process is restored as defunct. This means that the process is not running and is not responding to any signals. We can also see that the main entrypoint of the container is successfully restored.

Potential Solutions

There are several potential solutions to this issue:

Use a different checkpointing mechanism: Instead of using the runsc command to take a checkpoint, we can use a different mechanism such as docker checkpoint or criu.
Use a different container runtime: Instead of using runsc, we can use a different container runtime such as docker or cri-o.
Modify the container configuration: We can modify the container configuration to prevent the background process from being restored as defunct.
Use a different network namespace: We can use a different network namespace to isolate the container and prevent the background process from being restored as defunct.

Conclusion

In this article, we have explored the issue of checkpointing and restoring background processes using runsc. We have reproduced the issue, analyzed the problem, and provided potential solutions. We hope that this article has been helpful in understanding the issue and finding a solution.

Future Work

In the future, we plan to investigate the issue further and provide more detailed analysis. We also plan to test the potential solutions and provide more information on the results.

References

Acknowledgments

Q: What is the issue with checkpointing and restoring background processes using `runsc`?

A: The issue is that background processes are not properly restored after checkpointing and restoring a container. Instead, they are restored as defunct, meaning they are not running and are not responding to any signals.

Q: What are the steps to reproduce the issue?

A: To reproduce the issue, follow these steps:

Configure CNI (Container Network Interface) to create a network namespace for your container.
Create and configure a namespace for your container using the ip command.
Create and configure a container using the runsc command.
Run the container using the runsc command.
Launch a background server using the runsc command.
Take a checkpoint of the container using the runsc command.
Restore and start a new container from the checkpoint image using the runsc command.

Q: What are the potential solutions to the issue?

A: There are several potential solutions to the issue:

Use a different checkpointing mechanism, such as docker checkpoint or criu.
Use a different container runtime, such as docker or cri-o.
Modify the container configuration to prevent the background process from being restored as defunct.
Use a different network namespace to isolate the container and prevent the background process from being restored as defunct.

Q: What are the benefits of using `runsc` for checkpointing and restoring containers?

A: The benefits of using runsc for checkpointing and restoring containers include:

Efficient use of resources: runsc allows you to checkpoint and restore containers without having to recreate them from scratch.
Improved reliability: runsc provides a way to checkpoint and restore containers, which can help to improve the reliability of your containerized applications.
Simplified management: runsc provides a simple and intuitive way to manage containers, including checkpointing and restoring them.

Q: What are the limitations of using `runsc` for checkpointing and restoring containers?

A: The limitations of using runsc for checkpointing and restoring containers include:

Limited support for certain container runtimes: runsc may not support all container runtimes, which can limit its use.
Limited support for certain container configurations: runsc may not support all container configurations, which can limit its use.
Potential issues with checkpointing and restoring background processes: runsc may not properly restore background processes, which can cause issues.

Q: How can I troubleshoot issues with checkpointing and restoring containers using `runsc`?

A: To troubleshoot issues with checkpointing and restoring containers using runsc, follow these steps:

Check the runsc logs for errors: The runsc logs can provide valuable information about any errors that occur during checkpointing and restoring.
Check the container configuration: The container configuration can affect the behavior of runsc during checkpointing and restoring.
Check the container runtime: The container runtime can affect the behavior of runsc during checkpointing and restoring.
Use the runsc debug mode: The runsc debug mode can provide additional information about the checkpointing and restoring process.

Q: How can I prevent background processes from being restored as defunct when using `runsc`?

A: To prevent background processes from being restored as defunct when using runsc, follow these steps:

Use a different checkpointing mechanism, such as docker checkpoint or criu.
Use a different container runtime, such as docker or cri-o.
Modify the container configuration to prevent the background process from being restored as defunct.
Use a different network namespace to isolate the container and prevent the background process from being restored as defunct.