Skip to content

Incident Post-Mortem: Unexplained LXC Container Shutdown & Autostart Failure

Executive Summary

On Ubuntu 24.04 running native LXC (not LXD), multiple container instances stopped unexpectedly and failed to automatically restart. Investigation revealed that an automated system package update (unattended-upgrades) triggered a restart of lxc.service and lxc-net.service. Due to a systemd race condition between the bridge network creation (lxcbr0) and the container autostart routine, the containers failed to bind to the network bridge during service restart, leaving them in a stopped state.

Technical Details & Root Cause Analysis

  1. Trigger Event: unattended-upgrades executed an update for system packages (git, python3-jwt) at 06:20:59.
  2. Service Restart: System package triggers caused systemd to stop and restart lxc.service and lxc-net.service at 06:21:03.
  3. The Race Condition:
  4. lxc.service and lxc-net.service were restarted almost simultaneously.
  5. The container autostart script (lxc-autostart) fired before the virtual bridge (lxcbr0) and dnsmasq fully initialized and bound to their network sockets.
  6. Because the containers were configured with lxc.start.auto = 1 but could not attach to lxcbr0 at that exact millisecond, the startup sequence aborted.
  7. Manual Recovery: Running sudo lxc-autostart -a manually worked because the bridge interface had plenty of time to stabilize prior to execution.

Diagnostic & Troubleshooting Log

Below are the commands used during the investigation to isolate and confirm the root cause:

1. Checking Container Logs & Information

Native LXC does not keep persistent runtime log files by default. To check a running container's status or capture a debug log: 

# View general information and log path for a container
sudo lxc-info -n <container_name> -H

# Capture detailed startup debug output to a temporary file
sudo lxc-start -n <container_name> -l DEBUG -o /tmp/<container_name>-debug.log

2. Investigating Host Kernel and System Logs

To check if the containers were terminated by the Out-Of-Memory (OOM) killer or kernel panics: 

# Search for OOM events
sudo grep -i -E 'oom|out of memory' /var/log/syslog
# Or via journalctl
sudo journalctl -k | grep -i -E 'oom|out of memory'

3. Reviewing LXC Service Logs

Filtering journalctl for LXC-related services around the failure timestamp revealed the service restart event: 

sudo journalctl -u lxc.service --since "06:20:00" --until "06:25:00"
Output excerpt: 
06:21:03 systemd[1]: Stopping lxc.service - LXC Container Initialization and Autoboot Code...
06:21:10 systemd[1]: Stopped lxc.service - LXC Container Initialization and Autoboot Code.
06:21:11 systemd[1]: Starting lxc.service - LXC Container Initialization and Autoboot Code...
06:21:11 systemd[1]: Finished lxc.service - LXC Container Initialization and Autoboot Code.

4. Auditing System Package Upgrades

To verify what triggered the service restart at 06:21:03: 

# Check dpkg package installation history
grep -E "2026-10-20 06:2[0-5]" /var/log/dpkg.log

# Inspect unattended-upgrades log
grep -E "2026-10-20 06:2[0-5]" /var/log/unattended-upgrades/unattended-upgrades.log

Remediation & Permanent Solution

To ensure that lxc.service waits for lxc-net.service and the virtual network bridge (lxcbr0) to fully initialize before attempting to start containers, a 5-second execution delay was added to the systemd unit config.

1. Ensure Container Configuration Has Autostart Enabled

Each container's configuration file (located at /var/lib/lxc/<container_name>/config) must include: 

lxc.start.auto = 1
lxc.start.delay = 5

2. Configure Systemd Override Delay

Create a systemd service drop-in override: 

sudo systemctl edit lxc.service

Add the following configuration block above the comment lines: 

[Service]
ExecStartPre=/bin/sleep 5

3. Apply Systemd Configuration

Reload systemd to pick up the changes: 

sudo systemctl daemon-reload

Verification

You can manually test container autostart processing at any time without rebooting the host by executing: 

sudo lxc-autostart -a