Debugging Containers in Style, Oh My Zsh, tmux, and the Art of Not Suffering

You know the feeling. It's 2 AM, something's broken in production, and you kubectl exec into a pod only to be greeted by a blinking cursor on a bare sh shell. No colors. No tab completion. No history. You fat-finger a command, there's no syntax highlighting to warn you, and you're left squinting at a wall of unformatted text wondering if you typed grep or gerp.

Debugging containers doesn't have to feel like this. What if you could drop into a pod and get the same shell experience you have on your laptop? Oh My Zsh with autosuggestions, syntax highlighting, fuzzy history search, and tmux for splitting panes and persisting sessions. That's exactly what we built.

The Problem with kubectl exec

The most common approach to debug a pod is:

kubectl exec -it my-pod -- bash

This drops you into the running container itself. Here's why that's problematic:

1. Missing tools: Most production images (especially distroless or Alpine-based) don't ship with debugging utilities. No tcpdump, no strace, no vim, no htop. You can't even install them without a package manager.

2. Risk to the running process: Anything you do inside the container (installing packages, modifying files, consuming CPU/memory) directly affects the live application. One wrong move and you've crashed production.

3. Immutable containers: In environments with read-only root filesystems or strict security policies, you can't install anything even if you wanted to.

4. No persistence: If the container restarts, everything you did is gone.

Ephemeral debug containers solve all of these problems. They attach a separate container to the pod, with its own filesystem and tools, while sharing the same network, PID namespace, and process visibility as the target.

What's in the Debug Container

A debug container image based on Ubuntu 24.04, pre-loaded with:

• Shell: zsh with Oh My Zsh, syntax highlighting, autosuggestions
• Editors: neovim
• Networking: dig, nslookup, curl, wget, tcpdump, nmap, netcat, traceroute
• Process debugging: htop, strace, ltrace, lsof, ps
• Kubernetes: kubectl (auto-configured with the target's service account)
• Utilities: jq, tmux, tree, file, less

And smart shell integrations that automatically:

• Land you in the target container's filesystem
• Import the target's environment variables
• Configure kubectl for in-cluster API access
• Provide the t command to run target binaries via chroot

The image is available at: docker.io/k7bdevs/debug-container:latest

Two Ways to Use the Debug Container

Method 1: Interactive Session

The simplest approach. Start a debug container and land directly in an interactive zsh shell:

kubectl debug -it -n <namespace> <POD-NAME> \
  --image=docker.io/k7bdevs/debug-container:latest \
  --target=<container-name> \
  --share-processes \
  --profile=general \
  -- zsh

Let's break down each flag:

You'll see:

🔧  Debug container ready  (uid=0)
   • nvim / vim   – editor
   • k / kubectl  – cluster access
   • t <cmd>      – run a command in the target container (e.g. t mongosh)
   • Ctrl+R       – fuzzy search command history (fzf)
   • tmux         – terminal multiplexer

The shell automatically lands you in /proc/1/root, the target container's root filesystem.

The catch: When you type exit, the zsh process terminates, and the ephemeral container dies with it. Ephemeral containers cannot be restarted. You'd need to create a new debug session.

Method 2: Daemon Mode (Recommended)

Start the debug container with sleep infinity, it stays alive in the background. Use --container to give it a predictable name:

kubectl debug -n <namespace> <POD-NAME> \
  --image=docker.io/k7bdevs/debug-container:latest \
  --target=<container-name> \
  --container=debugger \
  --share-processes \
  --profile=general \
  -- sleep infinity

Note: no -it flag. The container starts silently in the background.

The --container=debugger flag gives the ephemeral container a known, predictable name. Without it, Kubernetes auto-generates a random name like debugger-abc12 and you'd have to look it up every time:

# Without --container, you'd need this extra step:
kubectl get pod <POD-NAME> -n <namespace> \
  -o jsonpath='{.spec.ephemeralContainers[-1].name}'
# debugger-abc12   ← random, different every time

With --container=debugger, you always know the name. Open a shell into it:

kubectl exec -it <POD-NAME> -n <namespace> -c debugger -- zsh

That's it. No guessing, no jsonpath lookups.

The advantage: You can exit this shell freely, it only kills the exec session, not the container. Open 5 terminals, close them all, come back an hour later, the debug container is still there. Run kubectl exec -it <POD-NAME> -n <namespace> -c debugger -- zsh again and you're back.

This is the recommended approach for any real debugging session.

Understanding --share-processes and the PID Namespace

When you pass --share-processes, all containers in the pod share the same PID namespace. This means:

• From the debug container, you can see all processes running in the target container
• ps aux shows the target's main process alongside your debug shell
• /proc/<pid>/ entries are visible for all processes across containers

Without --share-processes, the debug container would be an isolated island. It could share the network but couldn't see any of the target's processes.

┌─── Pod (shared PID namespace) ──────────────────┐
│                                                   │
│  ┌─── target container ────┐                      │
│  │  PID 1: app process      │                     │
│  │  /var/run/secrets/...    │  (volume mounts)    │
│  │  /data/                  │                     │
│  └──────────────────────────┘                     │
│                                                   │
│  ┌─── debug container ─────┐                      │
│  │  PID N: zsh              │                     │
│  │  /usr/bin/tcpdump        │  (debug tools)      │
│  │  /usr/bin/kubectl        │                     │
│  │  Can see PID 1 (app)     │                     │
│  └──────────────────────────┘                     │
│                                                   │
│  Shared: Network, PID namespace                   │
│  NOT shared: Filesystem (mount namespace)         │
└───────────────────────────────────────────────────┘

Notice what's NOT shared: the filesystem. Each container has its own mount namespace, its own root filesystem (/). This is where things get interesting.

Understanding /proc/1/root, The Gateway to the Target's Filesystem

Linux exposes every process's root filesystem through /proc/<pid>/root. This is a symbolic link that points to the root directory (/) as seen by that process.

Since the target's main process is PID 1 in the shared PID namespace:

/proc/1/root  →  / (as seen by the target process)
                  ├── bin/
                  ├── data/             ← Application data
                  ├── etc/              ← Configuration files
                  ├── usr/bin/          ← Target's binaries
                  └── var/run/secrets/  ← Service account token

The debug container auto-lands you in /proc/1/root on startup, so you're immediately browsing the target's filesystem.

This works because /proc/<pid>/root is a kernel-provided symlink that points to the root directory of that process's mount namespace. Since the debug container shares the process namespace, it can traverse this symlink to reach into the target container's filesystem, even if the debug container itself has a completely different filesystem layout.

Why this matters: Your target container might be a distroless or scratch-based image with no shell, no cat, no ls, nothing. But from the debug container, you can use your tools (the ones installed in the debug image) to read and inspect their files through /proc/1/root.

Why chroot Is Required, The t Command Explained

Here's a subtle but critical problem. You're in /proc/1/root and you see a binary in the target's filesystem:

ls /proc/1/root/usr/bin/mongosh
# It exists!

mongosh
# zsh: command not found: mongosh

Why? Because your shell's PATH searches the debug container's /usr/bin/, not the target's. The debug container doesn't have mongosh.

OK, let's try the full path:

/proc/1/root/usr/bin/mongosh
# Error: ENOENT: no such file or directory

Still fails! The binary starts to execute, but internally it resolves dependencies using absolute paths like /usr/bin/mongosh and /usr/lib/.... These resolve against the debug container's root (/), not the target's.

What the binary sees:
  /usr/bin/mongosh  →  Debug container's /usr/bin (NO mongosh here!)
  /usr/lib/...      →  Debug container's /usr/lib (WRONG libraries!)

What we need:
  /usr/bin/mongosh  →  Target's /usr/bin/mongosh ✅
  /usr/lib/...      →  Target's /usr/lib/... ✅

The chroot Solution

chroot (change root) changes what / means for a process. After chroot /proc/1/root, all absolute paths resolve against the target's filesystem:

chroot /proc/1/root mongosh
# Connecting to: mongodb://127.0.0.1:27017
# ✅ It works!

The debug container wraps this in the t command (short for "target"):

The t function wraps chroot /proc/1/root so target binaries resolve their dependencies correctly, and passes through whatever command you give it.

Examples:

t mongosh                    # MongoDB shell with full syntax highlighting
t cat /etc/app/config.yaml   # Read target's config files
t bash                       # Drop into a bash shell inside the target's rootfs
t env                        # See the target's environment

The beauty: you keep all your debug tools (nvim, kubectl, tcpdump) available in the debug container, and use t when you need to run something from the target.

Auto-Configured kubectl

One of the trickiest parts of ephemeral containers is that they don't inherit the target's volume mounts. The service account token, mounted at /var/run/secrets/kubernetes.io/serviceaccount inside the target container, is invisible to the debug container.

But with --share-processes, we can reach it through /proc/1/root. The token could be at either path depending on the container runtime and OS. On some systems /var/run is a symlink to /run, but symlinks don't resolve through /proc/1/root. So the debug container checks both:

/proc/1/root/var/run/secrets/kubernetes.io/serviceaccount/
  — or —
/proc/1/root/run/secrets/kubernetes.io/serviceaccount/
├── ca.crt      ← Cluster CA certificate
├── namespace   ← Target's namespace
└── token       ← JWT token for API authentication

The debug container's shell startup automatically:

1. Reads the target's environment from /proc/1/environ to get KUBERNETES_SERVICE_HOST
2. Finds the service account token by checking both /proc/1/root/var/run/secrets/... and /proc/1/root/run/secrets/...
3. Generates a kubeconfig at /tmp/.kubeconfig

So kubectl just works:

k get pods
k get svc
k describe pod <POD-NAME>
k logs <POD-NAME> --tail=50

Debugging in Style, Oh My Zsh

This is the part that changes everything. You're not just getting a shell, you're getting your shell. The debug container ships with Oh My Zsh and a curated set of plugins that turn a hostile debugging environment into something that actually feels good to use.

Syntax Highlighting That Saves You

The zsh-syntax-highlighting plugin colors your commands as you type. Valid commands glow green. Typos and unknown commands turn red, instantly, before you hit Enter. Strings get their own color. Flags are highlighted differently from arguments.

At 2 AM during an incident, this is the difference between confidently running tcpdump -i any port 27017 and accidentally running tcpdmup -i any port 27017, staring at "command not found", and losing 30 seconds of your life you'll never get back.

Autosuggestions from History

The zsh-autosuggestions plugin remembers every command you've typed. As you start typing, it shows the most recent matching command in grey text. Press the right arrow key to accept the whole thing, or keep typing to refine.

This is a game-changer when you're running the same kubectl, curl, or dig commands repeatedly during a debugging session. Type curl and the full curl -v http://some-service.default.svc:8080/health from five minutes ago appears. One keystroke and you're done.

Fuzzy History Search

Press Ctrl+R and start typing any fragment of a previous command. The search is fuzzy, you don't need to remember the exact beginning. Type tcpdump and it finds that long capture command you ran 20 minutes ago with all the flags intact.

No more pressing the up arrow 47 times hoping to find that one command.

Tab Completion for Everything

File paths inside /proc/1/root/? Tab-completable. kubectl subcommands and resource names? Tab-completable. Command flags? Tab-completable. The zsh-completions plugin adds completion definitions for hundreds of tools.

When you're navigating deep into /proc/1/root/var/run/secrets/kubernetes.io/serviceaccount/, tab completion isn't a nice-to-have, it's the only sane way to get there.

Pre-Configured Aliases

The shell comes with aliases that save keystrokes when every second counts:

k           # kubectl
kgp         # kubectl get pods
kgs         # kubectl get svc
kdp         # kubectl describe pod
kl          # kubectl logs
ll          # ls -alh

Small things, but they add up fast when you're running dozens of commands in a session.

tmux, Because One Pane Is Never Enough

Here's a scenario: you need to watch tcpdump output in one pane, tail application logs in another, and have a free shell for running ad-hoc commands in a third. With a regular kubectl exec session, you'd need three separate terminal windows, each running its own kubectl exec command.

With tmux in the debug container, you split one session into as many panes as you need.

Getting Started with tmux

tmux is pre-installed and ready to go. Just type:

tmux

You're now inside a tmux session. Here are the essentials:

# Split horizontally (top/bottom)
Ctrl+B, "

# Split vertically (left/right)
Ctrl+B, %

# Switch between panes
Ctrl+B, arrow keys

# Create a new window (like a tab)
Ctrl+B, c

# Switch between windows
Ctrl+B, n    # next
Ctrl+B, p    # previous

# Detach from tmux (session keeps running!)
Ctrl+B, d

# Reattach to a running session
tmux attach

Why tmux Matters for Container Debugging

Persistent sessions. If you're using daemon mode (sleep infinity), you can start a tmux session, detach from it, close your terminal, go grab coffee, come back, kubectl exec back in, run tmux attach, and everything is exactly where you left it. Your tcpdump is still capturing. Your log tail is still running. Your command history is intact.

Parallel workflows. A typical debugging layout might look like:

┌─────────────────────────┬─────────────────────────┐
│                         │                         │
│  tcpdump -i any         │  tail -f /proc/1/root/  │
│    port 27017           │    var/log/app/error.log │
│                         │                         │
├─────────────────────────┴─────────────────────────┤
│                                                    │
│  ~ ❯ dig my-service.default.svc.cluster.local     │
│                                                    │
└────────────────────────────────────────────────────┘

Top-left: live packet capture. Top-right: application logs. Bottom: your working shell with Oh My Zsh, autosuggestions, and all the tools. All in one kubectl exec session.

Shared debugging. If two engineers need to look at the same pod, they can both kubectl exec into the debug container and tmux attach to the same session. One person types, both see the output. Pair debugging inside a Kubernetes pod, try doing that with a bare sh shell.

A Complete Debugging Session

Let's put it all together. Your pod is having issues and you need to investigate.

Step 1: Start the Debug Container (Daemon Mode)

# Start in background with a known container name
kubectl debug -n <namespace> <POD-NAME> \
  --image=docker.io/k7bdevs/debug-container:latest \
  --target=<container-name> \
  --container=debugger \
  --share-processes \
  --profile=general \
  -- sleep infinity

# Open a shell — no need to look up the container name
kubectl exec -it <POD-NAME> -n <namespace> -c debugger -- zsh

Step 2: Explore the Target Filesystem

# You're automatically in /proc/1/root (target's filesystem)
ls
# bin  data  etc  home  lib  usr  var ...

cat etc/app/config.yaml
ls -la data/

Step 3: Run Target Binaries with t

t mongosh
# Connecting to: mongodb://127.0.0.1:27017
# Using MongoDB: 7.0.31

t bash
# Drop into the target's own shell environment

Step 4: Network Debugging

# Check DNS resolution
dig my-service.default.svc.cluster.local

# Check listening ports
ss -tlnp

# Capture traffic
tcpdump -i any port 27017 -w /tmp/traffic.pcap

# Test connectivity to another service
curl -v http://some-service.default.svc:8080/health

Step 5: Process Debugging

# See all processes
ps aux

# Monitor resource usage
htop

# Trace system calls
strace -p 1 -f -e trace=network

# Check open files
lsof -p 1

Step 6: Check Kubernetes Resources

k get pods -n <namespace>
k describe pod <POD-NAME> -n <namespace>
k logs <POD-NAME> -n <namespace> --tail=50

Note: The debug container uses the service account of the main pod to authenticate with the Kubernetes API. If that service account doesn't have RBAC permissions to list or view resources (pods, services, etc.), these commands will fail with a Forbidden error. The debug container doesn't bring its own credentials, it inherits whatever access the pod already has.

Step 7: Clean Up

# Just exit — the debug container stays alive (daemon mode)
exit

# When truly done, the debug container lives until the pod is deleted/restarted
# There's no need to explicitly clean it up

Key Concepts Summary

Why Not Just kubectl exec?

The debug container gives you a fully-equipped workstation attached to the pod, without touching the running application.

Built with debug-container, a purpose-built Kubernetes debug sidecar image.