Debug a running distroless container

Recently, I was messing with some stuff on the production server and I came across a running container that neither had bash nor sh, or any other shell so I couldn’t get inside the root file system and debug what was going on with the service. That was a bust at that particular moment, but I did not see any technical reason why that wouldn’t be possible to do. First things first, who would have deployed such a container? why and when?

Locked container from Unsplash

Distroless containers

It’s no secret, that distroless containers are gaining quite a bit of traction in the last few years since not only they are small which often reduces the cost of running them, but also they are much more secure since the less stuff your container has, the fewer stuff hackers can use to exploit it.

Distroless images are very small. The smallest distroless image, gcr.io/distroless/static-debian11, is around 2 MiB. That's about 50% of the size of alpine (~5 MiB), and less than 2% of the size of debian (124 MiB).

You can think of it as a slimmed-down Linux distribution image plus the application runtime, resulting in the minimum set of binary dependencies required for the application to run. I mean, do you really need tools like grep, ls, or bash in your production container image? In most if not in all cases the answer is NO.

But as mentioned in the beginning, that also imposes an issue, especially for people that already worked with things like microservices. We just love to open a shell inside an already running container, check its files or networking configuration, and burn a new container image out of it. But what about now, like is such a thing even possible?

Debugging a running container can mean several things, and I will just focus on viewing root file system and modifying while the container is running. It’s actually fairly rare you’ll do anything like this but I found it an interesting learning experience.

In Theory

If you know nothing about containers and the terminology I suggest giving a try to this amazing Learning series, created by Ivan Velichko. From here on I’ll consider you understand a thing or two about the containers and wonder what goes on behind the scenes. A far most important feature of containers is Linux namespaces.

Linux namespaces

As per Wiki: Namespaces are a feature of the Linux kernel that partitions kernel resources such that one set of processes sees one set of resources while another set of processes sees a different set of resources.

With containers is mostly all about the isolation and self-containment that is achieved by utilizing several types of (resource) namespaces:

• PID or process namespace enables to have multiple entirely isolated sets of processes that form process trees. To give you an example, every time a computer with Linux boots up, it starts with just one process called the root, with process identifier (PID) 1. On the other hand, the PID namespace allows one to spin off a new tree, with its own root process. The process that does this remains in the parent namespace (in the original tree), but makes the child process the root of its own process tree. With PID namespace isolation, processes in the child namespace have no way of knowing the parent process’s existence. However, processes in the parent namespace have a complete view of processes in the child namespace, as if they were any other process in the parent namespace.

PID namespaces

• network namespaces enable to have multiple networking environments, where each can have a separate interface, routing table, firewall rules, network interfaces, and IPv4 and IPv6 protocol stacks. It’s like having multiple Network Interface Cards (NIC) in a PC but only this time we have one, but virtualize multiple on top of it.
• mount namespaces isolate the list of mount points seen by the processes in a namespace. In other words, each mount namespace has its own list of mount points, meaning that processes in different namespaces see and are able to manipulate different views of the root filesystem.

Mount namespaces

• user namespaces maintain allows a different set of isolated processes have distinct user permissions. Each namespace inherits its permissions from the user who created the new user namespace.
• control group namespace (cgroups) limits and isolates the resource usage like CPU, memory, disk I/O, and network of a collection of processes.
• IPC, UTS, and others

According to this, if we are the root on the server where the container is running, we should be able to access these child namespaces, since the parent namespace should see it as a subspace.

In practice

As is common nowadays, I will create a Hello World script in golang and pack it into a distroless container image. The two scripts are as follows:

and a simple multi-stage Dockerfile to package the binary:

As a base, we took a distroless Debian, to build a container image with only our go scripts and its runtime dependencies. Many other distroless images are available here if you want to have a look.

Build and run the container. Since the container doesn’t have any shell utility, you can’t attach to it and you can verify that yourself.

Root Filesystem

Since the container is nothing other than a group of running processes, we should be able to see its metadata through the /proc/<PID> . But before that, we need to figure out what’s its process ID (PID). We can find this information using:

ps faux

which should output running container info somewhere at the bottom.

Then you will find the root file system of the container at /proc/<PID>/root . Note that you need to be a root for that and if you make changes to the files in this directory, it will also apply to the running (container) process itself.

If you’re just getting started with containers and want to optimize them, have a look at this great tool wagoodman/dive.

Conclusion

That's it for today's post, having such a trick or two in the bag can make you look really awesome in front of your colleagues, especially with containers that are here to stay at least for some time. Note that, if you are using the containers for projects where performance and security are not your main priority, just don’t overcomplicate your life and use whatever container base you feel is necessary. The difference is actually really tiny, but on a large production scale, it aggregates so keep this in the back of your mind.

Thanks for reading! :) If you enjoyed this article, hit that clap button below 👏

Would mean a lot to me and it helps other people see the story. Say Hello on Instagram | Linkedin | Twitter

Do you want to start using Medium? Use this referral link 🔗

If you liked my post you can buy me a Hot dog 🌭

Follow me for more related content, cheers!