I run Arch linux on my laptop and every now and then I run a little webserver for personal projects. Lately, I have been playing around with PHP. I always thought PHP was not very usable, perhaps because it is not scalable in its pure form. To maintain proper process and data management between different PHP files and using databases managed by MySQL or the likes, using PHP quickly turns in a job not for one man alone. Until I discovered the Laravel framework. It enables you to build a HTML5/PHP/Javascript app or website in a few days or even hours.
With this tool available to me, I thought: would it not be awesome to run a server where people could play around with a Python environment to develop their programming skills? So I started my project. Not soon after, I realized that this idea does not come free of any security concerns. If users of my small little webserver were to execute arbitrary scripts on the webserver, then they easily get access to files, processes, and could even perform system calls to the OS running the webserver. Think of simply doing `import os` followed by `os.system(‘rm *’)`. This is just a little script that would wreak so much havoc on a webserver if not properly sandboxed.
Of course, I thought to myself: this is easy to jail in. So I started to run a little Docker-in-Docker (dind) server as part of my webserver. Whatever code a user may run, it will be safely put inside a separately running docker process. I believed my server was securely protected against even quite skillful malicious actors.
Until I read one little comment somewhere reading something like “no way that running a server where people can run Python scripts can be done securely; think simply of a script that never terminates”.
How could I have missed this: a denial-of-service (DoS) attack!
while (True): print(42)
Suppose someone would run this program on my `safe’ webserver. This process would print 42 indefinitely. Woops!
No way, I thought to myself. I had bounded memory usage to 64MB in the docker command. The program would crash before it could take up all memory of the server. Still, I wondered how my implementation would respond to such a malicious or unskillful user? So I tried it.
All seemed to go well; it’s just that the front-end of the app only got stuck in a neverending little loading animation. All seemed OK, this was something I could probably protect against. I just have to make sure that the Laravel process manager that runs the Python docker command executing the user’s script is protected with a timeout. So I set out to fix it.
After I was done fixing this security issue, I realized my ventilation had been going crazy for a couple of minutes now. The CPU was using all its resources, memory was running out, and even available disk space was shrinking quite rapidly. Something was wrong in my system.
I checked `ps ux` to list all running processes, but in a quick glance, nothing seemed out of the ordinary. So I checked `top` to show CPU and memory usage per process. My third worst nightmare was going to come true: my operating system was going to crash completely.
Then I remembered, I left this process running myself! By executing the neverending script on the webserver, the script was still running on my local machine (which was running the webserver in this development stage)…
So I just sudo kill‘ed the process. No effect. The process was still alive. I wondered why storage space was running out. Probably because docker mounts some part of the disk for virtual storage, and output (an infinite amount of 42s) was stored to some buffer. Weird how the output buffer was not bound by the maximum of 64 MB of buffer space. Also, weird how sudo kill did not manage to kill the process. The only thing to do was…
Reboot.
The process was gone.
What did I learn? DoS attacks are real easy if users can just execute any script on the server without timeout control, even when contained in a Docker image.
Leave a reply!