Detect and restart hanging programs with Go - Pipe Cleaner

Detecting programs where the standard output has frozen can require a deep dive into terminal emulation basics. Go plumber Mike Schilli builds a plunger to free up the pipe works.

- Mike Schilli

If the browser stops halfway while loading a website and then freezes, experienced users know that usu-ally all it takes is clicking the reload button to make it work like clockwork on the next attempt. Or, if an rsync transfer suddenly stalls because the server has fallen asleep, admins will intuitively press Ctrl+C to abort, only to immediately restart the command and see it finish without a hitch in most cases. Scenarios where humans have to manually take control over running processes just because the computer fails to understand that an automatic restart would solve the problem are one of the last hurdles to a fully automated world.

The yoyo Go program presented in this issue supervises programs entrusted to it and will rejigger them like a yo-yo (as you know, yo-yos also need to be pulled up by a human hand to keep them moving). To do this, the utility monitors a supervised program’s standard output (along with its standard error), which typically features frequently changing patterns – such as a progress bar that indicates that something is still happening. If the display freezes, for example, because of a network outage or because the server has lost interest, yoyo detects this and restarts the program after a configurable timeout, in hopes of somehow fixing the problem this way.

Feels Like a Terminal