Eschew the rsyslogd PID file

Eschew the rsyslogd PID file

This PID file is not required, provides questionable operational value, and can break logging in a Discourse application container.

On startup, rsyslogd will read and self-terminate if it finds another process on the system with the same PID as that which was written to this file. This behaviour is especially problematic when running in a containerised environment:

  • The processes that make up a container are more likely to be terminated without grace. Any PID files persisted to the container’s filesystem will become stale after an unclean shutdown. (Separately, even when signalled with SIGTERM on graceful shutdown, rsyslogd will still fail to unlink this file.)

  • PIDs on Linux are assigned sequentially. When run in a unique pid namespace, a container’s process table is subject to little entropy. Thus, PID ‘collisions’ across container instantiations are not unlikely.

Altogether, it is easy for rsyslogd to DoS itself on startup by mistaking another process in the container (e.g.: nginx) for an existent rsyslogd process. Unlinking this file guarantees a clean startup.

Newer releases of rsyslog support -iNONE, but – of course – this feature is not supported in the rsyslog distribution included as part of Ubuntu 16.04:

rsyslog/ChangeLog at 527f19c56a80fd30354f32ad03bdacc1275f4aa8 · rsyslog/rsyslog · GitHub

Vixie crond and nginx also employ PID files, though neither is vulnerable to this failure mode. Vixie cron wraps the fd with a flock; the flock is used for mutual exclusion, not the underlying file itself. nginx does not appear to use its PID file for mutual exclusion.

diff --git a/image/base/rsyslog b/image/base/rsyslog
index 10b09f2..ecc41d4 100755
--- a/image/base/rsyslog
+++ b/image/base/rsyslog
@@ -3,4 +3,5 @@ exec 2>&1
 cd /
 chgrp syslog /var/log
 chmod g+w /var/log
+rm -f /var/run/
 exec rsyslogd -n

GitHub sha: 7ebdf2b8