Easy logging with logrotate and supervisord

| by Doody Parizada

Hi, I’m Doody, a senior backend developer here at Rounds. Today I’d like to present a simple way for efficient, long log rotation.

At Rounds we use supervisord to keep our servers up and running. Supervisord is a daemon that monitors selective apps, restarting them according to specific rules e.g. when they crash.

One feature supervisord provides is log handling. By default supervisord 3.0 will keep your stdout and stderr logs trimmed down to 50Mb before it rotates it to a log.1 file and keeps up to 10 backup logs. This can get you up and running with no fuss. Great!

Then came the time where we wanted more control over our logs. We needed to keep a really long backtrace on the machines and gzip the backup files, so that they don’t take too much disk space. Supervisord doesn’t give you much control over log rotation, but logrotate does.

Logrotate is a standard UNIX application that allows automatic rotation, compression, removal, and mailing of log files. We wanted to share our simple supervisord + logrotate setup that allows you to disable supervisord’s log rotation feature and replace it with logrotate’s:


[program:my_app]
directory=/opt/%(program_name)s
command=/opt/%(program_name)s/run

stderr_logfile=/var/log/supervisor/%(program_name)s_stderr.log
stdout_logfile=/var/log/supervisor/%(program_name)s_stdout.log

stdout_logfile_maxbytes=0
stderr_logfile_maxbytes=0

stdout_logfile_backups=0
stderr_logfile_backups=0

%(program_name)s is one of the template arguments supervisord passes into the file and is evaluated against a python dictionary.
The stderr_logfile and stdout_logfile set a specific file name instead of supervisord’s auto generated one. Setting x_maxbyte=0 disables limiting the log size, and x_backups=0 disables rotation backups.

Now we setup logrotate with the following configuration:

/var/log/supervisor/my_app_*.log {
daily
rotate 60
copytruncate
compress
missingok
notifempty
}

This config will rotate our log daily and keep 60 compressed backups without limiting the file size.
Logrotate is run by crond. We recommend having a default setup for logrotate that runs daily.
Make sure that if you want to rotate more than once a day, you will also need to setup logrotate to run more than once a day in your crontab.
Enjoy your new logging setup!

Category: ,

Have something to say?