Log Rotation with Mongrel
During the first month our application was running, we occasionally ran out of disk space on our main servers. When our mongrels couldn't write their log files, we started seeing 502 errors. I did a bit of googling and found a few recipes on the web for rotating log files. Unfortunately, none of them worked. They all relied on sending signals to mongrel to ask it to re-open the logs. This seemed to be a hit or miss proposition.
After reading through the log files for logrotate, I found the copytruncate option. Copytruncate copies the existing log file, then truncates the log in place. This means that you don't have to worry about sending signals to mongrel. It also means you might loose a few log entries. In our case, this is an acceptable risk to take.
To configure this, I add a file in /etc/logrotate.d for each application. That file looks like:
/u/apps/application/shared/log/production.log { rotate 12 missingok compress copytruncate }
That tell logrotate to keep 12 log files, to use copytruncate and to compress the log files after rotating them. Next, I add a script to /etc/cron.hourly to call logrotate:
#! /bin/bash logrotate /etc/logrotate.d/application
That's all there is to it. Our logs get rotated and we don't worry about disk space anymore. We could keep more copies of the logs if we wanted to, but we're generating almost 5Gig of logs per day, so 12 hours is plenty for us.