Log Rotation with Mongrel

We’ve built a couple of Rails applications that are handling a lot of traffic. Along the way, we’ve learned a few tips that help them scale. I’ll be sharing them over the course of the next few weeks. The first is about handling Log Rotation with Mongrel.

During the first month our application was running, we occasionally ran out of disk space on our main servers. When our mongrels couldn’t write their log files, we started seeing 502 errors. I did a bit of googling and found a few recipes on the web for rotating log files. Unfortunately, none of them worked. They all relied on sending signals to mongrel to ask it to re-open the logs. This seemed to be a hit or miss proposition.

After reading through the log files for logrotate, I found the copytruncate option. Copytruncate copies the existing log file, then truncates the log in place. This means that you don’t have to worry about sending signals to mongrel. It also means you might loose a few log entries. In our case, this is an acceptable risk to take.

To configure this, I add a file in /etc/logrotate.d for each application. That file looks like:

/u/apps/application/shared/log/production.log { rotate 12 missingok compress copytruncate }

That tell logrotate to keep 12 log files, to use copytruncate and to compress the log files after rotating them. Next, I add a script to /etc/cron.hourly to call logrotate:

#! /bin/bash logrotate /etc/logrotate.d/application

That’s all there is to it. Our logs get rotated and we don’t worry about disk space anymore. We could keep more copies of the logs if we wanted to, but we’re generating almost 5Gig of logs per day, so 12 hours is plenty for us.

Coming Wednesday: A book announcement