Wednesday, August 19, 2009

Fixing the site downtime issue

Every 4-6 weeks, I get an email or a tweet that lets me know that Running-Log.com is down. I've known the root of the problem for a while and was usually careful in not letting it happen. That's failed me the last two times, though, so it was time to finally automate the fix.

The site is allowed 5GB of disk space on the server. If that gets full, then bad things happen. In particular, the database shuts itself down and refuses to start again. Every time that someone visits the site, some things get noted in some log files and these files grow over time. Eventually, they fill up the available disk space and the site goes down.

The solution to this is log rotation. Instead of keeping one log file, you keep one log file per day. You keep a week's worth of logs and delete the oldest one each night. Once this is setup, you've only got a week's worth of logs at all times, which will generally be about a constant size. The size of the log files will no longer balloon and take up all the disk space.

I had tried setting up log rotation previously but it wasn't working. This morning I investigated why it was failing and remedied the issue. I'm happy to report that it's working now, so the site shouldn't go down periodically anymore.