Earlier today, the Tucson Citizen blog went down and we lost today’s photos. This was my fault – oops! Total downtime was a little over an hour. It is mostly restored now as I write this.

How this happened: Mark Evans pointed out a problem with a new blog that didn’t have any of the proper plugins loaded and I decided that the simplest way would be to disable the sitewide plugins and then re-enable them (just in case other blogs had a similar issue). Everything worked great until I attempted to disable our caching plugin. Apparently, deactivating that specific plugin causes files to be deleted. I have no idea why. The result of this disaster was a whopping 137,000+ deleted blog files (approximately 11GB) plus a White Screen Of Death (WSOD).

Fortunately, we keep really good backups (i.e. two mostly complete site backups), so very little was lost. Specifically, if you are a Tucson Citizen blogger, you lost any photo you posted after 1 a.m. this morning and will have to re-upload it. One backup is being left turned off until tomorrow just in case something else went horribly wrong that we don’t know about yet so that we can still recover from this. If you spot a problem that wasn’t there before, please let Mark Evans know.

Needless to say, the blog that had the problem still has the same problem. My “fix” didn’t fix it. Sigh.

On a positive note, this has been an actual, real-world test of our website backups. Not that anyone should ever have to find out if their backups actually work. But everyone here is very glad that they do work.

This entry was posted on Tuesday, October 25th, 2011 at 6:45 pm and is filed under Technical Glitches. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.