Had a fun time with the servers today. Starting sometime yesterday we had amazingly heavy traffic on the servers–well at least for us it was heavy. This has happened in the past when we’ve been Slashdotted, but this was different. We had a lot of comment spammers and bots all hitting the sites at once. Things got real, real slow last night and by this morning everything was at a crawl.
After some customer support calls though we managed to get the pipe bumped up to 3 MBits/sec. It took about four hours for things to settle down and after a handful of reboots and deleting of huge log files most of the sites came back up faster than ever. Whew.
TabletPCPost was the only unfortunate victim. Whatever was hitting it, managed to overload the system. So we deleted some extraneous session stuff, rebuilt the theme settings and whala got it back running again. It took the better part of the morning and afternoon.
At least it’s done. I wonder if anyone will notice the speed difference.