Today's network outage

This morning, most of our users were banned from the network. This was the result of my error and I am truly sorry.

I had added a spamfilter to the network to try to block a swarm of bots that were joining the network. Since the first filter did not work, I removed it and added a second one. This occurred at 9:11am EST. Promptly, most of our users were removed from the network with the message "Spambots not wanted". This also affected most of our network staff.

Visitors to the website and our forums trying to find out what happened brought the webserver to its knees. Unfortunately, with the webserver unresponsive, most of our server admins were not able to rehash their servers and add exemptions for themselves so they could work on correcting the issue.

Thanks to network staff, the issue was resolved completely by 10:22am EST.

This incident highlighted several different things:

  • Sometimes we all make mistakes.
  • Server admins should be exempted from spam filters so they can quickly address any issues with filters that arise.
  • Servers should not be dependent on the webserver to rebuld their configurations.
  • The webserver needs better hardware to handle an onslaught of visitors.

We are currently working on how to approach the technical issues.

I am going to take steps to make sure I do not repeat this. For starters, I am going to avoid doing anything with the spam filters until I have a reliable mechanism for testing them first.

Again, I'm truly sorry this happened. Please accept my apologies.