Server Downtime

Comments (6)

• 

  Administrator

  Well, it seems that the trouble may have been some sort of Denial of Service attack on our primary database server. We’re still investigating exactly what went on, but we THINK we’ve got the primary services back up and running now. More in the morning.

  July 19, 2011 at 2:40 am
• 

  ListingWare Admin

  OK…so, the previous info seems to hold true: there was some sort of rogue Denial of Service attack wandering around the Internet yesterday and our primary database server was affected.

  Denial of Service (DoS) attacks are meant simply to overwhelm server resources in attempts to either gain access to a server or prevent others from gaining access to it. Our server was not compromised to intruders – it simply got overwhelmed in a way that caused several of the large databases on it to become corrupted, causing the server to stop abruptly several times.

  This caused some pretty serious disk errors. It’s basically like an 18-wheeler slamming on brakes while doing 70 MPH on the freeway: lots of speed, lots of momentum, lots of chances to lose control.

  The bad news: as you saw yesterday, we were down much of the afternoon and some into the evening.

  The good news: it seems that all of the databases were recovered successfully with no apparent data loss.

  We are still checking the server thoroughly and will keep updating this post with what we find. Sorry again for the downtime.

  July 19, 2011 at 12:32 pm
• 

  ListingWare Admin

  Things have been intermittent all day – mostly up, but had to take the server down for a while in the afternoon to run some low-level hardware checks. Tomorrow morning, barring anything major, I should be able to send out an email to everyone with more info on the last two days of trouble.

  July 20, 2011 at 2:45 am
• 

  ListingWare Admin

  OK, things have been up and down again today – mostly up, but I know the downtime is tough to deal with. We’re going to try to make it up to you. Details to come soon.

  July 20, 2011 at 8:12 pm
• 

  ListingWare Admin

  We did a few more tweaks last evening and then a few more this morning. There were some periodic reboots for new fixes to take effect, but not long ones.

  We’re going to (hopefully) leave things alone and let them run today so you can have as little interruption as possible.

  July 21, 2011 at 1:45 pm
• 

  ListingWare Admin

  OK – fingers crossed that we’re pretty much back to normal. In fact, we’ve even been able to improve a few things. We’ll try to have a final wrap-up in the morning.

  July 21, 2011 at 10:11 pm

Comments are closed.