Community Server for Rent a Coder

4/28/2006 10:00 AM: Peak 10 (our data center colocation provider and ISP) is currently experiencing a network issue which is causing the site to be intermittently unavilable to you at times. Early indications lead them to believe this is a denial of service attack. At this point there is no reason to suspect it is targeted at us (versus the other thousands of companies in their data center). They are working to resolve it and additional information will be passed on to you once we know more.

11:39 AM: Peak 10 appears to have stablized and recovered from this attack.  For the last 15 minutes the extreme packet loss has stopped and access to the site is continuous again.  DOS attacks are notoriously hard to detect and stop, so if it is indeed fixed, I give them credit for doing it so quickly.

As an aside, this is the first outage we have had with Peak 10 since we colocated with them since last year.  The physical connectivity, facilities and the customer service have been outstanding.

11:48 AM: Received notice from Peak 10 that the outage has ended.

Dear Valued Customer,
Please be advised that Peak 10 engineers have resolved the following
impairment:
Date:                   4/28/2006
Ticket#:                75506
Start of Impairment:    09:35 EDT
End of Impairment: 11:35 EDT
Data Center(s):         All
Summary:   Network
Peak 10 network engineers were able to isolate the network targeted by the denial of service attack and connectivity has been restored.  As part of Peak 10's ongoing effort to deliver the highest quality service available, we will continue to investigate the circumstance associated with this event and will publish a detail of this impairment within the next two (2) business days.
It is Peak 10's mission to provide the highest level of service to our customers. 
We appreciate your business and look forward to continuing our relationship in the future.
Sincerely,
The Peak 10 Solution Support Team
If you need assistance, the Peak 10 Solution Support Center can be reached

1:57PM: Received notice that it was another customer at Peak 10 being targeted by the DOS attack, and not Peak 10 themselves.  They diverted the traffic and stopped the attack after figuring out who it was.

COMPLETED: Microsoft has fixed a # of bugs in SQL Server in an updated called Service Pack 1 (SP1).  This was installed on Monday night (April 25th) from 8pm-1am Tuesday night.

This upgrade contains about 20-30 fixes including one that helps the CPU (which is what is causing problems on the server right now).

During 10:30pm EDT till roughly 1:30am EDT, certain pages on the site (that access the database) timeout or do not display completely.  The problem is caused by a workaround/fix recommended to us by Microsoft to solve a memory problem with SQL Server 2005 during the backup window. The workaround was to turn off the "verify" feature of the backup, which stops the memory from being completely sucked up.  However, this means we now have to do the verification on a 2nd server by copying it to another server.  During this process, the copy also sucks up all the memory. It is a catch 22 situation at the moment.  We are working with Microsoft to get a resolution for this secondary problem as quickly as possible and thank you for your patience.

The site database was upgraded in February to a quad processor 64 bit server running the lastest version of the database software (Microsoft SQL Server 2005) to improve it's performance.   Unfortunately the upgrade to the new Microsoft SQL Server 2005 has caused the site to timeout it two ways:

1) 5/5/2006:RESOLVED with SQL SERver 2005 SP1

The new database is not correctly optimized due to a bug in SQL Server 2005's optimization wizard that causes it to crash on certain "database views".  This causes general sporadic timeouts and slowness in the database.  Microsoft has confirmed this as a bug and we are working with them to get a patch issued to us as quickly as possible. (3/16: Microsoft has created a fix, but testing it will take 3 weeks to make sure it doesnt break something else.  We are anxiously awaiting this fix and will update you when it comes in).

2) 5/5/2006RESOLVED: Moving the backup to a seperate drive fixed this problem. 

During the database backup between roughly 8:15 PM till a little after 9PM.  This should not be happening as the database should allow a backup to occur with stopping access.  A warning message has been added to the site during this time period and a 2nd ticket has been opened with Microsoft to resolve this bug.  (3/16: We have updated the server with 15k SCSI drives (the fastest type for servers) to lessen the backup window.  Microsoft has promised to see if they have any information on how a 24/7 site is supposed to do backups and best practices/etc..   We will update you when we know more.  4/5: No new info from Microsoft on ways to improve this yet...still pushing this issue with them. Also discovered memory issues with backups which has been referred to them.).

Update 4/30/2006:  This problem is due to a flaw in Microsoft's ASP (having to do with include files) that fragments the memory and there is no way to correct it without running into other problems which are more serious.  Short term we have doubled the memory on each server and bought a 3rd internet server for the Rent a Coder pool.  These servers are rebooted daily to keep the memory as defragmented as possible.  This problem now appears only once every two-three weeks instead of many times a day.  The long term solution is that we will be moving eventually to ASP.NET which does not have this problem.

--------------------------------------------

Description: The internet server will sometimes report the below error along with running out of virtual memory.  The parts in grey will be different from time to time, but the black portions will always be the same:

Microsoft VBScript runtime error '800a01b6'
Object doesn't support this property or method: 'xxxxxxxxxxxx'
/CrossWeb/include/BannedIpAddress.asp, line 369

Frequency: The problem occurs randomly.  Rebooting the server "fixes" the problem but is only a temporary solution. 

Status: There is currently a ticket open with Microsoft to resolve this issue.  It is caused by a flaw in Active Server Pages (ASP) that only triggers in very high volume situations.  Currently the server is being rebooted every day to attemp to minimize the times it appears.