Many of Project Wonderful’s servers are located in NYC, right there on Manhattan island. Before hurricane Sandy hit, we, along with our on-site hosts, took standard emergency preparedness procedures: making sure generators were on-site and fueled, making sure we had emergency contacts in place with our suppliers, and so on.
It’s helped, but we’re not out of the storm yet.
Before Sandy hit, we had enough diesel on site to ensure they could run 65 hours without needing more fuel. At around 8pm last night power was knocked out to one of our data centers and we switched over to diesel without interruption. Unfortunately, this data center then experienced severe flooding a few hours later, which has submerged and destroyed the pumps used to send fuel up to the generators, held above flood level. Here’s the notice we got:
The flooding has submerged and destroyed the site’s diesel pumps and is preventing fuel from being pumped to the generators on the mezzanine level. The available fuel reserves on the mezzanine level are estimated to support customer loads for approximately 5-7 hours. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost.
The building itself is being evacuated and no remote hands support will be available to assist in any equipment shutdown. Life safety is our number one priority and we are making plans to completely exit the facility. No customer access to the building is possible at this time either. Due to the evacuation, Internap will not be able to provide any exact updates on when the fuel will be exhausted and critical customers loads lost, but as noted, we believe it will take place in approximately 5-7 hours from now.
So unless access can be restored, we may lose some servers in the next few hours. In addition, another one of our datacenters, while still running off of diesel, has had power interrupted twice overnight: this was responsible for the slowdowns and sometimes unresponsiveness that you may have seen from Project Wonderful between 3 and 6 this morning. The databases in that datacenter have been now restored to full functionality and should be fine going forward, but if there is another power interruption you may see similar slowdowns until the machines are brought back up to speed.
Currently, 7 of our 9 webservers are fully operational, and our database cluster is operating as expected. We have enough redundant servers that losing a few of them does not impact performance, and as long as we can keep those servers up, you should not notice any issues with your Project Wonderful account. If anything seems wrong, please let us know in our service email, and be sure to follow us on Twitter for up-to-the-minute status updates.
Thanks for your understanding!
UPDATE OCT 30 11:07 AM: No further issues have been encountered, however, the datacenter with the fuel tank underwater and fuel pump destroyed is on the edge of shutting down:
Our DC Ops personnel are actively engaged in sourcing fuel supplies from vendors in the local area. As of now we continue to run on the available fuel reserves on the mezzanine level could stop supporting customer load at anytime. Once this fuel supply has been exhausted the generator will no longer be able to sustain operation and critical customer power loads will be lost.
We’ve elected to keep things up as long as possible, and we may be able to keep running even if this data center goes dark by shifting load to other centers. However if we are unable to continue serving ads, then we will suspend charging until we can recover, so nobody will be forced to pay for service that is not being reliably provided.
Unlike most other advertising networks, we offer a more robust asynchronous code option. This causes our ads to load in parallel to your site. That way, if anything affects us, your sites don’t slow down at all. It’s now the standard code we provide, but some of our older members may still be using non-asynchronous code. You can update your code by logging in, going to “My ad boxes” and selecting “code”.
UPDATE OCT 30 11:59 AM: Those generators have now failed, but load has been shifted to other datacenters. Project Wonderful remains up.
UPDATE OCT 31 6:00 AM: Two datacenters that provide fiber connectivity to the datacenter we’ve got our database machines in have lost power, which means we’re currently offline. We’re working hard to get things back up and will post as soon as service has been restored.
UPDATE OCT 31 1:00 PM: The servers are up, but we’re still working on getting a signal through to them. More information will be posted when we have it, and again we apologize for the downtime. If you find your site is loading slowly, we’ve put up a post on how to switch to the asynchronous code during this downtime that will remove any impact this service interruption has on your site.
UPDATE OCT 31 2:00 PM: Connectivity has been restored to the servers. Two servers are still offline, but we’ve build enough redundancy into the network that this isn’t a problem. We are back online!
UPDATE OCT 31 3:08 PM: One of our two currently-offline servers has been restored. These missing servers have had no negative impact to Project Wonderful, but they do add redundancy to the network. It’s nice to have them back :)
UPDATE OCT 31 3:23 PM: And now, all servers are back up and running! We’re back. We’re still running on diesel, but we’ve got enough fuel to last us. Hopefully there will be no more unexpected interruptions. Thank you for your patience and understanding so far.
UPDATE NOV 5 7:00 AM: All servers but one are off generator power and running on utility power again. We anticipate no further interruptions, but will post here when the final server has been taken off generator power. This may take several days.
UPDATE NOVEMBER 9 11:30 PM: All servers are now back on utility power. Hooray!