The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”
In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
hackermailman
Odd that they didn't have at minimum 1) a physical Emergency Power Off (EPO) button inside the datacenter, such that one button could de-energize everything -- usually required as part of building a datacenter due to code variance for the "low voltage" (<4000V in this context...) wiring. 2) EPO interlink to the smoke alarms, such that it would de-energize equipment automatically (maybe with delay) if smoke detected.
rdl
Interesting. The Sequence of Operations we write for the data halls I work on specifically do not shut down the CRACs on smoke alarm for this reason. The International Mechanical Code (IMC) specifically allows this where a air handler over 2000 cfm serves only one space because it's recognized that it will not spread smoke around a building. 2000 cfm is the threshold where you typically have to install smoke detectors, and typically they do shut down on smoke detection.
The CRAC won't feed a fire oxygen as it's just recirculating the same oxygen depleted air.
rainbowzootsuit
The episode of their podcast where Ron explores the facilities side of their data centers and offices is worth a listen too
Great story, I love those in-the-trenches emergency stories. I wonder if it would have impacted Jane Street in any meaningful way, aside from a fat bill, if it ended with submarine servers. They could have diverged their compute to another datacenter surely.
We mostly hear about engineers from Jane Street here, and not much from their traders and finance people. I wish we could have some other perspective sometimes.
They publish a great podcast, Signal and Threads, I really recommend.
thibsrn
Datacenters on the coast never made sense to me.
Reminds me of DirectNic's datacenter flooding during Katrina (New Orleans), we all kept posting on Somethingawful until they finally shut it down.
bluedino
Are there internet connected fire alarms that can be hacked to destroy the contents of some data centers? Like the European freezers?
sroussey
I did not know that data centers with sprinklers exist. I guess this is a very stupid idea in the first place.
If there is a fire the decision is: all equipment might be lost in a fire OR all equipment will be lost by water damage.
andix
I wonder why the answer was to turn off the servers instead of turn their cooling back on. Would it not be able to stabilize the temperature fast enough?
comments (10)
“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”
https://news.ycombinator.com/item?id=4720388 (https://web.archive.org/web/20121101194314/http://status.fog...)
https://arstechnica.com/information-technology/2012/11/how-o...
reticulates
hackermailman
rdl
The CRAC won't feed a fire oxygen as it's just recirculating the same oxygen depleted air.
rainbowzootsuit
https://signalsandthreads.com/the-thermodynamics-of-trading/
jddj
We mostly hear about engineers from Jane Street here, and not much from their traders and finance people. I wish we could have some other perspective sometimes.
They publish a great podcast, Signal and Threads, I really recommend.
thibsrn
Reminds me of DirectNic's datacenter flooding during Katrina (New Orleans), we all kept posting on Somethingawful until they finally shut it down.
bluedino
sroussey
If there is a fire the decision is: all equipment might be lost in a fire OR all equipment will be lost by water damage.
andix
sciurus