BLEAKFEED · all stories

Our Standby Generator Starts Itself Every Night at Twenty Three Minutes Past Eleven, Runs for the Length of a Real Outage, and Its Log Says It Took the Load, and the Grid Was Fine

Read in episode 057, Terrifying Data Centre Horror Stories Before Bed, from 14:14

The generator is tested once a month, on a schedule, for ten minutes, and it logs every start with the reason. Every night at twenty three minutes past eleven it starts with the reason code for a mains failure. The mains has not failed. It runs for four hours and eleven minutes, takes the load, according to its own log, and hands it back, and the fuel is gone, and four hours and eleven minutes is the length of the only real outage this building has had.

I am the critical power engineer at a data centre and the generator is the thing that means my phone can ring at three in the morning.

The generator.

A diesel engine the size of a lorry in a room at the back, with a tank, a transfer switch, and a controller that starts it when the mains goes and takes the building's load in under ten seconds, which is the number the customers are sold. It is tested monthly. Every start is logged with a reason code: test, manual, mains fail.

At twenty three minutes past eleven it starts on mains fail.


The run.

The mains is fine. The grid monitor shows the grid. The controller logs a mains failure anyway, and starts, and the engine runs, which I can hear from anywhere in the building, and the transfer switch logs a transfer to generator, which it has not done, because the building is still on the grid, and the load meters on the generator side show the building's load, four hundred kilowatts, on an engine that is connected to nothing.

Four hours and eleven minutes. Then the controller logs mains restored, transfer to mains, cool down, stop. And the fuel is gone. Four hours of diesel, every night, out of a tank I have to have refilled, for an outage that is not happening.


The number.

Four hours and eleven minutes was the outage. 2015. A substation fire in the town, the grid gone at twenty three minutes past eleven, and the generator had started, and it had taken the load, and it had taken it in twelve seconds, not ten, because a battery in the starter had been weak, and in those two seconds the UPS had sagged and a rack of a customer's kit had rebooted, and the customer had been a bank, and the two seconds had cost a sum that was in a report.

The engineer on shift that night had been the one who had signed the battery off the week before. He was not blamed, officially. He blamed himself, the way they do, and he left the industry, and I have not been able to find out what happened to him after, and I have a feeling I do not need to.


What has been checked.

The controller, replaced. It logs the mains fail at twenty three past.

The generator room, at twenty three past eleven, by me. The engine started. Not the way it does on a test, which is a controlled wind up. The way it does on a real failure, hard, the emergency start, the starter motor screaming and the engine catching and the whole room shaking, and the controller display showing MAINS FAIL, and the seconds counting, and at ten it showed LOAD ACCEPTED, and it was ten seconds. Not twelve. I have watched it fifty nights. It has been ten seconds every night. It has been, I think, practising.


What is done about it.

The fuel is on the budget as generator testing and the finance office has asked why we test for four hours a night and I have said we do not, and the diesel supplier has asked whether we are running a second building off it, and I have said no.


What I do.

I run critical power.

The battery. That is what it comes down to. A weak starter battery, signed off, and two seconds. The batteries are checked weekly now, by a machine, with a printout, and it was that night that made the rule.

I go to the generator room at twenty past eleven. I stand at the controller. When it starts, at twenty three past, hard, on mains fail, I watch the seconds, and at ten, when it shows LOAD ACCEPTED, I say, into the room, over the engine, ten seconds, that's in spec, and I sign the test sheet, a real one, on the clipboard by the door, with the time and the start time and the reason, and in the reason I write, mains fail, simulated, and in the result I write, pass, and I sign it, and I put his initials next to mine, from the 2015 sheet, which is in the file.

The run has got shorter. Four hours eleven, for the first year. Three hours forty last month. It is coming down. I think it is coming down by the length of time it took, that night, for somebody to say to that engineer that ten seconds would have been fine and twelve was not his fault, which nobody said, and I am saying it every night to a diesel engine at twenty three minutes past eleven, and the diesel engine, in the way that a machine can, is starting to believe me.

Also in this episode