Every rack in a data centre is metered, because power is the bill and heat is the enemy. Rack C17 is empty. Its power meter reads four hundred watts every night from about eleven to about six, which is one old server's worth. The sound logger over the row hears fans. The temperature sensor at the top of C17 reads heat rising off nothing. And a host name that was decommissioned in 2016 answers a ping from that rack's switch port.
I am the night operations engineer at a colocation data centre, which is a warehouse full of other people's computers, and C17 is the rack that keeps me honest.
The rack.
Forty two units of empty steel in row C. The customer that had it left in 2016 and took their servers and the rack was left, powered but unused, because it is cheaper to leave a rack powered than to have an electrician isolate it. It has a meter, as every rack does. The meter reads zero all day.
At eleven it reads four hundred watts.
The four hundred watts.
It is a number I know. It is what a particular model of server from about 2010 drew at idle, a two unit box with two power supplies and six fans, and the customer that left in 2016 had a row of them, and the last one out of C17 was one of those, and its host name was in the decommissioning ticket, and I have the ticket.
The power comes on at eleven and goes off at six. It ramps the way a server ramps on power up, a spike and then a settle. The sound logger, which is a microphone on the row for the noise compliance people, records fans in C17 for those hours. The right fans, by the pitch. The temperature sensor at the top of the rack reads a rise of two degrees, which is what that box put out.
And the switch port that C17 was patched to, which is still patched, because nobody unpatches, shows a link. Up. At a gigabit. And when I ping the old host name, from the management network, it answers. Sub millisecond. From C17.
What has been checked.
The meter, swapped. It reads four hundred watts at eleven.
The rack, at midnight, by me, with a torch and a thermal camera. Empty steel. On the thermal camera, in the middle of the rack, at the units where the box was, a warm shape. Two units high. The shape of a server. Warmer at the back, where the exhaust would be, with the plume rising off it to the sensor at the top, and I put my hand into the rack, into the shape, and it was warm, and there was air moving over my hand, backwards, the way a server pulls it, and the fans were loud, the way they are when you are that close, and there was nothing my hand could touch.
I pinged it while I was standing there. It answered.
Whose.
The customer's tickets from 2016 are in the system. The decommission was done by one of their engineers, on a night shift, alone, which was the way they did things, and the last ticket he closed was C17, host down, powered off, unracked, and the timestamp was ten to six.
The next ticket on his queue was never opened. I asked the customer's old account manager, who is at another company now, and she said the engineer had been found in his car in our car park at seven that morning, and that it had been his heart, and that he had been forty one, and that C17 was the last thing he had done.
He powered it off. Then he died. And every night at eleven, when his shift started, it comes back on, and runs till six, when he would have shut it down, and it answers to its name, because it is his.
What is done about it.
C17 is on the power report every night as an unmetered load and the finance people have asked who is paying for it and I have said nobody, and they have said then turn it off, and I have said it is off.
What I do.
I run nights.
A server that is up should be monitored. That is the rule. So I added it. The old host name, in our monitoring, as a host, on C17's port, with the checks a server gets: ping, uptime, temperature, and the dashboard shows it, green, every night from eleven to six, up, and red from six to eleven, down, and the graph of its uptime is the shape of a night shift, every night, for the year I have watched it.
I have also done what you do for a host that is running with no owner, which is put a comment on it in the monitoring, and the comment field says, owner, and I have put his name, from the ticket, and the account manager gave me the rest of it, and the dashboard shows a server that has been running unrackedly for nine years with a dead man's name against it as owner, and when it goes red at six the alert goes to me, and I acknowledge it, and the acknowledgement text is, shift over, and I have not been able to think of anything better.