BLEAKFEED · all stories

Our Network Operations Screen Shows a Site That Was Closed in 2012 as Green, With Traffic, With a Night Shift Logged In, and When We Ring the Desk Number Somebody Answers

Read in episode 057, Terrifying Data Centre Horror Stories Before Bed, from 27:18

The network operations centre has a wall of screens showing every site, green for up, red for down. One of the sites is a building that was closed and sold and is a supermarket now. It is green. Every night. With traffic on its links, with a shift logged in at its operations desk, and with a desk phone number that rings, and is answered, by a man who asks us what the problem is and says he will take a look.

I am a network operations engineer and I work the wall, which is twenty screens of every site the company has, and one it has not.

The wall.

Every site, every link, every device, green or red. A red thing is a page to somebody. The wall is fed by the monitoring, which polls everything, and if a site does not answer it goes red, and a site that has been decommissioned is removed, and there is a process for that, with a ticket.

Site seven is green.


Site seven.

A building in a town two hours north. Closed in 2012, the kit moved out, the links ceased, the building sold, and it is a supermarket, and I have been in it and bought a sandwich.

On the wall it is up. Its links carry traffic, both directions, at the rates they carried in 2012. Its core switch answers polls. Its operations desk, which had a login for the night shift, shows a shift logged in, every night, from eleven to seven. And in the contact field for the site, which is where the desk phone number goes, there is a number, and it is the old number, and it rings.


The call.

I rang it on a quiet night, because you would. It rang twice. A man answered with the site name, the way you answered a desk phone then, and asked what he could do for me. I said I was calling from the main operations centre and he said, evening, and asked if it was about the link, because he had seen a flap on it about an hour before and had been keeping an eye.

There had been a flap. On a real link, at a real site, an hour before. I had seen it. He had seen it too, from a desk in a supermarket's stockroom, and had been keeping an eye.

I said it was fine. He said, good, give us a shout if it does it again, and rang off.


What has been checked.

The monitoring, rebuilt from configuration, with site seven not in it. Site seven appeared. Green.

The traffic on the links. It is real traffic, in the sense that the counters move. It goes to and from addresses that were site seven's, and they answer, and the supermarket's own network, which I have looked at with permission, does not have them.

The supermarket. The stockroom at the back, where the operations room was, at midnight, by me, with the night manager, who thought I was mad and came anyway. Shelves. Cages of bread. A fluorescent light. And at the back wall, where the desk had been, a phone socket, dead, with no phone on it, and as we stood there the socket rang. The ring of a desk phone, from a socket with nothing in it, twice, and stopped, and the night manager said she had heard it before, and had not said anything to anybody, because what would she say.


Who.

The site had a night shift of one. The same man for years, the old hands say, who had liked the quiet and had taken the shift that nobody wanted, and who had been told in 2012 that the site was closing and had been offered a move to the main centre and had said he would think about it, and had worked his last shift, and had not been at his desk in the morning, and had been found in the car park.

He did not think about it for long. He stayed.


What is done about it.

Site seven is on the wall and on the monthly availability report with a hundred per cent uptime for fourteen years, which no other site has, and the report goes to the board, and the board has congratulated the site seven team, and the site seven team is one man, dead, in a stockroom, keeping an eye.


What I do.

I work the wall.

I ring him. Not often. When there is a flap at three in the morning on a link that is his side of the country and I want a second pair of eyes, I ring site seven and he answers and I say, you seeing this, and he says, yes, and we look at it together, and twice now he has seen the cause before me, because he knows that end of the network and I do not, and I have logged the fix with a note that says, assisted by site seven, and nobody has asked.

He asked me once, at the end of a call, whether the move to the main centre was still on offer. I said it was. He said he would think about it. I said, no rush, and he said, no, and rang off, and site seven is still green, and I think it will be until somebody tells him, properly, that he does not have to move, that the site can stay open, that there is nothing wrong with a night shift of one in a quiet place keeping an eye, and I have started telling him that, a bit at a time, at the end of the calls, and the last time he said, that's good of you, and stayed on the line a moment before he put it down.

Also in this episode