The hot aisle is the side of the racks where the exhaust comes out, and it is forty degrees, and sensors every metre map it so we can see a failed fan before a server does. Every night the map shows a cold spot. Person sized. It moves down the aisle at the pace of somebody walking, stops at each rack for a few seconds, and goes on, and at the end of the aisle it stops for longer, at the rack where the map has always had a warm patch that no sensor explains.
I am the facilities engineer at a data centre and the thermal map is the screen I would save in a fire.
The hot aisle.
The racks face each other in pairs and between the backs is a contained aisle where the exhaust goes, forty degrees, loud with fans, and a sensor every metre along it at three heights, so the map on my screen is the aisle in colour, red, and a failed fan is a redder patch and I see it before the server does.
At one every night the map has a blue patch.
The patch.
Twenty two degrees, in an aisle that is forty. The size of a person, by the sensors it covers, from the floor to about a metre eighty. It starts at the door end. It moves. One rack, stop, a few seconds, next rack, stop. The pace of a person walking an aisle and looking at each rack as they go, which is a thing engineers do, or did, when you could still tell a server's health by its lights.
At the far end it stops for about a minute at the last rack, which is the rack the map has always shown a little warm, half a degree, at one spot at the bottom, that I have never been able to find a cause for.
Then the patch goes. Not back up the aisle. Goes.
What has been checked.
The sensors, replaced along the whole aisle. The patch walks the new sensors.
The aisle, at one, by me, inside it, in the heat and the noise, with the map on a tablet. The blue patch was at the door end. It came towards me. I stood still. It got to me and the heat went. Not down. Went. I was in twenty two degrees in a forty degree aisle, for about four seconds, with the fans roaring, and then the heat came back, and the patch was past me, at the next rack, and I watched it go down the aisle on the tablet and I watched it in front of me, in the sense that I could see the exhaust air shimmer differently where it was.
Who.
The warm spot at the bottom of the last rack. Half a degree. I took the rack apart, which took a night, and at the bottom, behind the blanking panel, where nothing should be, there was a badge. A lanyard, the old company's, with a photo, and the plastic had gone soft in the heat over the years and the photo was faded but it was a man, and the name was on it.
He had worked here when it was the old company's site. I asked. The building manager from then is still around. He said the man had done the night walk, every night, the hot aisle, rack by rack, checking lights, which they did then, and had, one night, not come out, and had been found at the far end, at the last rack, on the floor, and it had been the heat, they thought, or his heart in the heat, and the aisle had been forty degrees and nobody had heard him over the fans.
The badge had come off his lanyard when they lifted him. Nobody had looked for it.
He is doing his walk. Every night. Rack by rack. Cold, because the heat is what took him. And he stops at the end, where he stopped.
What is done about it.
The blue patch is on the thermal log every night and the customer whose racks are in that aisle has asked why their exhaust temperature drops for four seconds at each rack in sequence at one in the morning, and I have said, airflow, and they have accepted that, because airflow is the thing nobody can argue with.
What I do.
I look after the building.
The rule now, which was not the rule then, is that nobody goes into a hot aisle alone, and that anybody in one carries a radio, and that the door is checked every fifteen minutes. It is a good rule. It is his rule, in a way, because it was written after him.
So I am his second. At one, when the patch starts at the door end, I go in. I walk behind it. Rack by rack. I have the radio. I say, at each one, when the patch stops, what you say on a walk, which is the rack number and, lights good, and the patch goes on, and I follow it. At the far end it stops at the last rack for its minute and I stop with it, and at the end of the minute I say, come on, out we go, that's the round, and I open the door at the far end, which is the door he did not get to, and the cold of the cold aisle comes in, and the patch goes through it, onto the map of the cold aisle, where it is not blue any more, because it is the same temperature as everything, and it is gone.
I have put his badge back on its lanyard, in a bag, in the office, on the hook where the night walk's radio hangs, and the radio is on top of it, and I take the radio at one and put it back at two, and the badge is under it, which I think is where a badge should be.