The Server Nobody Could Turn Off

Most offices that have been running software for more than five years have one. A machine somewhere it shouldn't be. Under a desk, on top of a filing cabinet, in the store room next to the spare chairs. Power cable, network cable, a thick layer of dust. Nobody remembers buying it.

This is a story about one of those. It's true. We've changed the details that would identify the company, and nothing else.

Nobody knew what it was

About eighty staff. Eight years of buying software from four different vendors, then two in-house developers, then no in-house developers, then a new one last year. Normal history. Nothing about the company was badly run.

They wanted to move to the cloud, so the first job was a list of what they actually had. Their IT guy walked us round the office and stopped in front of a beige tower in the corner of the accounts room. Sticker on the side from a shop that closed in 2019.

He said he doesn't touch that one. We asked why. He said because the last guy said don't touch it. That was the entire handover. One sentence, passed down from a person who left four years ago, and everyone had obeyed it since.

It had grown into everything

Nobody planned for that box to matter. It started as a file server. Somewhere in a shared folder there's still a scan of a 2016 delivery order for it.

Then it kept getting jobs. A scheduled task that pulls the day's sales out of the POS and pushes them into the accounting system. The licence server for the design team's software. The print queue for the warehouse labels. A script that emails the stock report at 7am. And a small service that syncs customers to their cloud CRM.

Every one of those was an afternoon's work by somebody who no longer works there. None of it got written down, because at the time it was obvious. That's the whole mechanism. Things are obvious to the person doing them, and the person leaves.

The Saturday they pulled the plug

We asked for one hour with it switched off, on a Saturday, with everyone told in advance. Everyone agreed it was a file server and a file server is a weekend nothing. Their IT guy pulled the plug at 9am.

The warehouse noticed first. Saturday is a half day for them and by 9:20 no carton could get a label. Around ten, two designers who came in to finish a deadline found their software refusing to open. No licence server, no software.

The rest surfaced on Monday. Saturday's sales never reached the accounting system, so finance started the week with a hole in the numbers. And the cloud CRM, the one that lives on a server in Singapore and costs them a subscription every month, had no new customers from Saturday at all. The counter staff had typed them in. They landed in a queue on a beige tower in Puchong that had no power.

One hour off. A day of firefighting, a week of finance reconciling, and a customer list that had to be retyped from paper slips. And this was the controlled version, on a day we chose, with people standing next to it. Imagine the power supply going instead, on a Tuesday in December.

Then there was the line that said sleep(3)

Inside the sync script sat one line. After pushing a batch of orders and before marking them as sent, the code waits three seconds. No comment above it. No ticket. No name in the commit. Just a pause.

Their new developer found it while cleaning up. Perfectly good instinct. Three seconds a batch across a few hundred batches was making the nightly job about twenty minutes slower for no visible reason. He deleted it, ran the job, watched it finish clean, and moved on.

Two weeks later finance came down. Some invoices had gone out twice. A handful of customers existed twice in the CRM. Same order, two rows, timestamps a few seconds apart.

The three seconds were holding a race shut

Two things wrote to the same table. The sync pushed the order out, and the CRM fired a webhook back to confirm it. When the confirmation arrived before the local record had finished committing, the lookup found nothing, decided this was a new order, and created it again.

Most nights that gap is a few hundred milliseconds and you'd never see it. On a slow night, or a big batch, or when the office internet is having a bad evening, the confirmation wins the race. Then you get two of everything.

Somebody hit this years ago, almost certainly at 2am with finance waiting. They tried a pause. The duplicates stopped. They went to bed. That's a reasonable thing to do at 2am.

What never happened was the next morning. Nobody went back to fix the actual cause, which was a missing unique constraint and an order of operations that assumed one thing finishes before another starts. Nobody wrote a line above it saying why. So the fix sat there for six years doing real work while looking exactly like rubbish somebody forgot to clean up.

This is what technical debt actually looks like

When people hear technical debt they picture messy code. Bad names, long functions, the stuff a developer complains about. That's the cheap version, and honestly you can live with a lot of it.

The expensive version is what this company had. The system works. The reason it works has left the building. The code is still there and the knowledge isn't, so every part of it is now a thing you can operate but can't safely change. You can keep the lights on. You can't move.

That's why the cloud migration they quoted at six weeks took four months. Two of those months were archaeology. Not building, not migrating. Finding out what the machine did by watching what broke when we took pieces of it away, in a copy of the environment where breaking things was allowed. We've written about where that bill comes from in technical debt: the silent tax on your business.

How a well-run company ends up here

Nobody made a bad decision. That's the part worth sitting with. Every step was sensible on the day.

The "don't touch it" rule is what a company invents when it has stopped understanding its own system. It's not laziness. It's the only safe rule left when nobody can predict what a change will do. The same instinct shows up in code, which we got into in the developer who needs a translator.

What it cost, in numbers they recognised

The owner asked us to put a figure on it, so we did. Rough, but not far off.

The last one is the biggest and the only one with no invoice attached. You never find out what you didn't build.

What to do before your version of this happens

None of this needs a big programme. It needs a few hours and somebody with the authority to ask annoying questions.

That last one is the cheapest insurance in this whole business and almost nobody buys it. Thirty minutes on a Tuesday morning would have saved this company two weeks of duplicate invoices six years later.

Where it ended up

The box is gone. Everything it did runs somewhere with a name, an owner and a page of notes. The sync has a proper unique constraint now, so there's nothing left for a three-second pause to protect. The whole write-up is shorter than this article.

What it took to get there was permission to break things on purpose, in a safe copy, with the budget to spend two months finding out what the company already owned. That's a hard thing to sign off on, because it produces no new features. It's also the only way anyone was ever going to touch that machine again.

Your version of that box exists. Somebody in your office walks past it every day. The real question is whether you'll meet it on a Saturday you chose, or on the morning its power supply dies by itself.