I was sitting at an outside table on a footpath in lower Manhattan, waiting on a breakfast I never ate, when the first plane went into the North Tower of the World Trade Centre at 8.46am on 11 September 2001.

The table faced the towers. On it sat a Sony digital camera, then still an oddity in a press pack that shot normally film. This was the first review of a digital camera and despite poor storage it passed with flying colours.

That camera produced some of the first images of the attack published in Australia, filed one hour over a dial up modem from a cafe after I had staggered into and out of the dust.

The cafe was deserted doors wide open and money still in the till.

Twenty five years on, the human toll is what the world remembers, and rightly.

But I spent the following ten days inside Ground Zero on a New York Police press pass, and what I watched there was something the technology industry has never properly reckoned with.

It was the largest infrastructure failure in the history of commercial computing, and it exposed a disaster recovery orthodoxy that was, in hindsight, close to worthless.

The story that never got written

I was in New York at the invitation of a retail software company looking to push into the Australian market.

At the time I owned CRN, iTNews, InformationWeek and Windows Magazine, and the hottest argument on the block was the dot com build out and what online retailing would do to physical stores.

These are original images shot with a first generation Sony digital camera from where I was sitting having coffee.

 

My  interview was scheduled for a downtown hotel in the financial district, close to the New York Stock Exchange.

I was staying uptown. It was a beautiful morning, so I walked the 45 minutes and arrived early.8.46am

I heard the roar before I saw anything. A few minutes later the first aircraft hit, directly above a retail concourse held on a long term lease by Australia’s Westfield.

I had spent years on Fleet Street in London writing about crime and terrorism. I did not need the second aircraft to tell me what I was looking at, but it came anyway, into the South Tower, and it shocked me and the rest of the world.

It was late evening in Australia. I called my wife in Sydney. Then the training took over.

As the towers came down in front of me, the thought that would not leave was this: hundreds of businesses were being erased along with thousands of people, and nobody watching had any idea yet how much of corporate America had just gone offline.

Getting inside

Having worked New York before, I knew the credential I needed. That afternoon I went to 1 Police Plaza and applied for a New York Police press pass, the document required to cross military and police lines. It was granted after vetting by the NYPD, the FBI and US security agencies.

At 5.55pm on 11 September 2001 I entered Ground Zero. Within hours the story had reframed itself. Not how many were dead, which nobody could yet say, but how authorities intended to replace the entire technology base of tens of thousands of employees at some of the largest companies in the world, and how the New York Stock Exchange, by then ringed by US military, was going to be brought back to life.

Working with Stuart Kennedy, Michael Sainsbury and Ben Woodhead back in Australia, we started filing a series of technology stories that ran for weeks.

The failure was not the data

Here is the part the industry has been quietly comfortable misremembering.

Most large firms did not lose their data. Overnight tapes were offsite and they survived. What broke was everything around the data.

The standard of the day was building to building, campus to campus, or across the Hudson to Jersey City. It looked like geographic separation on a slide. In practice it meant the same power grid, the same telco central office, the same transport network and the same pool of staff. Some firms in one tower are reported to have run their recovery site in the other. When both went, both went.

9/11 did not invent cloud computing. What it did was destroy the assumption underneath local and near local backup, that physical proximity was a cost saving rather than a correlated risk. Every argument for offsite, provider operated, geographically distant infrastructure that the industry spent the following decade making was validated in a single morning.

Wall Street’s problem was underneath the street

The NYSE never opened on 11 September. The first plane hit 44 minutes before the bell. What followed was a four day shutdown, the longest since 1933, and the exchange did not trade again until 17 September.

The restoration was not a damage problem. Eleven Wall Street was four blocks from the towers and structurally untouched. It sat inside the police frozen zone and it was coated in dust, so the floor needed cleaning, filtration and air testing. That was the easy part.

The hard part was below the pavement. Verizon’s central office at 140 West Street, adjacent to WTC 7, became the single biggest technical casualty of the attack. When 7 came down, steel and debris punched into the building and the basement cable vaults flooded. Hundreds of thousands of voice lines and millions of data circuits went with it.

That is the detail that should still be taught. Firms with intact data and intact staff could not trade, because the connection between them and the exchange no longer physically existed.

What followed was, by my reckoning, the largest technology rescue ever attempted. FEMA moved in on distributor inventory, servers and networking gear were air freighted in over the following days, kit was stood up inside shipping containers, and cable was run into buildings off the street. Resellers, the unglamorous end of this industry, did the work alongside federal agencies.

One observation from those ten days has stayed with me. IBM’s Lotus Notes proved materially easier to restore at scale than Microsoft Exchange. Resellers rebuilding tens of thousands of seats cursed the Microsoft product for failing to surface the information they needed to do the job. A quarter of a century later, anyone who has tried to recover a broken Microsoft 365 tenancy will recognise the complaint.

The device that won the week

Mobile voice networks in New York collapsed under load on the day. BlackBerry’s store and forward data service running over the Mobitex network did not. Devices were issued to US congressional staff shortly afterwards.

RIM’s credibility with enterprise and government, the foundation of the business it became, was substantially made that week. It is one of the few commercial stories to come out of 11 September, and it happened because one company had designed for a network that assumed failure.

Twenty five years on

The lesson of 9/11 for this industry was never really about towers or tape. It was that resilience you have not tested against the loss of an entire metropolitan block is not resilience, it is a line item.

Most of what the industry now sells as high availability still rests on assumptions nobody has stress tested since. We have simply moved the correlated risk from two buildings in lower Manhattan to a handful of hyperscale regions, and told ourselves the problem is solved.

I watched the old assumptions die in front of me. I am not convinced we replaced them with much better ones.