As of Thursday, October 13th, the traffic lights in Minneapolis were still having trouble.
On Sunday, Oct. 9th, a transformer blew out and brought down the computer that runs the Minneapolis traffic lights. Timers in the street kicked in to provide basic service, but by the following Thursday the central system was still not running. Five days is a long time.
Clearly out of options, they sent me an e’mail.
It turns out that the Minneapolis traffic light system is managed by a VAX system. VAX computers were made by Digital Equipment Corporation, which was purchased by Compaq, which was purchased by Hewlett-Packard. All of this, years ago. VAXen are old machines.
Twenty years ago, I ran VAXen. I had one on my desk in Chicago and one in San Francisco and one in New York. Back then, I really knew how to run a VAX, and I still have it on my résumé, and that’s how they found me. They did a web search for someone—anyone—with VAX experience who was nearby. I’m in St. Paul these days, so I’m nearby.
What is even more amazing about this story is that I was able to help them out. I called a friend who called a friend who called the folks in Minneapolis. My friend’s friend still runs VAXen. Over the ‘phone he was able to guide them through the steps necessary to boot the machine from the CD-ROM. From there, they were able to figure out that the boot drive had failed, but that the data drive was in good shape. Minneapolis does have backups. They’ll get through this. All they need is a new hard drive and someone who knows how to rebuild a boot drive from tape.
One thing to note about VAX computers: They run forever. One of my colleagues guessed that the boot drive had probably failed years ago, but that no one had noticed because the machine has not had to be rebooted in that long. Having worked with these systems, I’d say that’s probably a good guess. Also, there is still a company out there that is supposed to be providing support for this system. It’s just that the crash happened on Sunday, and Monday was Canadian Thanksgiving, and the support company is based in Canada. If your job is to support systems that almost never fail, you may not have considered that one would fail on your holiday.
There will be a lot of people who will whine and complain and claim that Minneapolis messed up here. They’re wrong. Kudos to the folks in Minneapolis Traffic Engineering. Here’s what they have done for their city:
- They saved taxpayer’s money by maintaining and running a completely sufficient system instead of replacing it every few years for no good reason. If it ain’t brokie, why pay to fixie?
- They did have a backup plan and it went into effect. The traffic lights are working, they’re just not running at 100%. Backup tapes, a service contract, and a fail-over system form a completely rational contingency plan.
- When the normal and correct plans of action did not produce results, they got creative. They found a solution by going to the community. Okay, so I’m across the river in St. Paul. My mayor is friends with their mayor and that’s good enough for me. I’m not sure where the expert we eventually found actually lives. Possibly in Minneapolis, but who cares? It’s a Democracy, folks, and that means we are all in this together.
Sure, they could have had a full-time VAX specialist on staff and a complete set of spare parts, but keep in mind that state funding to Minnesota cities was cut a few years ago. Who’s going to provide the money to pay for that level of support? Certainly not the current legislature.
For my small part, thank you Minneapolis Traffic Engineers for allowing me to be of service.