Coincidence? Maybe

Once upon a time, a long, long time ago, in the late 1970s and early 1980s to be precise, I was a computer programmer/analyst. It was the wrong career choice for me; I’ve bounced out of high technology three times now, and landed hard—very hard—each time. And I don’t job hunt well.

But one of the lessons I learned was that if something went wrong once, it mightonly might—be a freak occurrence. If it happened twice, I didn’t need to wait for a third time; something was indeed really wrong. Chief Engineer Montgomery Scott’s words once, when left in command of the U.S.S. Enterprise on the original Star Trek thus resonate: “Fool me once, shame on you. Fool me twice, shame on me!”

Something like that happened today. I woke up this morning and not wanting to disturb my cat, I grabbed my smart phone and started to browse my email. There were two email alerts from the Wall Street Journal (three more would arrive later) indicating that the New York Stock Exchange (NYSE) had halted all trading. This, in my recollection, is kind of unusual. I figured “the shit was hitting the fan,” but in the absence of details, I didn’t know what.

I decided this was an excellent time to fill my gas tank (just in case) and I was also low on coffee, so I headed out, thinking of two possibilities. First, it might be contagion from the Chinese stock market, which is resisting all government efforts to stem a decline that’s looking a lot like the crash in 1929.[1] The second possibility was a technical glitch.

The argument against contagion from China is that other markets did not halt trading. The argument against a technical glitch is that United Airlines and the Wall Street Journal were also affected. In addition, it turns out that Seattle’s 911 system went down, albeit briefly.[2] Recall my threshold: I don’t need three occurrences; I only need two. And in this case, I’ve got four, in what I presume to be four fundamentally unrelated systems. Also, it turns out that the NYSE outage is, by far, the longest of its kind in history.[3]

Allegedly, “[t]he culprit was later determined to be a router” in the United Airlines shutdown.[4] I have not yet seen an explanation for the NYSE or the Wall Street Journal outages. I’m less troubled by the Seattle 911 outage because it was brief and, frankly, if it hadn’t happened concurrently, or at least roughly so, with the others, the Wall Street Journal and Seattle 911 outages probably wouldn’t have reached my threshold for newsworthiness.

My advice for the NYSE would be that they should be forthcoming about the cause of their outage. The less I know, the more I suspect it was security-related, and corporations are notorious for trying to cover up security problems on their systems. And given the coincidence, I’m not sure I buy the United Airlines explanation either; it would be much more convincing if their outage hadn’t happened so close to the others.

There’s a fine line between speculation and paranoia here and I’m trying to stay on the speculative side of this. The fear of over-reaction is probably what’s silencing folks who have a lot more expertise in this than I do. But there are questions here that won’t be satisfactorily answered by a CNN interview with “a spokesperson from the Department of Homeland Security [who] said that there is “no sign of malicious activity” in either the United or NYSE troubles.”[5]

Update, July 8, 2015: Since this post was originally published, I have found a New York Times story in which it is claimed the problem was due to a software update that contained a bug. The times, however do not add up:

In the silence that followed, traders on the floor, looking for information, flocked to a ramped area of the huge complex on Wall Street where employees of the exchange are located. The traders were told that the exchange had been updating software before the trading day began, and that the software had contained a bug that could not be fixed without shutting down the system.

The exchange’s employees on the floor proceeded to manually cancel around 700,000 trades that were in the system when it was shut down, according to [Peter P. Costa, a trader on the floor with Empire Executions]. Once they had finished, the exchange rebooted its system, a process that took around 45 minutes, Mr. Costa said.[6]

I can imagine it might indeed take a while to cancel 700,000 trades, but 45 minutes for a reboot seems excessive on a mission-critical system.

For what it’s worth, and given the source, I’d say it isn’t worth much, I’m not alone in my suspicion:

“Well, I’d like to know some more information,” Sen. Charles E. Schumer (D-N.Y.) told reporters at the Capitol. “To have three outages in three important places in the same day raises a lot of questions. I haven’t gotten answers yet.”

Asked about the “coincidence” of three major technical failures in a single day, FBI Director James Comey told the Senate Judiciary Committee that the incidents had “obviously that caught my attention.”

“We’re not big believers in coincidence either. We’ve been in contact with all three companies to see what’s going on,” Comey said. “We do not see any connection to a cyber breach or a cbyer attack…It does appear that it’s not a cyber intrusion.”[7]

Update, July 9, 2015: The New York Stock Exchange publishes a market status history which seems to offer something of an explanation for what happened.

  1. [1]Timothy B. Lee, “China’s stock market crash, explained,” Vox, July 8, 2015, http://www.vox.com/2015/7/8/8908765/chinas-stock-market-crash-explained; RTT News, “Asian Shares Tumble On China Worries,” July 8, 2015, http://www.rttnews.com/story.aspx?Id=2520319; Jeremy Warner, “The really worrying financial crisis is happening in China, not Greece,” Telegraph, July 8, 2015, http://www.telegraph.co.uk/finance/china-business/11725236/The-really-worrying-financial-crisis-is-happening-in-China-not-Greece.html
  2. [2]Adam Chandler, “The Day the Computers Betrayed Us,” Atlantic, July 8, 2015, http://www.theatlantic.com/business/archive/2015/07/computer-glitches-united-stock-market/398018/; Katherine Krueger, “NYSE Resumes Stock Trading After Blackout Lasting Nearly 4 Hours,” Talking Points Memo, July 8, 2015, http://talkingpointsmemo.com/livewire/nyse-suspends-all-securities-trading
  3. [3]Darla Cameron, “The NYSE just had its longest computer-problem related closure ever,” Washington Post, July 8, 2015, http://www.washingtonpost.com/blogs/wonkblog/wp/2015/07/08/the-nyse-just-had-its-longest-computer-problem-related-closure-ever/
  4. [4]Adam Chandler, “The Day the Computers Betrayed Us,” Atlantic, July 8, 2015, http://www.theatlantic.com/business/archive/2015/07/computer-glitches-united-stock-market/398018/
  5. [5]Adam Chandler, “The Day the Computers Betrayed Us,” Atlantic, July 8, 2015, http://www.theatlantic.com/business/archive/2015/07/computer-glitches-united-stock-market/398018/
  6. [6]Nathaniel Popper, “The Stock Market Bell Rings, Computers Fail, Wall Street Cringes,” New York Times, July 8, 2015, http://www.nytimes.com/2015/07/09/business/dealbook/new-york-stock-exchange-suspends-trading.html
  7. [7]Drew Harwell, Thad Moore, and Jacob Bogage, “NYSE resumes trading after unprecedented shutdown,” Washington Post, July 8, 2015, http://www.washingtonpost.com/business/economy/nyse-trading-has-been-halted/2015/07/08/46b51974-2588-11e5-b72c-2b7d516e1e0e_story.html

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.