ATMs failed in Idaho, Wyoming delayed lottery results, and 911 call centers in Washington, Arizona, Missouri and other states struggled with busy signals, dropped calls and missing location information.
At the Northern Colorado Medical Center in Greeley, staff couldn’t access vital patient records online. And in parts of New Mexico and Montana, Verizon faced service disruptions through no fault of its own.
Press reports have linked a long list of troubles to network problems suffered by telecommunications company CenturyLink, based in Monroe, La., two days after Christmas.
For about 30 hours, from the early morning hours of Dec. 27 until late on Dec. 28, chaos reigned on CenturyLink’s system. Western states that depend most heavily on the company’s fiber-optic system were hardest hit, but reports of outages and slower speeds came in from Alaska to Florida, according to downdetector.com.
“CenturyLink experienced a network event on one of our six transport networks beginning on December 27 that impacted voice, IP, and transport services for some of our customers. The event also impacted CenturyLink’s visibility into our network management system, impairing our ability to troubleshoot and prolonging the duration of the outage,” the company said in a statement.
Technicians were left scrambling trying to pinpoint the root cause, and that resulted in them losing time on fixes that didn’t work. New Orleans as ground zero was an early suspect, and then it was San Antonio, Texas. Teams, which had to make physical site visits, went into action in Kansas City, Mo., and then Atlanta, and so on.
But as they tried fixes in different areas, the problem didn’t go away. Making matters worse, the reporting system that gathered customer complaints also failed.
The source of all that turmoil and hours of angst for affected customers came down to one piece of equipment — a faulty third-party network management card in Denver, according to the company.
But how could one bad piece of equipment in Denver disrupt internet and phone service in large swaths of the country and impair critical services to thousands of customers for hours on end? And could it happen again?
Those are two questions the Federal Communications Commission, which has launched an investigation, wants answered, not to mention state utility regulators, computer scientists and irate customers.
A Sorcerer’s Apprentice
Provided by Walt Disney ProductionsMickey Mouse leads an enchanted broom through the segment for “The Sorcerer’s Apprentice.”
In the classic Disney film “Fantasia,” Mickey Mouse casts a spell on a broom to get it to carry the water buckets that he, as the apprentice, is using to fill a cistern for the sorcerer, who has just left the room.
Mickey then falls asleep and things go horribly wrong. The broom carries way too much water. Waking and realizing his predicament, Mickey tries to smash the broom to pieces. But the splinters turn into dozens of new brooms, carrying hundreds of buckets of water. The chamber gets flooded.
Computer scientists borrowed the term “Sorcerer’s Apprentice Syndrome” to describe what happens when a part of a network sends out “packets” of bad information …read more
Source:: The Denver Post – Business