GREENFIELD — As far as NineStar Connect was concerned, the only way to make up for a bad day on Feb. 7 was to refund some money – quite a bit of money.
A core server crash followed by a targeted attack by hackers that caused three disruptions over the next two days prompted the regional communications and electric cooperative to credit just over $84,000 back to customers in March.
“It was the perfect storm,” said David Spencer, NineStar director of marketing. “We had a core server that didn’t indicate it was failing and then did not fall back to the redundancy systems; then there was the DDoS attack.”
DDos stands for Distributed Denial of Service. Such an attack occurs when multiple hackers attempt to overload a system with so much information that it can’t handle legitimate traffic, and service is disrupted.
The storm started at about 12:35 p.m. Friday, Feb. 7, when a core router began to “flutter” at NineStar’s Maxwell campus.
Because the breakdown was not a single, catastrophic event, the backup systems didn’t catch the imminent failure and take over for the malfunctioning router, company officials said.
Service to the company’s Internet, phone and cable television customers was down for a little over four hours while technicians removed and replaced the router, which also required moving some 100 fiber connections in the network, according to a Feb. 10 letter to customers.
Later that night, the company’s network came under an attack that targeted some of its serving routers by trying to overload them. Subsequent attacks occurred on Saturday morning, Saturday evening and again Sunday morning.
Though the company said the outages were caused by equipment failures and an outside attack, it opted to reimburse its customers, “who are our friends and neighbors,” Spencer said.
“That’s how we differ from other people,” he said.
The week after the outages, NineStar’s residential Internet, telephone and cable television customers were notified they would receive a 10 percent credit on their March bills for the service interruptions.
The company’s large institutional clients, such as Hancock Regional Hospital, generally operate under a service agreement that specifies the parties’ rights and obligations, including reimbursements for disruptions.
The credits for those customers filled a broad range, from $110.42 for the Hancock County Public Library to $13,520.77 for the hospital.
County Auditor Robin Lowder said the county government’s total reimbursement came to $252. Government offices essentially were shut down for the afternoon by the disruption.
Though Feb. 7 was a difficult day for NineStar and its customers, Jon Miller, Hancock Regional Hospital controller, said the company “was a great partner” and kept the hospital advised of the situation as the afternoon progressed.
The hospital’s “dark fiber” network, which it operates independently on fiber optic cable leased from NineStar to connect its various campuses, buildings and health networks, was not affected by the outages, Miller said.
“However, transmission of radiological images and encrypted patient information over the Internet was affected, though none of the information was compromised,” Miller said.
Spencer said response to the credit has been “overwhelmingly positive.”
“Many see the 10 percent credit as going above and beyond, being their actual outage time for the month was only 2 percent,” he said in an email.
In addition to advising customers of the March credit, the company’s letter assured users none of their information was compromised by the service disruption or the denial-of-service attacks.
“We take great pride in our network, and while the DDoS attack was no fault of our own, we have since put in place additional measures to help mitigate future attacks,” Spencer said in an email Monday.
The credits did not affect the company’s electric customers.