USD left in the dark after university wide blackout: SDG&E stops power to USD after line malfunction

By Jackson Somes
NEWS EDITOR

Two years ago an electrical power outage swept over much of the Southwestern United States. From San Diego to parts of Arizona this blackout put residents in a sea of darkness. This was the largest power outage in California history, yet during this time USD shone brightly as a beacon on a hill.

Even though USD was able to overcome the widespread blackout of 2011, at approximately 6:30 a.m. on Oct. 10, the campus lost power for several hours. This caused a cancellation of classes and a shutdown of campus technological services such as the campus WiFi.

Many students awoke on Oct. 10 to a text message from the university, alerting them that classes had been cancelled first until 10:00 a.m., then later until 12 p.m. To some students, such as junior Tony Alarcon, this message came as a blessing.

“Honestly, I was ecstatic because it got me out of a midterm I was unprepared for,” Alarcon said.

Other students were less excited about the effects of the power outage.

“I was unable to complete my lab work because the system was down” said junior Elia Rivas. “Now I have to struggle to finish it and study for a midterm.”

Junior Amanda Gates was more concerned about the loss of WiFi on campus.

“I didn’t have access to my email so I wasn’t sure if I had to go to work,” Gates said.

The difference between the region wide power outage of 2011 and the campus wide power outage on Oct. 10 was the location of the error. In 2011, the error occurred outside of USD and the campus was unable to receive power from utility provider San Diego Gas & Electric. Once USD stopped receiving power, the university’s backup generators fired up to continue the flow of power within the university. On Oct. 10, the error occured within the USD electrical framework.

Assistant Vice President of Facilities Management Mark Norita explained the difference between the two blackouts.

“Two years ago the only place that was lit up was USD and that is because the problem happened outside of the campus,” Norita said. “This one [the recent power outage] happened internal internally? to the campus.”
USD utilizes four electrical relay lines to provide electricity throughout campus. On Oct. 10, a fault was recognized in one of these relay lines causing SDG&E, USD’s electrical provider, to shut off the flow of power coming to the university to protect their own system. A fault is any kind of electrical disruption that prevents the flow of electricity. Norita likened the fault on one of the relay lines to a lamp cord. When the lamp cord is plugged in the light comes on, but if you fray or damage the cord, the light goes out.

Once the problem had been identified and a solution remedied, SDG&E could restore the flow of power to USD. There are two parts to this investigation according to Norita, finding the location of the fault and identifying the cause. The relay lines are titled one through four and the fault has been narrowed down to Line 1.

Director of Facilities Services and Planning Kimberly Carnot explained the process. “We have tested Lines 2, 3 and 4 and they are all fine” Carnot said. The fault on Line 1 has been identified in the area near the Hughes Administration Center and Loma Hall.

With the fault identified on Line 1, all of the power that normally flows through Line 1 was added Line 2. Each line is capable of holding the power capacity of the entire campus. This redundancy is intentional and designed for events like the recent power outage.

Different parts of campus saw an end to the power outage at different times because of the multiple line system. Lines 3 and 4 were restored as soon as the isolation of Line 1 occurred. The rest of campus was restored by 11:30 a.m.

The reason the generators were unable to provide power was due to their own safety. “The generators send out power and then they realize that there is something wrong with the lines and they power back down because you can’t push power through an area where there is a fault” Carnot said. “It’s like hitting a racquet ball into a wall,” she continued. “It goes and bounces back because it can’t get past that point, causing the generators to power down.”

Because the generators were unable to provide power for the campus and the length of the power outage, many of the university’s systems and network devices suffered a hard crash.

“If the generator had turned on we’d all be golden with our technology,” Vice Provost and Chief Information Officer Christopher Wessells said. “The network would have stayed up, the phone systems would have stayed up, all the computers would have stayed up, email, web, everything.”

USD has four data centers that house devices that help manage university technology including everything from phone services to the wifi network, wired network. Currently, there are roughly more than 350 servers that run within the data centers. In the event of a power outage, such as the one on Oct. 10, these data centers run on emergency backup batteries. These batteries are capable of running for about 35 minutes before they lose power.

“When there is a power outage, our data centers are designed to run on these batteries and then as the university’s main generators are activated, the data centers are designed to switch over to the backup generators,” Wessells said.

The problem during the Oct. 10 blackout was that the power outage lasted longer than 35 minutes, forcing all four data centers into a hard crash. This crash is what caused the exchange email servers, campus WiFi, MySandiego portal and University of San Diego’s domain name server among many others to go offline.
Information Technology Services then had to wait for power to come back to campus in order to get the servers running once again, but the process was not immediate. “On a hard crash it takes a long time because you run into problems,” Wessells said. For example, a hardware failure with one of the university’s firewalls delayed the process by two hours, according to Wessells. Once the power was restored, it was a process of manually rebooting the 350 servers and working through the problems to have the campus restored to normal conditions.

Line 1 still remains down and the cause of the fault is still unknown as of 5:45 p.m. Oct. 13. (this is not a complete senta…..) According to Norita, the cause of the fault is still a wide open question. “It could be a number of things,” Nortia said. “I’ve seen these things [electrical faults] where it could be a rat went in there and chewed it [the line] up, or it could be just a deterioration over time in the insulation or the conduit, it could be that water got in there.”

Both Norita and Carnot commended the Facilities Management team for working quickly and efficiently in restoring USD to the status quo. “When you lose a whole campus it is very stressful. Then you really find out how good your team is by the way they handle that kind of stress,” Norita said. “How do they handle the stress? And do they go about their work calmly even when things seem to be at their worst. The team here did just that.” Carnot added to this sentiment. “Our team was not panicked, they were methodically doing what they had to do. It was very focused and it was very calm,” Carnot said.

Wessells similarly lauded the ITS department during this isolated crisis. “My staff stepped up big time,” Wessells said. “Overall the staff did a terrific job under very difficult circumstances.”