ON APRIL 15, 2003, AT 11:40 A.M., A BLACKOUT OCCURRED IN SOUTH CENTRAL TEXAS that covered most of Brazos County, including the Bryan/College Station area. The outage extended nearly four hours before power was restored. This article discusses the events leading up to the outage, the root cause of the outage and corrections that Texas Municipal Power Agency (TMPA) implemented to prevent future operations of this nature in its service territory.

TMPA serves the cities of Bryan, Denton, Garland and Greenville, Texas, including the surrounding counties. The municipal has a 462-MW coal-fired unit at Gibbons Creek Substation and a transmission system consisting of 100 miles (161 km) of 345-kV lines, 150 miles (241 km) of 138-kV lines and 11 substations.

THE EVENT

Gibbons Creek Substation is a breaker-and-a-half 345-kV to 138-kV substation with nine 345-kV breakers, eight 138-kV breakers and two wye-wye autotransformers each with a delta tertiary. The two 138-kV and 345-kV buses are identified as the North and South Buses.

The event began when the King 345-kV line at Roans Prairie (one bus away from Gibbons Creek) was de-energized for maintenance. The Frontier Power Plant (also at Roans Prairie) was online with an output of 800+ MW (1350 A), with all power flowing back to Gibbons Creek as a result of the open King 345-kV line at Roans Prairie. Flow was in on the 345-kV Roans Prairie line and out on the O'Brien line. As a result of the increased current flow, three CHC breaker-failure-current-detector relays at Gibbons Creek (on Roans Prairie and O'Brien lines) picked up.

At that point, breaker-failure relaying operated on CB 5120 (O'Brien line), which initiated operation of the North Bus (345 kV and 138 kV) breaker-failure scheme. Breaker-failure relaying on CBs 5090 and 5080 (Roans Prairie line) operated, and 5080 initiated operation of the South Bus (345 kV and 138 kV) breaker-failure scheme. In all, 14 breakers (345 kV and 138 kV) operated in half-second. Gibbons Creek Unit 1 stayed on-line at full power connected to the Twin Oak West line with one breaker (5020) closed. With both 345-kV/138-kV autotransformers de-energized, the main sources of power into Bryan/College Station (B/CS) were lost. Only one 138-kV line remained energized into B/CS (from the North), but it could not maintain the voltage in the B/CS area, thus deteriorating until the line tripped a few minutes later, resulting in a complete blackout. Relay targets were found only on the breaker-failure schemes, current detectors, timers and lockout relays. No primary relays operated for a fault.

In addition, the primary station service power source to the control house was lost and the SCADA system stopped functioning. This left the dispatchers at Garland Power & Light (Garland, Texas, U.S.) and the Electric Reliability Council of Texas (ERCOT) with no information and no control of the substation. Manual switching had to be done by phone using orders from dispatchers. Recovery was further slowed due to inoperative panel indicating lights and CHC current detector relays with contacts stuck closed.

EVENT-CAUSE ANALYSIS

No faults were indicated on any of the monitoring equipment or protective relays in or around Gibbons Creek Substation. So, what initiated the breaker-failure operation? Recall that breaker-failure fault detectors on the 5120, 5080 and 5090 were picked up on load. Investigation revealed that a 10-Ω resistor in parallel with a 500-mfd capacitor used as an impulse circuit to speed up the breakers was burnt open on breaker 5120. This allowed the breaker-failure initiate bus to go high, and when the CHC picked up on load, it caused a breaker-failure trip on circuit breaker 5120. A two-tier breaker-failure scheme (second tier on the 345-kV North Bus) then generated a breaker-failure trip, which cleared the 345-kV North Bus and sent a trip to the 138-kV North Bus breaker-failure scheme, clearing that bus as well.

From there, the system should have stabilized with no more trips; however, CHC breaker-failure fault detectors on breakers 5080 and 5090 stuck in the picked-up position, resulting in breaker-failure trips that led to the loss of the 345-kV and 138-kV South Buses. It was later found that a “holding coil” in the CHC breaker-failure current detectors that affects the dropout action of the detector was out of adjustment on the relays for breakers 5120, 5080 and 5090.

After being tested and readjusted, the relays functioned properly; however, it was also discovered that the pickup and dropout current levels decrease significantly when applying a polyphase test rather than the single-phase test, as described in the CHC instruction book. Relays originally set for single phase at 1000-A pickup were found to operate at 675 A when tested polyphase. It was also discovered that the second-tier bus breaker-failure tripping time was set faster than the line-terminal tripping time, resulting in a miscoordination between the two schemes. This combination of failures led to incorrect initiation and operation of the breaker-failure schemes, resulting in an outage of all of the lines supplying power to a university campus and its surrounding municipalities.

Other problems associated with the outage included SCADA RTU modems that ceased to function when the station battery voltage dropped to 117 Vdc and multiple grounds were discovered on the dc battery system. Also, the memory in the modems did not retain their settings and came back on with the wrong configuration. And, in circuit breaker 5120 (with the open resistor), trip coil #2 was found shorted.

Additional tests were performed to confirm that the red-light circuit would supply enough current to power a SAM timer. It was also discovered that the SAM timer will operate on half battery voltage (i.e., a ground on the positive or negative supply voltage). Tests verified that all the SAM timers were working as designed. CT ratios were checked and verified, and line relay settings for the Roans Prairie line were reviewed and verified. Trip time tests were performed on the involved 345-kV breakers and were found to be in range: 0.9 to 1.4 cycles; the same breakers were checked for an open pole condition and no problems were found. The station battery tested fine.

CORRECTIVE ACTION

After a rigorous root-cause analysis that identified the open resistor in breaker 5120, the CHC sticking contact problem on three breakers and the SAM timer operating sensitivity, the following corrective actions were approved:

• Modify dc circuits for the breaker-failure schemes on all 345-kV and 138-kV breakers to include blocking diodes in the primary and backup tripping circuits, isolating the red-indicating lights, local control switch trip contact and SCADA trip contact from the breaker-failure trip bus.

• Replace all CHC current detector relays and SAM timer relays with Basler Electric microprocessor relay BE1-851.

• Use features in the new relays to monitor trip circuits to each breaker and to alarm for an open-circuit condition.

• Eliminate the two-tier breaker-failure scheme.

• Repair/replace any nonfunctioning indicating lamps on control panels.

• Eliminate dc grounds in the substation.

• Replace the RTU modems and their power supplies.

Black & Veatch (Overland Park, Kansas, U.S.) was hired to design the wiring modifications and determine the relay settings. Isolating diodes have now been added to the trip circuits and other wiring modifications have been made. The second-tier “bus-tripping” breaker-failure schemes have been removed so that one scheme will trip all necessary devices to isolate a bad breaker. The new Basler relays have been installed, programmed and tested. Trip circuits for both trip coils in each breaker are being monitored, and alarms are wired to send data to the dispatchers on several items, including open-trip circuit and relay failure.

NEW RELAY TECHNOLOGY

The Basler Electric BE1-851 microprocessor relay was chosen to replace the CHC/SAM breaker-failure scheme. The current detectors and breaker failure logic were set up to operate in seven cycles, providing a total clearing time of 13 cycles. The circuit breaker interrupting duty is being monitored and logic has been programmed into the relay for added security against false operations.

The microprocessor relay has all the features normally expected in a numeric, communicating multifunction relay system, including three-phase, ground and negative-sequence overcurrent protection with a four-shot recloser, breaker failure with fast dropout current detector (less than one cycle), breaker monitoring, self-checking, event recording, oscillography, control, metering and standard communications protocols, all in an integrated system.

In addition, and very important to this application, is the availability of the relay in a “retrofit friendly” S1 case that fits directly into the SAM timer cutout of the panel, eliminating the need for panel cutting and vibration that could potentially cause another unwanted outage. Also, the relays can be in-case tested and drawn out for inspection.

TMPA has learned many lessons in investigating and correcting the protection problems associated with the blackout. Fortunately, the technology available in today's multitasking microprocessor relays has enabled it to improve reliability by correcting and eliminating many potential paths of failure. Now, TMPA's protection design is streamlined and more aptly ready to respond to different system contingencies.

---

Rick Gurley is the transmission manager for Texas Municipal Power Agency (TMPA), where he has been employed for nearly three years and is responsible for overseeing the planning, engineering, construction and maintenance activities for TMPA's transmission system. Formerly, he worked for investor-owned utilities for 22 years in various engineering and supervisory roles involving transmission line and substation systems. Gurley earned his BSEE degree from Texas Tech University and is a certified professional engineer in Texas. rgurley@texasmpa.org