Planning and training leads to plan activation and an 18-day rebuilding effort to restore the system after a devastating hurricane.
ON SEPT. 13, 2008, HURRICANE IKE MADE LANDFALL ON THE TEXAS COAST, near the city of Galveston. Ike was a Category 2 hurricane in terms of its 110-mph (177-kmph) wind speed and a Category 4 in terms of its 14-ft (4.3-m) storm surge. What would these conditions do to the electrical substation system? Devastation is the first word that comes to mind. And that is what happened. Fortunately, CenterPoint Energy (Houston, Texas, U.S.) managed to return service to all customers capable of receiving service within 18 days because of the planning, training and implementation that surrounded the utility's restoration efforts.
PLANNING AND TRAINING
CenterPoint Energy has an extensive emergency operating plan (EOP) covering every aspect of hurricane preparedness and response. In the utility's EOP annual review and update process, changes to personnel assignments and organization planning occur to ensure the plan is up to date and that everyone throughout the company knows their emergency-response responsibilities, which range from mutual assistance planning to logistics (material procurement, housing and caterers) to public relations, in addition to all of the system-restoration activities.
Mandatory training pertains to both preparing the substation system and the individual employee. June 1 is the official opening of hurricane season. Therefore, CenterPoint Energy holds an EOP preparation and review day every May. On that day, the substation operations department performs refresher training with its electricians, staff and management.
The substation evaluation center (SVAL) is set up. The location is on the northwest side of CenterPoint Energy's service territory, approximately 90 miles (145 km) from the coast. The SVAL is activated to verify proper functioning of the radio system, telephones, computer connections and all support activities. Field crews practice calling in information to SVAL. The information is prioritized and forwarded to other departments. This step verifies communications within the company and validates computer programs, which capture damage information.
Field crews are also given a refresher course on black-start restoration philosophy, including manually synchronizing breaker reclosing. Industrial customers are contacted to verify evacuation plans, including leaving batteries in service with station power for charging and access to power equipment.
A checklist of all storm plan activities is revisited. This checklist includes items from renting special equipment to verifying that employees, out with serious illness, are properly secured or evacuated for the impending storm.
Individual employee training is an ongoing process, communicating what is expected of every employee to support the company's restoration activities. Employees are also provided hurricane tips on how to prepare their homes, such as storing water, topping off gasoline in cars and generators, filling prescription medications, storing ample amounts of nonperishable food, boarding up windows and picking up loose items in the yard.
Making plans for the employee's family is covered, too. This includes making the employee who has never experienced weather of this severity aware of the physical dangers associated with wind and storm surges. Employees are encouraged to preplan their family's evacuation, if necessary, in plenty of time for employees to be available for work and not be concerned about the well being of their family members. And, there is also a list of employees in known evacuation zones for housing needs during the EOP event.
As Hurricane Ike entered the Gulf of Mexico, CenterPoint Energy initiated its EOP. The steps in the EOP and training were put into action. Field crews were sent to every substation prior to the storm to return to service any equipment that was out for maintenance, if possible. Substation yards were policed for loose material being used on substation projects. The flood-prevention gate at the Texas Medical Center substation was closed and sealed. A truck was left inside the flood-prevention wall in the event roads were impassable due to flood water from a nearby bayou. A truck was placed in an elevated parking garage on Galveston Island in the event access to the island was impeded by flood or storm debris. Trucks, equipment and documents were moved from low-lying coastal areas to higher ground. Employees were allowed time to evacuate their families and prepare their homes for the impending storm.
The day before the storm, some employees, previously defined as storm riders, were placed in secure facilities out of the storm surge areas. All other substation employees were previously defined as first responders, who are asked to come in to work as soon as the wind and flooding subside to safe levels for travel. These employees also are given the option to dwell at the secure facilities. For Hurricane Ike, most employees took the latter option.
In addition, when distribution circuit breakers began to experience faults and operate to clear them, the automatic reclosing capability was removed to avoid multiple faults on the substation transformers. Automatic reclosing was left in service on transmission lines. The philosophy behind this decision is that transmission lines are designed for high winds and right-of-way clearance minimizes small-tree-limb damage from occurring.
Tropical-storm-force winds and strong wave action began to pound the coast around 4 p.m. on Sept. 12. The eye crossed the shoreline at 2 a.m. Tropical-storm-force winds left the CenterPoint Energy service territory 10 hours later. During the height of the storm, distribution circuits began to lock out at the rate of 50 to 60 circuits per hour. The system load was lighter than normal because of the evacuations and the shutdown of industrial facilities. At 11 p.m. on Sept. 12, the system load was approximately 6500 MW. At 6 a.m. on Sept. 13, the system load bottomed out at 1631 MW.
More than 90% of CenterPoint Energy's more than 2 million metered customers were without electric service. Of the 267 CenterPoint Energy substations, 98 were either completely de-energized or partially de-energized. Of the 137 industrial customer-owned substations, 56 were either partially or completely de-energized. During this time, supervisory control and data acquisition (SCADA) was monitored to begin assessing the substation system. After the storm passed, CenterPoint Energy discovered that approximately 60 communication circuits were damaged, thus limiting remote-control capabilities to the associated substations.
As soon as it was safe to travel, substation field crews began to assess substations. Later that evening, two helicopters were dispatched to the coastal areas from Baytown to Freeport, Texas, to begin an aerial assessment of substations and road conditions. During the first day of assessments, the utility determined that substations on the mainland suffered minor damage. Galveston Island was a different situation. Five of the six substations on Galveston Island were damaged by Hurricane Ike: West Galveston, Galveston 26th Street, Moody, West Bay and Stewart. In particular, the West Galveston, Galveston 26th Street and Moody substations were flooded by the storm surge from Hurricane Ike and suffered severe damage. The Stewart and West Bay substations were not damaged as severely as the other three substations on Galveston Island but did require some repair.
CenterPoint Energy's initial assessment determined that 199 transformers were out of service. The majority of transformers were de-energized because of a loss of transmission service. The utility found that 33 transformers tripped by protective relaying. These transformers had a set of diagnostic tests before being re-energized. During this test event, no transformer failed because of through-fault damage. Related to the transformers that were out, 49 substations were completely de-energized, meaning there was no station service available to maintain a battery charge. At these locations, plans were implemented to re-establish battery-charging capabilities. The plans ranged from re-energizing transmission lines, which would provide the battery-charging supply, or using portable generators to charge batteries that were without station service.
Within four days of the storm's passage, all substations on the mainland were able to feed any distribution circuit that was ready for service. In no case was there a request to re-energize a circuit where the substation was not ready to provide service. While breakers at 27 substations on the mainland needed repair or replacement, every distribution circuit has redundant breakers that can be employed to restore service.
On Galveston Island, there were three substations severely damaged by the storm surge. The 26th Street substation, which feeds transmission service to the medical center, is approximately 8 ft (2.4 m) above sea level and had approximately 4 ft (1.2 m) of water in the control house. Also, the West Galveston substation is 7 ft (2.1 m) above sea level and had approximately 5 ft (1.5 m) of water in the control house. After assessing the damage to the substations, CenterPoint Energy decided to bypass West Galveston and rebuild the 26th Street substation.
The 26th Street substation was staffed with 35 electricians, three crew leaders and one supervisor. The job would entail replacing four 138-kV breakers, six 12-kV breakers, one motor operator, associated relaying, two 130-V batteries, one 24-V battery and three battery chargers. This crew of dedicated employees was given the task on Sunday morning, Sept. 14. The aforementioned equipment was replaced and put in service on Wednesday, Sept. 17.
Over the next 10 days, an additional two 138-kV breakers, twelve 12-kV breakers, and associated relays were also replaced. The amount of work accomplished in a very short time frame was monumental. There were 120 relays, more than 5000 lugs, associated terminal blocks, fuse blocks, knife switches and control cables replaced and re-commissioned. All equipment went through normal commissioning tests. CenterPoint Energy took no commissioning shortcuts and, above all, omitted no safety procedures. Everything was accomplished by working 16-hour days.
CenterPoint Energy placed the West Galveston distribution substation back in service 10 days after the storm's passage by installing a mobile transformer and 12-kV bus work. The actual construction only took five days, but there were no distribution customers ready for service and all service could be tied to adjacent substations if necessary. Two months later, a second mobile substation was installed to improve the area's reliability. Six of the 10 circuits were connected to the two mobile transformers.
The transmission side of the substation has been rebuilt. The control house is built on piers to an elevation of 20 ft (6.1 m) above sea level. Additionally, all breaker foundations are built to 17 ft (5.2 m) above sea level. This puts the control cabinets at approximately 20 ft above sea level. The distribution substation is being built to these standards.
The third severely damaged substation, Moody, had five distribution breakers and wiring terminal boxes rewired within eight days of the storm's passage. There was approximately 1.5 ft (0.5 m) of water in the control house, necessitating some minor relay replacement. This substation was approximately 12 ft (3.7 m) above sea level.
Never underestimate the value of dedicated employees and what they can do when they know the plan and the societal needs dependent upon a robust electrical system. Before sending employees in flooded areas, consult with medical experts regarding precautions to take. Tetanus and hepatitis are potential health threats that must be considered due to contaminated water covering equipment.
When the system is unloaded, high voltage may be present. All capacitor banks and static VAR compensators should be taken off-line at the onset of a storm. For unplanned de-energizing of a power system, such as in a hurricane, the logic of the control system may end up in an undefined state. Upon re-energizing the system, care must be taken to disable the automatic reclosing capability of all breakers, or unintended breaker closing may occur.
Safety messages to expect the unexpected should be reiterated to crews every day.
Watch for generation islands. When multiple transmission lines lock out, islands may form and not be apparent because of a remote opening two or three substations away.
Additionally, begin tracking data before a storm makes landfall to facilitate reporting to executives and regulators. And preplan accounting codes and setup cost allocation projects for major areas of damage.
Martin Narendorf (firstname.lastname@example.org) is accomplished in the areas of system operations, reliability, engineering and operations software development, power quality and harmonics solution technologies. Formerly the general manager of the electric utility serving Sao Paulo, Brazil, Narendorf is now the director of substation operations for CenterPoint Energy. Narendorf is involved in the installation, operation and maintenance of substation facilities and additional areas of cyber security and renewable technology initiatives. Having contributed to the fields of automated metering and distributed generation, Narendorf has authored IEEE Power & Energy Society technical papers on arcing fault detection and harmonics.
Rand B. Westbrook (email@example.com) has completed 38 years with CenterPoint Energy and is the senior manager of real-time operations. Thirty-four years of his career has been spent in substation operations, where, during Hurricane Ike, he served as senior manager of substation performance. Rand holds a BBA degree in organization and behavioral management, a BSET degree in power delivery and a MBA degree in management, all from the University of Houston.
Note: This article is based on content from a presentation at the Southeastern Electric Exchange Conference in Atlanta, Georgia, June 2009.