When Dominion Virginia Power heard about the attack at Pacific Gas and Electric’s Metcalf substation in the spring of 2013, it was clear the need for security in the electric utility industry had changed forever. Shortly after the incident, Dominion began planning an aggressive substation security program. Developing a comprehensive and successful resiliency strategy requires looking at the big picture. Resiliency planning is something Dominion had been doing for years in the wake of Hurricane Isabel, but the Metcalf attack precipitated the utility’s development of a multifaceted approach to securing its substations.
To the outside observer, Dominion’s substation security program may appear to be just hardened perimeter fences. However, the scope of the plan goes much deeper. When it comes to substation security, there is no panacea or one-size-fits-all solution that works for every utility owner or operator. The correct security solution will vary from state to state or region to region. This point is emphasized by the structure of the new North American Electric Reliability Corporation (NERC) CIP-014 standard on security for critical substations.
CIP-014 is not prescriptive in the security measures each owner or operator must install for compliance. The specific security plan is left open for the utility to determine what is appropriate. As the electric transmission industry actively searches for solutions that work best for customers and service territories, Dominion’s substation security program serves as a case study.
Unique Footprint and Requirements
Dominion is a large vertically integrated utility serving more than 2.5 million U.S. customers in Virginia and North Carolina, many of which have a considerable need for reliable power delivery. Dominion’s $4 billion electric transmission system is configured with about 1,250 miles (2,012 km) of 500-kV lines serving as the bulk power delivery system. The remaining 5,000-plus miles (8,000-plus km) of transmission lines include 115 kV and 230 kV configured to support the load centers. The system also includes more than 450 transmission class substations.
When discussions first began about the substation security program, it quickly became apparent that a truly inclusive plan would require coordination and communication from all levels within the organization. To create a strong plan, the team would have to work through intensive details because conditions, customers and permitting requirements vary greatly from site to site.
Dominion made it a priority to engage internal and external communication groups early in the process. It was understood customers near selected facilities would have many questions about proposed changes to the physical presence of a substation. Dominion took a proactive approach to communication at the onset by reaching out to the U.S. Department of Defense, Homeland Security, local law enforcement and various other organizations. This communication effort set up the program for success with input and support from these parties.
Early internal conversations made it apparent many parties from all corners of the organization needed to be involved in developing the overall security program. It was more than just the substation department or the security department. It became a true cross-functional team with more than 100 people representing the entire Dominion organization. Communication between all departments and multiple disciplines proved to be a key to success throughout the security initiative.
Path to Real Solutions
Discussions continue throughout the electric utility industry about how critical systems and assets should be protected. While it may be true a determined attacker with enough time could find a way to thwart any system, Dominion is determined to implement a program that makes this prospect difficult and to develop a system to respond and recover from events in a reasonable period of time for its customers.
Although the substation security program was a new initiative for Dominion, it needed to support the utility’s established resiliency strategy to assess, prevent, detect and recover. This strategy not only includes perimeter hardening but also encompasses cyber security, communications and equipment redundancy, hardening the System Operations Center, protecting critical substation assets, and recovery from all events — both man-made and natural. The recovery component of the plan has aided Dominion greatly over the years for events more frequently attributed to Mother Nature such as hurricanes.
A key question emerged during initial discussions: How can the utility adapt its current strategy now to include new threats, physical breaches and, of course, the unknown and undefined?
The first step to answering this question was to assess Dominion’s substation portfolio. Planning had to account for the utility’s large geographical footprint and critical customers. Locations where it would be challenging to restore power also were identified. These included hard-to-reach sites, locations with special equipment on-site and locations with significant line interconnections among a host of other factors.
Assets were categorized and ranked according to their potential impact on the system and proximity to customers with sensitive energy reliability needs. Dominion’s ranking algorithm was developed to be consistent with the NERC CIP-014 assessment, but it also takes into account critical-service customers. The algorithm was broader based on specific circumstances and requirements, and it included Dominion’s unique system resiliency modeling software.
The Dominion system ranks substations on a four-level hierarchy based on a comprehensive assessment of most critical to least critical to customers. Those considered the most critical comprise a small percentage of Dominion’s substations and receive the most comprehensive security package. The ranking and assessment system is designed to be flexible and account for a dynamic and changing load pattern in which certain facilities can be upgraded and others downgraded. Worst-case contingency scenarios are constantly being evaluated and plans may change periodically.
The most important factor is that every station is targeted to undergo a hardening process, and the ranking system is proving to be an excellent tool to make the most prudent investments in enhanced security. It ensures Dominion is focusing its efforts on the most vital components of the system.
Building on a Resiliency Strategy
An important component of the program was standards development in support of the overall four-point resiliency plan to assess, prevent, delay and recover. The core element of the program is a perimeter barrier system with anti-cut and anti-climb properties that also incorporates anti-ram and anti-dig features. The main goal of this hardened perimeter is to prevent a potential threat from attempting to access a critical station. Various security systems are integrated into the perimeter barrier to detect and
assess a potential threat.
Multiple barrier manufacturers were evaluated in the course of developing a platform that met all requirements of the security and substation teams. After looking at many suppliers, Dominion made its selection based on the product’s capacity to be customized to support the utility’s resiliency strategy. Dominion required a platform that would be flexible for future needs and changes. Multiple standards were created in a design process that took close to a year, supporting every aspect of the perimeter barrier and security systems.
The engineering design and security system requirements are constantly evolving as new threats emerge. Dominion continually reassesses solutions and standards to respond to those changing threats. This dynamic process requires security to work in close collaboration with the engineering, permitting and construction teams.
Dominion’s perimeter barrier varies in height from 12 ft to 20 ft (3.6 m to 6 m) with anti-cut and anti-climb properties. The barrier material also provides some level of visual screening and enhanced security measures. The selected exterior color was chosen to match substation assets, which helps to better screen equipment. Detection and assessment systems installed in conjunction with the perimeter barrier are monitored by Dominion’s corporate security fusion center. The monitoring team includes professionals trained in intelligence and video analytics who can quickly assess patterns and events as they happen and rapidly communicate with appropriate law enforcement agencies and system operators.
Another instrumental factor for the program has been the communication and collaboration by all departments involved with the effort. While many groups are involved and dedicated to the substation security program, Dominion assigned accountability for the overall execution to the electric transmission group. By having a single group drive each project forward, the implementation schedules have stayed on track. Dominion’s leadership team also communicated goals to the entire organization, conveying that a successful deployment was dependent on everyone’s contributions.
Every project requires significant input from Dominion’s standards, engineering, security, permitting, construction, communications and information technology groups. Weekly meetings are held with up to 30 representatives from the different divisions. In the beginning, coordination was challenging with so many parties involved. However, participants began to refine their specific roles and responsibilities as the program progressed. The detailed execution is a complicated process because each project is unique and has many moving parts. Every group must manage and complete their tasks to reach the final completion of each job.
How It Gets Done
With the plan in place for Dominion’s substation security program, it was time to transition from strategy to execution. With more than $1 billion in capital improvement projects already underway, Dominion was concerned the additional work required by the security program could strain internal resources. The utility selected Burns & McDonnell to assist with execution of the plan. This decision was based on a long-standing relationship with Burns & McDonnell and the company’s knowledge of Dominion’s culture and operations.
The overall process for the substation hardening project has evolved greatly since the program’s inception. In the beginning, there was no sample project to go by. The team and the entire process were created from the ground up. There have been many learning opportunities along the way, but the substation security team now has a streamlined approach for implementation. Once a site is identified for a security upgrade, the project execution process kicks into gear. It all starts with communication because a large number of groups must be involved.
A site visit is the first item on the agenda, and all the parties who need to be involved are required to participate. This can include permitting, security, engineering, construction, IT and other groups as needed. The goal of the substation walk-through is to develop a security plan based on observed site characteristics and input from the team. This becomes the blueprint for what will be done at that substation, including what components are going to be added and modified. Potential conflicts, obstructions, civil features and site access modifications are noted with respect to potential substation vulnerabilities and monitoring locations for security systems. The permitting team consults on any ordinances or restrictions within the local jurisdiction that could create delays in construction.
At this stage, the necessary surveys are initiated. It is hard to overstate the importance of these surveys when doing a perimeter hardening project. Accurate and detailed surveys are collected for the perimeter topography, overhead transmission and distribution lines, and underground obstructions. This information is vital for the engineering teams to mitigate electrical clearance violations and for the fence manufacturer to avoid potential conflicts.
The second stage is a design review. Field notes are collected from all parties involved and put into the plan. This may include potential substation electrical clearance issues as noted by the engineering teams. The security teams will then confirm where they want to place monitoring locations and anti-ram systems based on the layout of the perimeter.
Approvals are secured from all department representatives. Buy-in from all parties is crucial to the plan. It may be the most important step in the entire implementation process because it ensures engineering design and construction will proceed in the right direction. Concurrently, permitting professionals provide input on limitations for construction and infrastructure at the site. Going to this level of detail on the front end has proven to save time and money on the back end of the project, where changes can be costlier.
With the security plan finalized, the detailed design and permitting is ready to proceed. This stage may require up to four months and includes working with the fence manufacturer to customize designs based on layouts and surveys of the site. This stage includes tackling any potential underground obstructions or overhead clearance issues identified by the surveys.
Detailed substation and transmission engineering designs are created so changes can be executed, including relocation of substation equipment or elevation of transmission lines. Civil teams talk with the permitting group to identify any changes that need to happen on the site that could trigger a site plan review and conditional-use permit. Any outside parties who were not part of the original evaluation may need to be contacted. For example, there could be a 19-ft (5.8-m) distribution line coming in that cannot clear a 20-ft (6.1-m) fence. That means the line will have to be converted from overhead to underground, and it will require close coordination with Dominion’s distribution organization.
Once detailed design issues are lined out, it is time for the construction preparation stage. By this stage, all permits should be in hand and materials ordered with manufacturing lead times known. The engineering design packages are issued and the final approval is given to the manufacturing design. Once everything is stamped, sealed and ready to go, the construction crews can get started. Identification of construction limitations is crucial during this stage of the process.
Dominion has learned from experience every substation is different. Each design and construction plan must be customized for a given location. The construction groups must work and communicate with the permitting groups so they understand what they can and cannot do to prevent costly delays.
The final stage of the process is construction. This involves establishing survey control points first. On many sites, safe electrical clearances may be only a few feet to the fence.
If control stakes are not precisely in the correct places, big problems can result. The survey is absolutely vital, and Dominion’s survey crews have prevented many potential construction issues.
Next, construction will establish erosion and sediment control points followed by installing temporary fencing. Dominion created a protocol to maintain a safe perimeter with appropriate fencing during construction activities. At this time, receiving and accounting for materials is also very important. The construction department must verify all material ordered has been delivered with the right quantities and specifications. Even though no construction project is perfect, constant communication and coordination can address any issues that arise.
The typical duration of a project is around 12 months. This includes up to four months for engineering, depending on the station, the process and the challenges. Permitting applications are typically filed during the first three to four months and approvals are secured during this time frame. For a typical station, Dominion generally estimates a six- to eight-month construction window, depending on where the station is located and the site conditions.
Challenging Conventional Thinking
Though the entire industry is aware that NERC CIP-014 is a guideline utility owners and operators must follow for compliance, the specific installation requirements are quite flexible. In Dominion’s case, because of critical customers with high sensitivity to power reliability, a concerted effort to harden perimeters and implement sophisticated asset monitoring was appropriate and necessary.
The Dominion substation security program has shown how a truly comprehensive and successful program requires constant communication with involvement from many outside groups and across the organization. Every project needs individual attention, and all parties involved must collaborate to reach the overall security goals.
The focus on substation security and resiliency design to protect against outside threats is a new approach for the transmission and distribution industry. The conventional thinking of protecting the public from the substation has been turned on its head. As the industry enters the new era of protecting the substation from the public, it is obvious most transmission owners and operators now must think in terms of preventing, detecting, delaying and responding. Though each must adopt the plan that works best for their unique service profile, some aspect of hardening and improving perimeter security will almost certainly be a component of the plan.
Bob McGuire ([email protected]) is director of electric transmission project development and execution for Dominion Virginia Power. His group is responsible for the development, engineering and project management of all major transmission and substation projects.
Dominick Piccolomini ([email protected]) is manager of substation engineering for Dominion Virginia Power. His teams are responsible for substation physical design, protection and control design, communications design, material management, equipment specifications and substation standards.
Jason Meidinger ([email protected]) is a project manager in Dominion Virginia Power’s electric transmission group. He is responsible for the oversight and execution of Dominion’s substation security program.
Keith Graham ([email protected]) is a team lead for Dominion Virginia Power’s substation engineering and design standards group. His team is responsible for the development of the substation physical security standards, including written specifications, design methods and standard drawing development for the security systems installed at substations.
Keegan Odle ([email protected]) is a senior project manager in the transmission and distribution division of Burns & McDonnell. He is an electrical engineer specializing in the execution of electric power substation projects from 12.47 kV to 500 kV. He has published multiple white papers on substation security and presented on NERC CIP-014 across the United States.
