The Bradley 500-kV substation is the first fully automated, multirelay vendor project in the United States to implement the full suite of IEC 61850 communications. Additionally, the ultimate goal of the Tennessee Valley Authority (TVA; Chattanooga, Tennessee, U.S.) is to provide IP connectivity via PowerWAN to any field device or intelligent electric device (IED) located within its 80,000-sq-mile (270,000-sq-km) service territory. The Bradley substation and the PowerWAN communications are both exploring new mechanisms for end-to-end integration.

THE REALITY

As an industry, we are looking for that all-encompassing solution: unified data access, whether it's in native or standardized format, and whether it's real-time, historical, translated or raw. The solution also should provide integrated access controls that work with one seamless click. Some have come closer than ever before to achieving this vision, and some might even argue they have been able to realize this goal through highly integrated architecture. And, after all, they didn't use that much custom code. And, they would like to expand it to do just this one more thing.

That single, all-encompassing solution has not shown up to save the day. And there is a good reason: it doesn't really exist. It may sound attractive, but if the end-to-end solution came in one package, it would be prohibitively expensive. It also would force North American utilities to toss aside communication architectures that have been developed over the past few decades and that often overlap multiple prior and current designs.

THE GOAL

Our goal is to have a modular, reusable and scalable plan that is as robust as it is secure. TVA plans to accomplish this goal with its Bradley 500-kV substation and PowerWAN information architecture. Together, these projects play a major role in the future for remote-resource access and management.

TVA used rigorous engineering and documentation, early and often, when working on the Bradley and PowerWAN projects. Modular designs with well-defined purposes and points of interface have facilitated phased, scalable rollouts and led to a clear understanding of the effect of problems that arose.

Policies and procedures are significant drivers for system requirements, enabling long-term maintainability without depending on the original architects. The examination of security at all levels — from individual components all the way up to the enterprise — will put TVA on track to meet North American Electric Reliability Corp. (NERC) Critical Infrastructure Protection (CIP) requirements and beyond.

SYSTEM ARCHITECTURE

The most visible evidence of a new, disciplined approach is in the system architecture. The systems use new technologies such as IP-based communications and the IEC 61850 suite of protocols. While the technologies are not the goal, they enable the design principles driving the next-generation grid.

BRADLEY SUBSTATION

The substation uses Generic Object-Oriented System Event (GOOSE) messaging for protection and a client-server model for reporting and control. The design implements two redundant trains of protection and breaker control relays, each with a local operator interface panel and a gateway controller. All of these components implement the 61850 object model, even though the link from the control center to the substation gateway is distributed network protocol (DNP).

The substation is a pilot project and has certain safety nets in place while TVA gains confidence in new technologies. Digital fault recorders (DFRs) are still in the substation, but in future designs, fault data will be gathered via a virtual DFR spread throughout the relays. GOOSE messaging is backed up by hardwire trips in a parallel path to the breaker, with the breaker not concerned with where its signal originates.

A third and non-redundant group of IEDs in the substation supports circuit breaker and transformer monitoring, synchrophasor measurement, metering, DFR and maintenance mechanisms. None of these devices currently implement IEC 61850; instead, they rely on protocols such as DNPi, IEEE 1344, MODBUS, telnet, ftp and http. The bad news is that these various protocols still have integration challenges; the good news is that TVA has saved money by consolidating these functions onto a common substation Ethernet network.

SECURITY: BUILT-IN NOT BOLT-ON

The Bradley and PowerWAN projects employ many centralized security features, with the building blocks for this functionality starting at an elemental level. All devices on the network have a specific understanding of the role they play in the overall security scheme. Any activity not falling within those bounds is rejected and flagged for further investigation.

To migrate its fleet of IEDs toward this model, TVA has provided vendors with a secure device specification. While TVA does not expect all vendors to meet the entire specification immediately, sharing it enables clear communication regarding IED features and capabilities, and provides TVA with a measuring stick during the selection process. Interestingly enough, several of the features requested in the specification have started showing up in firmware releases for equipment already deployed in the Bradley project.

The NERC CIP standards dictate the measures utilities must take in identifying and protecting critical cyber assets. While the telecommunications infrastructure itself is not covered by the current NERC CIP standards, many of TVA's substations and enterprise applications will fall under their purview. This will have significant implications for access to these assets on both technical and procedural fronts. Ultimately, the electronic security perimeters required by the CIP standards will encompass the remote devices deployed and maintained by a broad array of organizations within TVA. The design philosophies for both Bradley and the PowerWAN enable TVA to implement a single, unified security policy to address all remote device and data access.

PowerWAN ARCHITECTURE

The PowerWAN secure device access gateway is designed to abstract the authentication, authorization and addressing for all remote device and data access. It resides in the demilitarized zone between the PowerWAN and the enterprise, and integrates with TVA's existing identity management, eliminating the need for community logins and published IP addresses for individual IEDs.

The PowerWAN information architecture project represents new and improved functionality for remote device and data access. The PowerWAN will provide a common, unified network for all transmission-related remote devices. Traditionally, asset owners might be solely responsible for determining field devices and communication mechanisms used to acquire remote data on an application-by-application basis. As a result, communications infrastructure was unnecessarily bound to business function.

A primary tactical goal of the PowerWAN is to rectify this situation and consolidate all transmission system communications under a single deployment scheme. By standardizing on IP-based communications, TVA can leverage a wide range of industry-standard solutions, addressing everything from network management to endpoint security. This allows transport mechanisms to be selected according to data needs and environmental constraints and independently from application.

MIGRATION PATH

In addition to building the data path of the future, the PowerWAN provides a mechanism for migrating existing TVA systems to IP-based solutions using traditional or legacy protocols. The migration path provides the enterprise with an easy, self-paced choice rather than a rough transition to new technology. Further, it allows TVA to better leverage its assets, including the PowerWAN, through more efficient bandwidth usage.

Critical to the migration path is the emergence of the IEC 62351 standard, which is still in development. IEC 62351 will help delineate industry-standard security solutions for many topics, including the usage of TCP/IP, VPN tunnels and “bump-in-the-wire” solutions. The standard will apply to newer technologies, such as IEC 61850, and profiles including MMS, network management and those derived from 60870-5 like DNP.

The IEC 61850 technology will be able to describe configuration data in a standardized XML format. This function promises to eventually eliminate the dreaded point list of DNP and replace it with a text file of self-describing data. TVA eventually would like to use this tool to facilitate the dynamic exchange of configuration data between the substation and the control center, but that aspect of standards harmonization is still in development.

SUBSTATION CONFIGURATION

In theory, one should be able to bring capability description files from disparate IEDs into a substation configuration tool, map functions to a one-line diagram, generate IED-specific configurations and export those configurations back out to the individual IEDs. TVA, however, has faced challenges that are difficult to pin down to any single vendor or component.

While there is no shortage of highly capable IEDs, many configuration tools are still in the maturation process. Further, TVA has yet to find a dedicated purpose-built and independent substation configuration tool. Such a tool could go a long way toward resolving many of the issues of uncertainty and doubt.

TVA still expects to come out of the experience with a single self-describing substation-configuration file for Bradley, which is in itself a huge step forward over the DNP point list. But the real potential of a complete substation configuration file is to be leveraged in the transmission-system planning and design process, and mapped to the control center representation of the system. This, however, is still on the drawing board because of a second issue: standards harmonization.

The relevant standards for agreement with IEC 61850 are IEC 61970, the Common Information Model (CIM), and IEC 61968, the Generic Interface Definition (GID). IEC Technical Committee 57 has a working group dedicated to the harmonization effort. However, 61850 has a different viewpoint than 61968 and 61970, so this is not an easy process. TVA does anticipate this will be resolved over time, eventually enabling the industry to pursue an integrated transmission-system design pipeline.

ENGINEERING PROCESS

While the architecture changes are significant, the changes taking place in the engineering process itself are equally important. These changes reflect a new way of doing business, one where a small investment in engineering is offset by big long-term dividends.

A substantial portion of the total cost of ownership for a substation is in post-commission testing, maintenance and modifications. Therefore, TVA has adopted the philosophy of modularity in key areas of substation design. Fundamental changes from the use of enabling technologies have rippled out to have far-reaching effect on the long-term picture. A singular, robust network now supports multiple functions and has replaced multiple application-specific networks. Multifunction IEDs are reducing the total number of devices deployed, and system protection logic is now developed without regard to vendor-specific products or implementations.

The reduced amount of discrete wiring is one of the most noticeable effects of this new TVA philosophy. Networked communications infrastructure offers a flexibility that revolutionizes existing construction practice, enabling modular, pre-engineered, pre-wired and pre-configured assembly techniques. The majority of connections between devices now are implemented as virtual connections using the substation's Ethernet LAN instead of discrete wire.

BENCH TESTING

The upfront investment in the TVA lab facility has proven absolutely invaluable by providing a testing “sand box,” especially since the system integration is being done with in-house resources. Long before the physical site was constructed, TVA was able to complete basic device interoperability testing, as well as full-scale validation of the system design, including device logic. This has significantly reduced overall project risk compared to past projects. Full-scale testing was previously not practical before the final system build.

While interoperability validation did not go as smoothly as either TVA or the vendors had hoped, the process identified some showstoppers early enough for changes to be made without a major impact on the schedule. In a traditional build, these issues would not have been discovered until final testing of the installed system and could not have been resolved within the project budget and schedule.

LESSONS LEARNED

The most significant lesson out of this experience for TVA was learning that type testing of only the devices might not be enough in the realm of 61850. This engineering standard was intentionally designed to accommodate many configurations and methodologies. Vast portions of the standard's object models are marked as optional to avoid being prescriptive with regard to product lines. Unfortunately, this same flexibility can be extremely daunting when first figuring out how to represent your individual design. Some portion of those optional objects will be required, and it will be up to you and the vendor to negotiate the details. In short, if you want multiple vendors, you will need a 61850 expert on your side, lest you run the risk of being caught between pointing fingers.

TVA expects to come out of the experience with solid documentation of the Bradley design. They also will abstract the Bradley design into a template for any size or design of site, allowing them to one day run through a list of options and derive a site-specific, tailored configuration. The particular configuration may never have been seen before, but all the interdependencies should be well understood. No single portion of the design will be new.

While the path to end-to-end integration is not painless, the PowerWAN and Bradley projects are offering opportunities to leverage new tools and learn new ways of doing business.

---

Brian Smith is a principle design engineer in TVA's Transmission Power Supply/Electric System Projects group. He received his BSEE degree from the University of Tennessee at Chattanooga in 1991. bpsmith@tva.gov

Darren Highfill is a utility communications security architect for EnerNex Corp. He is one of the system architects for the PowerWAN — TVA's new wide-area IP-communications network — and has been heavily involved in the integration of the Bradley County 500-kV substation. Highfill is a certified information systems security professional (CISSP) and holds bachelor's and master's degrees in engineering technology from East Tennessee State University. darren@enernex.com