In today's environment, a utility must have a reliable supervisory control and data acquisition (SCADA) system to monitor and control its transmission network, and to track the real-time system load. The traditional SCADA system consists of remote terminal units (RTUs) in the substations, a SCADA master station at the headquarters and a telecommunications network connecting the two. If any part of this system fails for an extended period of time, the economic consequences to a utility are devastating. With this in mind, Georgia Transmission Corp. (GTC), Tucker, Georgia, U.S., needed a reliable and effective method to protect its entire SCADA network.
GTC is a not-for-profit cooperative that provides electric transmission service to 39 electric membership corporations (EMCs) in Georgia. GTC owns approximately 2500 miles (4023 km) of transmission lines at voltages ranging from 46 kV to 500 kV. This transmission system provides points of delivery for the 39 EMCs at more than 700 substation locations around the state. These substations provide for the delivery of more than 6500 MW of power.
GTC uses a network of leased telecommunication circuits to communicate with RTUs located in its distribution and transmission substations. In 1999, the manufacturer of the telecommunications equipment advised GTC it was discontinuing hardware and software support for its proprietary equipment. This vendor's support was essential for GTC to continue the maintenance and expansion of the existing digital telecom network.
“Because the data from the substations is used to buy and sell power for our EMCs, plus monitor and control our substation equipment, we didn't feel that we could risk a major failure of our telecom network and not have vendor support to fall back on,” George Blomeley, GTC's senior vice president of system performance said. “It was obvious that we had to make a change and it had to happen quickly.”
The new telecom system had to support the existing SCADA communications and incorporate new requests. Protection engineers wanted the ability to access data from the substation relays at their desktops. Maintenance engineers needed remote access to the monitoring devices installed in the substation equipment. SCADA engineers wanted to access the maintenance ports of the RTUs and the revenue meters. Field personnel needed to logon the corporate local area network (LAN) to view e-mail and reliable voice communications from the substations.
In considering all of these needs, GTC chose to upgrade the network to frame-relay technology, which enabled the company to reduce its telecom operating costs significantly, while enhancing the capability and flexibility of the communication system. Because this technology is also an industry standard, many different manufacturers provide interchangeable equipment. This capability ensures that GTC can avoid a proprietary pitfall.
The frame-relay project consisted of replacing the equipment at the five existing backbone nodes with frame-relay equipment and switches. These five nodes use GTC's existing T1 backbone network to loop the state (Fig. 1). From these nodes, GTC interconnects with the local telephone company's frame-relay cloud to reach every substation.
GTC upgraded each substation with a frame-relay access device (FRAD), which is connected to a 64 kbps frame-relay circuit. This FRAD connects the frame-relay network to existing substation devices such as the revenue-metering packages, RTUs, protective relay panels and, in many cases, the EMC's substation equipment.
At GTC's headquarters, a diagnostics system was installed for continuous monitoring and troubleshooting of the network. With all the substation communications concentrated into the frame-relay circuit, it is critical to monitor the circuit continuously and to make all repairs quickly.
The project installed a FRAD at each of the 39 EMC headquarter offices. This provided the respective EMC with access to the frame-relay network for its communication needs. Several EMCs have used the network to connect their district offices to their headquarters, so they could send and receive voice and data over the frame-relay network.
The main purpose in moving to frame relay has always been to ensure a reliable communications network for the SCADA system. GTC realized many additional benefits.
GTC currently leases all its communication circuits through BellSouth and other telecommunications companies in Georgia. The monthly leased costs for the new frame-relay circuits are significantly less expensive than traditional digital circuits, even though the frame-relay circuits at 64 kbps have more capacity than the 9.6 kbps of traditional digital circuits.
The previous telecommunications system had only two communication channels that were limited to a maximum of 4.8 kbps each. With the new frame-relay circuits, GTC has seven communication channels and an Ethernet port, each with the ability to communicate at up to 64 kbps. GTC is able to communicate to the RTU, the electronic protective relays, revenue metering and smart maintenance devices in the substation via the same communication circuit.
The development of the original revenue-metering system centered around a bank of computers dialing-up the substation metering recorders once a day and dumping the load data into a translation system. GTC worked with the translation software vendor to modify the program to accept an Internet Protocol (IP) address and a telephone number. With the proper IP address, the program accesses the frame-relay network and links to the correct recorder in the field. The quality of the frame-relay circuit is much better than the voice circuit, so the baud rate could be increased and all 700 recorders can be polled within one hour. This will eventually allow the control center operators to crosscheck the revenue-metering data to the SCADA real-time load data to ensure no control area imbalances exist.
GTC's distribution and transmission substations contain many smart protective relays. While the relays provide needed fault information, the data was normally hard to collect because of problems with dial-up communications. The frame network acts as a gateway to connect the asynchronous communications from the relays to the LAN at GTC's headquarters. A protection engineer can logon the relay from his desk and upload or download any information he needs.
Frame relay is changing the way GTC looks at maintenance in the substation. Most of the substation equipment GTC currently purchases comes equipped with smart monitoring devices. Because of the large amount of data moving through GTC's SCADA network, the SCADA system limits the amount of maintenance data. However, with access to the frame-relay network, the substation maintenance group is beginning to develop a maintenance system that will communicate to the monitoring equipment directly.
GTC has voice circuit installations at each substation, primarily to communicate to the revenue metering but also to provide voice communications for the field personnel. The frame-relay network allows for portable Ethernet phones. Eventually, this will allow GTC to remove the substation telephone lines and use the frame-relay circuits for voice communications, resulting in significant savings for the company.
Field engineers and technicians, with appropriate security access, can access the corporate LAN from any substation by plugging into the Ethernet port on the FRAD. This allows immediate access to maintenance work order systems, relay sheets and previous test results for the substation equipment.
The EMCs can piggyback onto the frame-relay network and communicate to their own RTUs, meters and protective devices inside the substations. This could result in significant savings to the EMCs, which, in the past, leased data circuits or installed their own radio systems.
With the expanded communications capabilities, GTC is able to communicate to the maintenance ports of the RTUs, relays and meters. This will allow GTC to perform troubleshooting and configuration changes remotely, saving future trips to the substations by field personnel.
The proliferation of smart devices communicating to the RTUs has made it difficult to maintain RTU configurations. In the past, GTC kept a diskette inside the RTU with the current configuration on it. Many times, the SCADA technician would respond to a trouble call and discover the diskette missing or damaged. The technician would then have to take a standard configuration and modify it, which could take hours. Using the frame-relay network, GTC is currently uploading all the RTU configurations and putting them on the corporate LAN. In the future, when a technician is called out to a substation, he will be able to download the current configuration and install it in the RTU.
Frame relay offers many possibilities. The frame-relay network allows transportation of voice or data between locations on the network. One suggestion has been to use the frame-relay system as the backbone to an EMC statewide digital radio network. It also could tie the EMC PBXs together to create an EMC statewide phone system, eliminating long-distance charges between EMCs. An EMC intranet also could be created to transfer Internet-sensitive data. While some of these ideas would require purchasing additional telecommunication bandwidth, the frame-relay network is capable of supporting these options.
While the installation of the frame-relay network went relatively smooth, GTC learned several lessons from this project.
Even though the bandwidth was increased into the substation, the capacity of the circuit is limited. GTC programmed the frame-relay network so the RTUs had priority over Ethernet and other asynchronous communications. However, GTC will have to monitor the SCADA communications closely to ensure it is not affected by the increasing communication requirements from the substations.
GTC provided the BellSouth representative with an on-site office so he could closely coordinate his field personnel with GTC's field personnel. This greatly reduced potential problems and revisits to the substations. Before GTC personnel arrived at the substation to commission the frame-relay circuit, BellSouth already had verified that the circuit was good from the substation all the way back to GTC.
Even though GTC ran a pilot project before beginning a statewide implementation, the pilot project did not include substations in more rural areas of the state. After installing frame-relay circuits in some of these areas, GTC started experiencing a firmware glitch in the FRAD that caused the device to lock-up on noisy circuits. If GTC had included these types of substations in the pilot project, it could have fixed this problem prior to the statewide installations.
To ensure the relay project would be insulated from the normal chaos of day-to-day operations, GTC brought in a contract project manager to handle the logistics of coordinating the equipment and material, panel fabrication, contract installers and coordination with BellSouth. This allowed the project to remain on schedule even during periods of increased workload in other areas of the corporation.
Software applications designed to use voice circuits to communicate to substation devices also will work over frame relay. However, GTC discovered most of the software packages would have to be modified to accommodate an IP address and telephone number. GTC contacted these vendors early in the project to ensure that these programs would be ready.
The frame-relay project had to interact with internal and external areas of the company, including engineering, substation construction and EMCs. GTC's routine communication of project scope and schedule changes was essential to ensuring everyone was on the same page.
Some of the frame-relay circuits took longer to install into the substations because the local telephone company had to install additional equipment or cable on its system. Because several RTUs communicate to the same SCADA master port, these ports had to be bridged between the old network and the new network for months until all RTUs on that master port were switched to the frame-relay network.
Recently, GTC was commissioning a critical substation RTU when it discovered there was a problem with the RTU configuration. Noel Engelman, the electronic maintenance engineer responsible for this RTU was out of town at the time. “When I called back to the office to get my messages, I learned that there was a problem,” said Engelman. “I used my laptop to dial-in to the corporate LAN, and from there, I logged into the maintenance port of the RTU. I uploaded the configuration, made the necessary changes and then downloaded it back to the RTU. I was able to fix the problem from 3000 miles (4824 km) away.”
As the utility environment continues to change, GTC has in place a telecommunications network that will allow it to meet future needs.
David Van Winkle is manager of substation maintenance for Georgia Transmission Corp., which he joined in 1989. He received the BSAE degree from the University of Georgia and the MBA degree from Georgia State University in 1999. He has more than 12 years of experience in substation maintenance, testing and SCADA.
Keith Porterfield is manager of telecommunications for Georgia System Operations Corp., which he joined in 1991. He holds the BSEE and MSEE degrees from Columbia University and a J.D. from Georgia State University. He has more than 15 years of experience in the fields of SCADA and telecommunications.
Charles Nash is a principal consultant for EnerVision Inc., a consulting company located in Atlanta, Georgia, U.S. Nash received the BSEE degree from Tennessee Technological University. He has performed a number of projects implementing technology research, load management systems, and transmission and generation system electronic support for both distribution and generation/transmission cooperative clients.
Frame-Relay Basics for Utilities
Frame relay is a basic communications protocol that has been widely adopted by telephone companies in the United States. The power of frame relay does not lie in the protocol itself but in its almost universal acceptance by service providers and equipment vendors. One may implement a network based on a mix of Cisco, Motorola and Nortel equipment and have a high level of confidence that the systems will interoperate. Vendors continually implement products that accommodate asynchronous, synchronous and Ethernet connections.
The key advantage to frame relay over X.25 IP is its simplicity and efficiency. A permanent virtual circuit (PVC) is the basis of frame relay. The PVC establishes over a single physical circuit a virtual circuit with several logical connections. PVCs don't require a full address in each packet and instead use a control identifier that the network switches match and route.
Despite the advantages, there are several obstacles to overcome when implementing frame relay in a utility environment. Some utility applications, Remote Revenue Metering, for example, require switched connections. SCADA typically requires point-to-multipoint connections; native frame relay can support neither. In a large network, it is extremely expensive to support any-to-any connectivity, because each potential communication pair would require an additional PVC. The most common solution is to use a second protocol, encapsulating either an IP or X.25 packet within frame relay. The Frame Relay Access Device (FRAD) then becomes two virtual machines in one: an X.25 device which establishes dynamic and multi-point connections, and a frame relay device that acts as the aggregate transport link for the various X.25 connections. This approach provides the needed flexibility lacking in native frame relay and reduces the costs of multiple PVCs to each site.