Supervisory control and data acquisition (SCADA) was originally deployed at Grant County Public Utility District (GCPUD; Ephrata, Washington, U.S.) in the late 1980s to improve response times to outages. Most communication throughout GCPUD's 2800-sq-mile (7250-sq-km) rural service territory was provided by a 2400-baud radio system with a SCADA protocol that was proprie tary and unique to GCPUD. As we entered this century, the hallmarks of this system included bandwidth limitations, lack of expandability and communications failures. So in 2002, GCPUD initiated a project to upgrade its entire system.
Without any previous experience with modern protocols, GCPUD was facing many challenges. In prior projects dealing with leading-edge technologies, GCPUD had encountered equipment whose capabilities were overstated. Knowing the real capabilities and workings of the various devices was critical to project success.
Also, at the time, there was much protocol uncertainty. Serial DNP was the North American de facto standard, but deployment of networked DNP was just beginning. UCA deployments were limited, and IEC 61850 was not yet finalized. Furthermore, many other protocols were in use. Each had pluses and minuses; an obvious selection was not apparent. There was a lack of industry-recognized protocol-conformance standards; therefore, no one knew how well the products adhered to relevant standards. And, because the new system would likely incorporate multiple manufacturers' products, an additional challenge was to determine if the devices “played well” with others.
GCPUD's overall challenge was to determine if the entire system would function as expected and if it would provide the desired SCADA functionality.
GCPUD thought the best way to meet these challenges was to perform functional testing. Due to a shortage of in-house expertise, the utility solicited assistance from experts in the field and commissioned a white paper from Power System Engineering (Madison, Wisconsin, U.S.).
The broad scope of the white paper:
Outlined the entire SCADA procurement and installation process
Assessed the existing systems and provided a defensible reference point for future work
Offered an overview of the present state of the technology
Estimated the impacts of the proposed additions
Compared GCPUD's needs to available opportunities
Included recommendations and an implementation plan with guidelines
Outlined a final plan that included a general architecture, initial budget and a preliminary labor-resource determination.
One of the specific recommendations was to conduct a pilot project to verify product capabilities, standards conformance, interoperability and system performance. The first phase of the pilot — the “Bench Test” — was performed in the lab. Extensive bench testing would decrease commissioning time, GCPUD believed. The second phase of the pilot — “Substation Beta” — was performed at the first substation and allowed for identification and resolution of situations not adequately modeled during the Bench Test. Substation Beta also allowed GCPUD staff to gain experience with a deployment in a real substation.
THE BENCH TEST
Five panels were constructed with the IEDs selected, and the panels were configured to represent the way actual devices would be installed. The test panels were assembled and the testing was done in GCPUD's Electronics Shop. One of the main goals was to select possible devices for the Substation Beta. A significant component of the Bench Test was the selection of the OSI application layer protocol. Both UCA and DNP/IP were compared with identical devices.
Software and test equipment were also evaluated. Configuration software was studied for ease of use and functionality. Manufacturer-supplied value-added software was also evaluated. The substation and communications groups had little experience with substation local area networks; therefore, network analyzers were investigated. Documentation of the Bench Test results comprised part of the design and configuration of the Substation Beta.
BENCH TEST FINDINGS
The many issues discovered during the Bench Test justified the time and cost of this pilot phase. One of the major issues involved the remote control scheme, which required that a pulse be used to cause a breaker operation. The DNP protocol of one of the tested devices did not support a pulse command, so a pulse was simulated with two separate commands. During testing, one of the commands failed to be delivered and a control switch failed. In this case, the symptom was the failure of the switch, but the real failure was the system did not recognize that a control action had not been fully completed.
A second issue was identified while troubleshooting this problem. The Ethernet switch being tested was configured with its factory-default settings. This caused network traffic problems with the single-switch architecture and many data packets were not delivered.
A third issue was incorrect tag numbers in the DNP documentation for a control device. The manufacturer recommended one of two alternate control methods; unfortunately, remote control could not be established with this method. The second control method was eventually implemented with success.
A fourth issue was the use of an OSI communications stack, which was not identified in the vendor's literature. Because GCPUD's network was designed for TCP/IP traffic, communications could not be established.
Finally, several devices were supplied with manuals that were out of date, and several devices arrived with firmware that required upgrading before they would operate correctly on the network. By the completion of the testing, a significant amount of time had been dedicated to communications with the vendors to resolve these issues.
THE SUBSTATION BETA
The final pilot phase was to install equipment selected from the bench testing at Substation Beta. This site was selected for convenience and to be representative of other substations. Several additional problems were discovered and resolved during this phase.
The first issue was that the SCADA master had difficulties reading unsolicited report-of-state changes in the field devices. Although this was identified during bench testing, it was attributed to development software at the time. This problem resurfaced again during commissioning testing and many hours were dedicated to its resolution.
The next discovery was that network-routing parameters did not optimize bandwidth usage and would not allow multiple concurrent connections. This could have been identified earlier if the Bench Test had been expanded to multiple locations. However, time and resource constraints prevented this, so the problem was discovered and resolved during the Substation Beta commission testing.
Load tap changer (LTC) position indication required extensive rework to accommodate the exacting timing requirements of the LTC. Bench Test experience reduced the amount of time required to troubleshoot and resolve this issue. Substation Beta allowed the identification of these complications, but GCPUD decided to defer implementation until control algorithm improvements could be developed. Bench Test experience and documentation was a significant contribution to the resolution of this Substation Beta issue.
Finally, all of these lessons learned and experiences gained were documented to improve future designs.
BENCH TEST ADVANTAGES
The Bench Test was located in the shop, avoiding extended exposure to subzero or greater then 90°F (32°C) temperatures. The lighting was better. Exposure to harsh sunlight-caused computer screen washout was minimized. Furthermore, telephones and hardwired high-speed Internet access in the shop facilitated access to vendor assistance. This also simplified downloading firmware patches and upgrades.
The test panels were located on benches, and the test devices were at eye level. This placed fewer physical demands on the testers and allowed testing to continue for longer continuous periods of time. Simulated substation panels enabled rapid identification and resolution of problems. The shop was located close to other technical resources, which further reduced labor costs.
At the shop, testing involved fewer people at any given time. In an actual substation, at least five employees were required for any SCADA testing, while testing at the bench rarely involved more than two employees. In addition, the shop environment avoided the jurisdictional device “ownership” issues that are part of the substation environment.
Safety and training were enormous side benefits. During the Bench Test, technicians were only exposed to low-voltage power supplies. In addition, the isolated nature of the Bench Test prevented problems with devices or procedures from causing a customer outage or equipment damage. These items decreased stress and provided an atmosphere conducive to experimentation. The Bench Test also provided an excellent opportunity to train the workforce and to learn the capabilities, limitations and idiosyncrasies of each device.
Recently, standard conformance tests in IEC 61850 received final approval. Third-party suppliers or individual manufacturers can perform these tests. As of this writing, at least one organization is currently providing third-party conformance testing services and others are preparing to offer these services.
Standards-based conformance tests are an important requirement for the correct performance of interconnected IEDs. By itself, however, this testing is insufficient to guarantee correct operation of multiple IEDs from various manufacturers. A determination of the ways the IEDs interact and the ability of the complete system to perform as anticipated is an essential step that is outside the scope of conformance testing.
Existing standards at the time of this project lacked the degree of granularity necessary to ensure consistent implementation of protocols within devices across various vendors' offerings. For this reason, extensive bench testing was required. Future deployments that take advantage of standard conformance testing are expected to proceed more rapidly since testing should be limited to interoperability and functionality.
PREINSTALLATION TESTING IS KEY
The preferred time to find and fix problems is before equipment is put into service. Preinstallation testing is intended to evaluate devices, software and communications systems, and demonstrate that they will provide the desired functional capabilities. Even systems and equipment that are “mature technologies” can have hidden problems and issues.
Testing on the bench cannot cover every possible combination of devices and software and every possible operational configuration, but it can provide an excellent opportunity to evaluate prospective devices, test interoperability, train employees and develop expertise with new technologies. Knowledge and experience gained from bench testing will minimize, not eliminate, the surprises discovered in the field. The bench allows the deployment of new technologies more quickly and with less aggravation, which will capture the benefits of technology as soon as possible.
In the end, each utility must decide how to proceed with new technology implementations. Test in the field during commissioning with all the associated problems and operational issues, or test in a controlled environment. Recognizing that bugs and surprises will continue to exist, regardless of where and how much testing is done, GCPUD chose to test on the bench. Where and how to identify and fix problems is one of the engineering choices for a project. The success of each project is defined by our choices, so choose wisely.
The author would like to thank Tom Lebakken, John Tweedy and Dan Stark of Power System Engineering for their support and assistance.
Joseph White is the SCADA and apparatus engineer at Grant County Public Utility District. For the past 21 years, he has practiced electrical engineering for both publicly and privately owned utilities. His duties have included standards, substation design, distribution design, planning, maintenance and protection. He is a registered professional engineer in the state of Washington and a member of IEEE. firstname.lastname@example.org
BENCH TEST PANELS
Grant County Public Utility District (GCPUD) had five panels designed to represent actual substation configurations for the Bench Test. The panels included three configurations for feeder breaker protection and control; one configuration for the station communications node; and one configuration for miscellaneous communications and control.
Using individual panels for each IED allowed GCPUD to experiment with different options for control and protection. Also, by building panels for each option, communications between each panel could be evaluated and optimized.
Software was a significant consideration. Network analyzers, protocol analyzers, configuration software, and miscellaneous value-added software were tested and evaluated. The ability to “test drive” the software, in addition to the hardware, allowed GCPUD to experiment with the configuration and operational aspects of both the individual devices and the modeled sub-system.
GCPUD acknowledges and expresses appreciation to the following organizations for their assistance and cooperation during this process: AREVA, Barrington Consultants, Basler Electric, Schweitzer Engineer Laboratories, QEI, Beckwith Electric, Electroswitch, RuggedCom, CISCO Systems, Netgear, Dymec, Siemens, Comtrol, DataComm for Business, SISCO, KEMA and Applied Systems Engineering.