Overton Power District No. 5 (OPD5; Overton, Nevada, U.S.) has had a supervisory control and data acquisition (SCADA) system in service for more than four years. The system is highly adaptable because it was designed using readily available software and a flexible communications architecture that uses commercial off-the-shelf (COTS) equipment.

OPD5 assembled a team of integration engineers, network engineers and its own staff who were charged with developing a hybrid serial and Ethernet network, providing OPD5 the ability to grow, improve and modify the system as needs change.

OPD5 wanted to use Ethernet because of all of the research and development going into the business sector. Although numerous innovative technologies were available and implemented, many electric utilities had not put them into use. The business side of OPD5 already had success with Ethernet with its small office network that OPD5 planned to build upon.

Mission-Critical Networks

As part of the program to add a new control system, OPD5 wanted to combine its business and SCADA network to enable interaction between substations, dispatch centers, business centers and remote operator consoles. Because Overton Power is a small district, the staff must be flexible and knowledgeable about all parts of the organization. The ability to quickly, simply and intuitively move data and commands is paramount.

Also, OPD5 finds that many service calls can be performed after hours by staff from home, so the system needed the flexibility to support remote operator interfaces located in personal residences. OPD5 wanted to create access to data, yet provide secure communications to protect the power system and proprietary information.

For OPD5, the robustness of its business systems is of primary importance. Security constraints for the business system were even more demanding than the mission-critical SCADA system. The harsh reality is that failure of the SCADA system is inconvenient, but personnel can be dispatched to manually operate the power system. The electronic business system cannot be manually operated, and its failure prevents revenue collection and cash flow, which can be catastrophic to the utility.

Choosing an Integrated IED

OPD5 documented that availability and security are imperative criteria for measuring network performance. At this utility, intelligent electronic devices (IEDs) in the substation communicate via serial connections to SEL-2030 communications processors. The SEL-2030 communications processors are applied in the same configuration in both transmission and distribution class substations, and are set to collect and concentrate required IED data and provide specific control functions. They are physically located in station control buildings or mounted in outdoor enclosures.

Figures 1 and 2 illustrate point-to-point “star” topologies that provide fast, efficient and robust transmission of measurement data and control actions.

Modbus remote thermal unit (RTU) or SEL Input/Output (I/O) protocol is used between the SEL-2030 communications processors and the control centers. A second separate telnet link to the SEL-2030 communications processors provides engineering access to connected IEDs. This link allows operations to communicate directly with the IEDs through the communications processor from anywhere on the network, local or remote. Using the SEL-2890, OPD5 also added Web pages directly to each communications processor. Once the connection is established, the communications processor becomes “transparent” and passes data to and from the IEDs.

Communications between the Overton office and the service center in Mesquite, Nevada, travel via Overton's own Tl line. OPD5 uses high-speed asynchronous digital subscriber line (ADSL) WAN connectivity to the Overton office, Sandhills Substation, Tortoise Substation, Logandale Substation and system operator's houses.

As shown in Fig. 3, the Overton substation is connected to the Overton office via single-mode, fiber-optic cable. Glendale Substation is connected to Tortoise Substation via single-mode, fiber-optic cable, and Moapa Substation is connected to Tortoise Substation via multimode, fiber-optic cable.

In Mesquite, wireless Ethernet is used between the Bunkerville Substation, Yucca Switchyard, Mesa Substation, and several Q1000 meters and reclosers. Direct multimode fiber connects Painted Hills transmission and distribution substations to the Mesquite office. Mesquite and Pulsipher Substation is connected to the Mesquite office via single-mode, fiber-optic cable. Single-mode, fiber-optic cable connects Well 33 Substation to Mesa Substation.

Data Access and Security

The addition of Ethernet to each station supports portable human machine interface (HMI) and engineering access workstations. Laptop computers, when connected to the substation Ethernet network, provide local substation HMI functionality, visibility, access to the regional SCADA dispatch displays, and visibility and access to any HMI screen in any substation throughout the system. Engineering access (via transparent connections) is provided to all local IEDs, as well as any other IED in any other substation. These same HMI and engineering access capabilities are provided via various permanent workstations throughout the system. Figure 4 shows the communication rack in a typical substation.

Wonderware Manufacturing Management Information System Factory-Suite 2000 was chosen to provide HMI and data management. The Wonder-ware software used in the project included InTouch HMI, IndustrialSQL Server database and ActiveFactory client. Updates to Wonderware applications are automatically deployed using Wonderware's Network Application Development tools. This approach helps ensure timely and efficient upgrades to the HMI drivers and data management tools. One “View Node” application fits all present and future HMI installations. This simplifies the process of maintaining and adding functionality to the HMI system.

Security Access to Substation Data

Information security refers to the methods employed to ensure the privacy of substation data and information. But information security also addresses the integrity of data and commands as well as authentication of the source of received data and commands. Access security includes prevention of access by persons who are attempting unauthorized electronic intrusion into the communications system, as well as prevention of access to a legitimate user without appropriate permission.

The ubiquity and convenience of Ethernet has led to the expansion of Ethernet IP-based networks in utility environments. As with many utilities, OPD5 chose a hybrid design that relies heavily on proven, robust and direct-connect serial links merged with Ethernet to provide local area netwok (LAN) and wide area network (WAN) solutions. Unfortunately, Internet protocol (IP), as designed, has no native security mechanisms. Therefore, the transit of IP packets across untrusted WANs (such as the Internet) must be protected with security and encryption techniques.

The SEL relays and SEL-2030 communications processors in the integrated system design provide unparalleled LAN information security through passwords, permissions, and monitoring and control strategies. These features are available over traditional copper or fiber communications cables and wireless connections, as well as over Ethernet.

WAN Network Security and Processes

OPD5 chose to use IP security protocol (IPSec) to perform encryption. IPSec offers a rich variety of options for exchanging and securing keys and different levels of encryption strength and keying. The role of IPSec is to shroud the data in a secure form for transport across unsecured (insecure) networks. Many strong encryption formats are available. OPD5 selected equipment that can be upgraded via firmware as new encryption formats and enhancements are available in the future.

IPSec itself is actually a security framework of methods that provide confidentiality, integrity and authentication (CIA). Each LAN is a subnetwork, or subnet, secure by virtue of being physically separate from WAN connections. The OPD5 design provides CIA coverage between any two endpoints on different subnets in the system. These subnets are often connected by untrusted WAN connections.

For IPSec to do its job, a security relationship must be established between two endpoint peers capable of building an encrypted tunnel across the WAN. Once completed, the encrypted tunnel “virtually” connects the two endpoints as if they were on the same LAN or subnet, thus creating a virtual private network (VPN).

The ongoing security policies at OPD5 are a key part of its security system strength. The policies are unique to the OPD5 environment and are kept confidential; however, the elements of a successful security plan are outlined. Ongoing data collection, monitoring and management are essential to keep a security system up to date and capable of thwarting the next new attack.

Network Security Plan Outline

An organization can develop a single all-encompassing policy or a suite of policies. At the minimum, the policy should include company positions regarding:

  • Viruses and antivirus software
  • Operator and end-user access
  • Physical security and monitoring
  • Operating system security standards
  • Remote computers
  • Remote access and wireless communications policies
  • New employee orientation
  • Departing employee procedures
  • Password complexity policy
  • Password change policy
  • Backup and recovery of files
  • Vendor/visitor policy
  • Employee email and Internet acceptable-use policy
  • Audits

These policies should be put into practice as well-understood processes. A primer on creating security policies can be found at http://www.sans.org/newlook/resources/policies/Policy_Primer.pdf.

Data Confidentiality and Integrity

Using CIA coverage, OPD5 data are protected from malicious outsiders by firewalls and access control and authorization as defined in the security policies. The confidentiality of the data is ensured by encryption technology. Integrity is provided by shared secrets between two trusted endpoints as well as identity mechanisms used to validate the authenticity of the peer, as shown in Fig. 5.

Authentication is provided to end users by secure log on to Windows 2000 active directory domains and by the complex negotiation of session secret keys that last for a defined number of seconds or for the amount of data transmitted and then are changed.

IPSec includes mechanisms for providing enhanced IP transit, including Encapsulation Security Protocol (ESP) and Authentication Header (AH). OPD5 chose to use ESP because of its ability to enfold the entire message (or datagram) being sent in a secure wrapper. Furthermore, the method includes the real internal source IP address in the original IP header, as seen in Fig. 6. This is important because the encapsulation of the actual originating IP address inside of the encrypted wrapper hides important information from a potential hacker and makes the connection more secure.

For the virtual private network (VPN) endpoints to build their secure tunnels, a process called the Internet Key Exchange must occur. Matching security policies on OPD5 WAN connection endpoints ensure that a security trust relationship can be built so that key negotiation can occur in a secure fashion. The Internet Key Exchange process is illustrated in Fig. 7.

The addition of system and security event logging provides another level of protection and brings all the information from geographically dispersed sites to one logging console.

OPD5 uses a Windows 2000 domain for authentication and security of computer and user accounts. A domain is a logical grouping of computers, users, printers and services in a common security framework. The OPDS domain services log-ons for end users of the system and printer file services and files on the network drives. As these services are accessed, event records are logged in an event record database. This security database is accessible from anywhere in the network by the computer system administrator. The responsible administrator can tell who has logged on and for how long, what objects or files were accessed, and if there seems to be a security problem, such as locked-out accounts or records of failed log-on events.

Conclusion

The role of security policies and well-defined processes cannot be understated. Periodic maintenance upgrades, password changes, and security audits and improvements are keys in the cycle of continuous security review. The continuous improvements brought about by the new SCADA/EMS system help ensure that OPD5 will remain a technically and financially secure power provider for many years to come.

Kevin Streett majored in construction management at Boise State University. Prior to joining Overton Power District No. 5 in 1988, he worked as a construction foreman and supervisor in the United States and overseas, building substations, transmission lines and distribution systems. He is presently operations supervisor at OPD5, responsible for substation construction and maintenance, metering and operations.
kstreett@opd5.com