The security of facilities and data has always been an important consideration for the electric utility industry. However, cyber security has become far more important than anyone might have imagined only a few years ago. Regulators, utilities and vendors are rushing to meet an entirely new taxonomy of challenges in the management and operation of supervisory control and data acquisition (SCADA), energy management systems (EMS) and distribution management systems (DMS).
A Thorny Issue
One especially difficult challenge falls squarely on the shoulders of the electric utility SCADA/EMS/DMS system operator, or the dispatcher. Traditionally, system operators have been recruited from the field operations side of the utility, but few have an information technology (IT) background. They primarily have been concerned with ensuring the safe, reliable and efficient operation of the electrical grid. However, in today’s world, the system operator is on the front line in the event of a cyber attack. Unfortunately, few system operators are likely to have the IT knowledge and experience required to recognize and respond to such an incident. This is an especially thorny problem for small- to medium-sized utilities without the luxury of 24/7 on-site IT support staff.
Not surprisingly, situational awareness is perhaps the single most important asset in responding to a cyber security incident. But comprehensive situational awareness in a complex control system is not an easy thing to provide. Business requirements of modern control systems make these systems quite complicated. Control systems may have dozens or even hundreds of client and server computers, historians, communications servers, networking devices, security monitors and log managers. Assuming each of these has excellent security monitoring capability, it is still exceedingly difficult to provide a system operator with the comprehensive and up-to-the-minute situational awareness necessary to detect a cyber attack, much less effectively respond to it.
DOE Development Funding
In 2011, Siemens’ Smart Grid division received a grant from the U.S. Department of Energy (DOE) to develop a tool called Cyber Security Manager (CSM). The goal of CSM was to monitor control system components and the system’s networks for signs of intrusion or other security-related problems, and to communicate its observations to operators in terms they would understand.
Furthermore, CSM would be able to suggest corrective actions operators could take. The DOE required Siemens to work with a team of industry advisors to ensure the project would consider the very real needs of electric utility control centers. The project recruited advisors from CenterPoint Energy, Omaha Public Power District, Sacramento Municipal Utility District, New York Power Authority and Westar Energy. Additionally, cyber scientists and analysts from Pacific Northwest National Laboratory (PNNL) participated. Development of the product is nearing completion and a companion demonstration project on a live customer-owned control system is well underway.
Watch a Lot of Stuff
A key to cyber security situational awareness is to watch a lot of “stuff.” Stuff includes things like system log (syslog) messages from servers; network traffic (netflow) data coming from networking devices; output from anti-malware scanners; application logs from EMS, DMS and SCADA systems; alarms about multiple failed login attempts; failed control actions; attempts to use insecure tools and utilities (for example, FTP and telnet); and network packets to and from unknown Internet protocol (IP) addresses, or to and from unexpected transmission control protocol (TCP)/IP ports/services. It also includes out-of-date virus signature databases, digital certificates nearing expiration and a long list of other items.
Though there are ways to find all this information without a new tool, there is no existing single solution that can gather this information and quickly and efficiently present it to a system operator in a language the operator will understand.
The goal is for CSM to be this comprehensive, understandable tool usable by a system operator. CSM does not replace other monitoring and forensic tools used by cyber security administrators. Rather, CSM distills information into actionable insights for control systems operators and control systems support analysts by creating and communicating the following:
- Alarms. Sending these to its own web-based user interface and, if desired, to the control system itself for inclusion in alarm lists or other user interface displays.
- Control system logs, alarm messages and SCADA data. The mechanisms used to obtain data from the control system permit bidirectional data flow.
- Syslog messages. Cyber security best practice is to preserve system logs for use in post-event analysis and forensics, as well as for use potentially by law enforcement agencies for investigation and prosecution. To ensure complete logs are available, CSM writes information to a syslog server, such as any alerts or alarms it generates, summaries derived from netflow data, and intermediate information used in its decision-making processes.
Operator User Interface
CSM has a menu-driven, web-based user interface with more than 50 displays (pages). A small minority of these displays are intended for use by system operators. The others are for use by system engineers, support analysts and trainers as part of CSM’s built-in operator training simulator capability. CSM can act as a stand-alone operator training simulator, or it can be used in conjunction with the Siemens Spectrum power operator training simulator. Access to the displays is role based.
Following recommendations by the industry advisors, CSM places alarms into six categories from a security alarms table in the top-level display. Typically, if a control center alarm category is active — indicating there is at least one current situation that has an alarm — the user can drill down by clicking a detail button. The user can see additional detail about individual alarms.
Additional elements on displays can be used to view and enter information:
- Recommended actions to be performed by an operator or support analyst; these are configured by the CSM owner to adhere to a utility’s policies and procedures as well as to use vocabulary familiar to the user
- Contact information for on-call support personnel
- Notes about what was done to handle a specific event
- CSM sensitivity, which can be adjusted based on the current security posture of the utility’s or partner’s systems. For example, if a corporate local area network (LAN) or an inter-control center communications protocol (ICCP) communicating partner is known to be under attack, it may be desirable to increase CSM’s sensitivity to connection attempts through perimeter firewalls.
CSM was developed to run under IBM’s AIX operating system and uses an Oracle database. Initial development and testing has been done with Siemens’ Spectrum Power 3 Energy Management System and the design is such that using CSM with another control system is intended to be easy. A port to Linux is currently under investigation to make CSM accessible to a wider audience.
CSM is designed primarily for use in a comparatively static networked system like a control system. It is configured to know what the components of the control system are and how they communicate among themselves and the outside world. CSM is not designed to operate in a dynamic network such as an office local area network, where changes are frequent.
CSM is not a stand-alone magic bullet for cyber security. However, it holds the promise of being able to collect information from a variety of disparate devices and distill that information into actionable intelligence for use by a control system operator. This allows the system operator to become an active participant in recognizing and responding to cyber security incidents.
The authors would like to express their appreciation to Mark Flanary, PMP (Siemens project manager), Mike Stemper (CSM product manager) and Dave Taylor, CISSP (CSM principal investigator) for their assistance in preparing this article.
The authors also note that this article is based on work supported by the Department of Energy under award number DE-OE0000517.
Valentine A. Emesih (firstname.lastname@example.org) is director of the control systems department in CenterPoint Energy’s electric grid and market operations. He is responsible for various control systems used to securely monitor, manage and control advanced metering system meters, as well as electric distribution and transmission system field devices. Prior to joining CenterPoint Energy, he held engineering, system development and project management positions for electric utility automation systems vendors — Ferranti International Controls (now Ventyx -ABB ) and Johnson Controls (now ARINC Inc). Emesih has BSEE and MSEE degrees from The University of Texas at Austin and Auburn University, respectively, and is a professional engineer.
Bruce Oliver (email@example.com) is the supervisor of energy management systems for the Sacramento Municipal Utility District’s grid operations and planning. He is responsible for systems used to control and monitor the generation, transmission, distribution and gas pipeline systems. He has a BSEE degree from California State University, Sacramento, and is a licensed professional engineer in California.
CenterPoint Energy| www.centerpointenergy.com
Department of Energy| http://energy.gov
New York Power Authority| www.nypa.gov
North American Electric Reliability Corp. | www.nerc.com
Omaha Public Power District| www.oppd.com
Pacific Northwest National Laboratory| www.pnnl.gov
Sacramento Municipal Utility District| www.smud.org
Westar Energy| www.westarenergy.com
Sidebar: Sample Use Case: Attempts to Breach a Perimeter Firewall
A typical best-practice configuration is to isolate an intercontrol center communications protocol (ICCP) server with firewalls from the control system from which it receives data, and links to neighbor and partner utilities. In such a configuration, communications between the ICCP server and partner utilities very likely employs a routable protocol (for example, TCP/IP). This means the local control system does, in fact, depend on those partner utilities for at least some small part of its security. If a partner utility’s control system is compromised, the attack surface of the local control system does increase somewhat.
It is incumbent for a utility not only to configure the two firewalls in a least-privilege manner, but also to closely monitor attempts to create connections through those firewalls. One perfectly valid way to do this is to configure the firewalls to create a syslog message when a connect attempt is denied, route those messages to a log manager, or security information and event management (SIEM) tool, and configure the log manager to issue alerts or alarms, perhaps sending messages to support personnel. CSM can enhance this process as follows:
- CSM is configured to receive syslog messages (either directly from the firewalls or from a log manager/SIEM tool).
- CSM is configured, using something called security event criteria, to parse those messages.
- Security event criteria contain information that specifies how to parse the messages using regular expression processing, as well as information about how many rejected messages are required to generate an alert to a network support analyst or possibly an alarm to a system operator.
- CSM can be configured optionally to create an alarm on the control system itself.
Sidebar: Sample Use Case: Requirements for TCP/IP Ports
Various North American Electric Reliability Corp. (NERC) critical infrastructure protection requirements are focused on TCP/IP ports and services. Specifically, the system owner is required to perform periodic assessments to ensure only those ports and services required for system operation are enabled. These assessments are typically performed with a network scanning tool such as Nmap or Nessus.
Cyber Security Manager (CSM) also is useful to network support personnel as a tool for discovering misconfigurations and investigating suspected intrusion attempts. Even a small network can generate a huge volume of netflow data, far more than a network engineer can examine without specialized tools. In configuring CSM, a system administrator includes information about IP addresses and ports and services:
- IP addresses of all servers, clients and networking equipment that are part of the control system
- Information as to which clients are expected to communicate with which servers, including client and server ports and protocol.
Connection from an unexpected server port likely would be declared a violation by a NERC auditor. And, the details of such an instance typically would not cause an alarm to be issued, nor would it ever likely be of interest to a system operator. However, CSM can be a part of a continuous security assessment and compliance regime, as well as a powerful tool for use by network support personnel in investigating network misconfiguration and misuse.