Those who have been working with the North American Electric Reliability Corporation's (NERC's) critical infrastructure protection (CIP) standards for any length of time no doubt have already experienced CIP drift. Drift usually occurs because with CIP, each utility staff participant forms his or her own interpretation about what meets compliance and what does not. Without a compass or CIP audit barometer, electric utilities must focus teams or individuals on addressing these differences of opinion and presenting a unified CIP program or a cohesive set of single programs. After all, in a short period of time, there is a need to reveal to an audit team the functions of the CIP compliance or cyber security program. Start by keeping it simple.

Standards, Status and Gaps

The NERC CIP standards are applicable to a wide swath of electronic hardware, software and business processes, and include operations, legal and regulatory departments. CIP standards have roots and language originating in the information technology (IT) security field, with the intention of securing utilities' transmission and generation systems, reducing risk and, hence, making the bulk electric system more reliable. These standards are overlaid across qualified transmission and generation equipment and architectures, much of which were designed with reliability in mind, but probably not IT security. One can argue the value of this effort, but the reality is there is a convergence of IT-type systems and utility functions.

An obvious approach to CIP is to operate a sound cyber security program, with program elements and controls meeting CIP compliance. The intention is to increase the reliability of the bulk electric system, is it not? But how does a utility map a cyber security program — with concepts based in risk, controls, forensics, computer incident response and compromise — to the regulatory CIP requirements? And who is required to complete this daunting task? And what is the measure of success?

Garnering the coordination and cooperation of system owners, security personnel, transmission/generation operations and regulatory in the same room on a regular basis is helpful. But, to some extent, each must be able to walk in the others' shoes and agree on direction. It is also critical to identify gap areas where one group is responsible for another's compliance. The picture of status includes gap responsibilities.

The Starting Framework

A good way to maintain and unify a program is to diagram or whiteboard the framework as a simple cycle:

• Inventory cyber assets and include access point and systems to monitor and control electronic security perimeters; inventory is critical.

• Identify who has access to the inventory.

• Ensure the inventory is updated when changes occur and that changes are tested.

• Provide an ongoing maintenance cycle for changing passwords, installing patches, reviewing ports and services, verifying monitoring, and checking logs and log parameters for the inventory.

• Make backups of the inventory.

• Confirm the ability to restore the inventory.

Also, the inventory must be segmented physically and electronically by design. When this design is implemented, it must be documented and used to describe compensating measures. This is required throughout CIP, but first may require the purchase of technologies or changes in design or both. Procedures, policies and standards are needed, but the simple operational framework must be addressed first.

Part of this framework is identification of a set of operational tasks. The tasks are the maintenance plan for the CIP program. The tasks can be taken directly from the CIP standards, such as performing a review of default accounts. Review each CIP standard until it is an action item or list of instructions. Once those are described, identify two types of tasks: a date-driven set of tasks and an event-driven set. Date-driven tasks include time-driven actions such as annually, quarterly or every 90 days. For example, for annually, on a given date, the utility will review the inventory of critical cyber assets. From the tasks, one can identify a controls test matrix, which will allow for auditing the task completion quality. In basic terms, there is a task, a task owner, verification the task meets the compliance criteria and verification the task is completed.

Event-driven tasks are based on an unplanned event, such as the release of a security patch, a NERC alert or a system event related to cyber security. The focus is seeing the distinction between the two types of tasks and following a defined procedure for each. The formula is a task, a documented procedure and an evidence documented output from the operation.

Once tasks are defined, follow up with documented procedures or policies. From a documentation perspective, there are multiple ways to unify the cyber security program. It is recommend system owners distinguish first between procedures and work instructions, not necessarily using work instructions for primary audit or program evidence. Work instructions are specific operational or system vendor tasks such as: “Microsoft instructions to restore system Y, by copying file X to drive C and execute command R.”

Work instructions are not required for an audit. Rather, they are used for specific system maintenance. These instructions may be required at some point and may need to be shown as part of a detailed data request, but they are too complex to use when explaining a CIP program. Keep it simple.

Outlining Procedures

Use CIP regulatory language and use the same format, numbering, titles, classification and content for all departments. Format policies and procedures differently so they can be clearly distinguished and identified. If specific systems procedures related to CIP do not exist, draft CIP specific parent procedures using CIP standards for numbering, such as: D1 AC IN-0200 - Cyber and Critical Cyber Asset Identification Procedure. These “parent” procedures will have the task language as content (that is, review and update the list of critical cyber assets annually).

Next, identify each CIP requirement that requires a “child” document and label it with an alphabet classification title or other distinguisher in the document. For example, the aforementioned CIP-002 procedures require a list of critical cyber assets. In this case, the list would be titled something such as D1 AC IN-0200A - Cyber and Critical Cyber Assets. Note the “A” in the title. This is to distinguish it from the parent procedure and distinguish it as output evidence of the completed procedure. In some cases, there may be multiple child procedures — C, D and so forth for a single parent — but all are tied back to that single-parent procedure. In essence, there is a procedure with a related output evidence document.

It also is key to assign a document organizer role to maintain the structure and organization.

With tasks, procedures and output in place, there is a need to maintain organization across business units. This is the most difficult task since business units may use different maintenance programs or ticketing response systems. Use a calendar function, reminder system or trouble-change control system to schedule jobs for the identified tasks. For example, each June 1, a job kicks off to review D1 AC IN-0200A - Cyber and Critical Cyber Assets. Once it is reviewed, enter the review data in the ticket system. The utility now has a program scheduled with tasks, procedures and evidence. There are other ways to schedule, as well.

Another potential way to unify the program is to identify and agree to areas where both individual business units can maintain compliance and areas where unified multidepartmental approaches are required. An electric controls system cyber security policy could be part of other corporate policies, but implemented at each applicable business unit. The change control and configuration management portion, however, is best left to system owners. The downside to this framework is that it is difficult to explain at audit time.

Identifying Risk

Many utilities put a due amount of diligence into risk discussions. Some question what the risk is to certain events, and rightly so. What if someone could access a utility's control network, substation equipment, generation turbine controls, or environmental or chiller plant equipment? What is the risk? CIP asks for a risk-based assessment, and the risk is tied to the cyber or critical cyber asset equipment. Utilities would be wise to spend some focus here and have the ability to argue the nth degree on its behalf.

Legacy utility systems were built for reliability, and a risk-based assessment of the impact to the bulk electric system may reveal that systems are or are not as critical as first considered. A good strategy is to migrate all nonessential cyber asset systems into a demilitarized zone (DMZ), a network outside of the control systems network. This reduces the CIP footprint and identifies the CIP coverage scope. But primarily, be prepared to argue the risk, and clearly show how and why system security posture and risk are related. If a utility can show risk has been examined, and sometimes to multiple degrees, it is easier to clarify and communicate the program.

Simply put, older technologies usually lack the ability to be automated for logging, patching, accounts, passwords, backups, monitoring or automated audit configuration checks. Maybe these tasks were not part of a regular maintenance cycle. There are third-party equipment and application security solutions that can be placed at strategic locations on the network. However, these may not interact well with or be scalable to utility infrastructure goals. They also require investments in technologies and qualified staff to maintain and operate. The greater the risk, the greater likelihood this investment is required.

Consider also that manual processes pose risks. Either way, at least consider centralizing the management and maintenance of the systems that can be automated, and manually maintain and document the others.

What is the measure of how a program is functioning and to what extent? And how can the quality of a sustainable program be shown? Being able to perform an internal quality analysis of a program is critical. There are many methodologies for doing this, including audits, testing controls and so forth, but these approaches can be expensive, time consuming or inaccurate. In addition, while critiquing the work product of others, it is easy to create rifts between groups. A voice of reason is required here. It is good to rely on a combination of IT and utility staff.

Transmission and generation may need to be more willing to let their IT or regulatory partner make recommendations or assist them in their work and operations. IT and regulatory need to understand the unique operations or legacy environments; and regulatory or legal may need to argue on behalf of their groups and the industry. It may require staffing a person on-site until the program operates. After all, many utilities have environmental requirements and operate environmental programs in the same manner. Measuring the program requires an unbiased and qualified voice to obtain a statistical measure of confidence.

For measuring a program, applications such as the Department of Homeland Security's Control Systems Evaluation Tool (CSET) can help, but it is the quality of the program tasks and organized evidence that will produce consistency. The CSET is only as good as the implementation of procedures, policies and maintenance activities. It is a good starting point tool for evaluating and organizing, at least on the interim. On a larger scale, governance, risk and control applications do well, too.

Using statistical sampling of evidence quality, along with a well-organized evidence library, will yield valuable program status measurements. Depending on the size of the organization, sample about 10% of the evidence if resources allow, and once these samples are completed, use them for drafting management status reports. In addition, consider peer business unit reviews to give an unbiased purview. Pay attention to gap areas where one department is responsible for compliance or tasks. Those are critical and can fall below the radar.

Reaching the Goal

At this point, there is a sustainable, documented and measurable program. There are great amounts of in-between work not covered here, but the basics are provided. Following a technical defense-in-depth security posture is one addition, and the identification of controls, testing and role ownership, and who will cover those physical, technical, and administrative controls, is another.

Also, there is no replacement for ongoing security training and awareness at the front lines. Operators who, day in and day out, maintain and observe the grid or patch the server are the front line for potential security events. They know the business, how to respond and escalate, and can add security to their extensive resume of skills. Transmission planners, engineers and IT also know their business and how to help solidify the program. Rely on them.

At the end of the day, CIP will prove to be a vast security awareness program. Many professionals already make arguments or raise questions about the overall value or effectiveness of these standards. Opinions and findings in the area of a cost-benefit analysis should be considered. Has the industry really reduced risk to critical assets, or could the cost and effort be better spent on training? These same questions were asked about environmental regulations 10 years ago, but are mere afterthoughts today. As the convergence and development of technologies, applications, microsystems and networked systems become more prevalent within the utility industry, there will be a continuing need for an established set of rules to prevent potential failures.

---

Barry Jones (bjones@rohan.sdsu.edu) is a project manager for a utility CIP initiative. He is a registered certified information systems security professional, with a background in IT enterprise network engineering and operations and NERC CIP compliance.

Sample Basic Event Diagnosis Matrix

Symptom	Denial of service	Malicious code	Unauthorized access	Inappropriate usage
File access attempts	Low	Medium	High	Low
Tier 1 system	High	High	High	Medium
Port scans incoming, unusual	High	Low	Medium	Low

Company mentioned:

National Electric Reliability Corporation www.nerc.com