Utilities have been making great strides in optimizing operations and maintenance activities while enabling grid-edge and smart grid improvements. Throughout these efforts, a key issue has been the ongoing tension between the information technology (IT) world and the operations technology (OT) world.
Individual interviews with utility decision makers, consultants and thought leaders, together with an analysis of recent results from research by Penton Media involving participants from U.S. electric utilities, provide insights into the great ways in which the industry is addressing IT/OT convergence, but also the key areas in which additional improvements are being sought.
IT Versus OT Solutions
What is the difference between software solutions used at utilities and software used for personal computing needs, or software used in other industries? Although some overlap exists, there are significant differences.
For one thing, the software systems utilities use on the operations side — which include outage management systems (OMS), supervisory control and data acquisition (SCADA), distribution management systems (DMS) and others — are all highly specialized. In contrast, the solutions used for personal computing are not. Similarly, cross-industry enterprise solutions such as enterprise resource planning (ERP) also involve few-to-many mapping, where a few types of software solutions serve the needs of many types of users.
This few-to-many dynamic creates economies of scale for IT solution providers who can afford to develop robust well-integrated solutions, whether for a Mac or PC on the personal computing side, or the ERP solutions of SAP, Oracle or the like on the enterprise side.
However, when it comes to utility OT solutions, it is not a mapping of a few software solutions to many users. It is a mapping of many specialized software solutions to a relative few users. The economies of scale and benefits of standardization can be impacted negatively in many ways, such as higher software license fees and associated costs for integration, support and other maintenance-related expenses. In addition, the relative scarcity of associated specialized skills can create additional costs as well as uncertainties about achieving and maintaining full-solution benefits.
Benefits of IT/OT Convergence
The benefits of surmounting IT/OT difficulties are tremendous. However, a repeated theme for achieving those benefits is designing systems with the end in mind.
Chris Kelly, vice president of electric systems engineering at National Grid, said, “We have to be very sequential in how we do things. We have our O&M folks who install the equipment, and we have communications systems personnel who do the commissioning of the communications systems, and we have the power folks who do the power-side commissioning.”
The result? One of the reasons why IT/OT convergence can be so beneficial is it can enable the communications and control for grid-edge devices to be standardized.
Given the various teams in these more complex value chains, a lag in one part can slow down the whole installation process. Kelly explained, “With IT/OT convergence now, we are more confident that these systems can be installed and they will work without our having to address how to get each device to fit into our communications network. There would be too many one-off situations very quickly if we had to worry about the actual protocols going on with these various grid-edge devices. And the benefits include being able to eliminate the need for one of those specialized skills sets to have to touch each type of device.”
History of Divergence
Jim Taylor, CTO at Tucson Electric Power, paints a multi-decade picture about how things have changed from an IT/OT perspective. Taylor recalls, in the past, sizing and configuring the distribution system relied on rules of thumb, typical design requirements and the experience of employees familiar with the requirements to provide service. This usually involved personnel from planning, engineering and field construction.
In today’s environment, with the increase in distributed solar generation, planning processes and related engineering solutions are much more complex. Taylor stated, “Even though the field work may seem similar, things have changed considerably. Here’s a recent example: We were preparing for a volt/VAR optimization pilot project and wanted full implementation in six months for two different substations and multiple feeders. During the project kickoff meeting, approximately 25 different departments were represented. The involvement from so many departments is indicative of the rapid change in our industry and increasingly specialized skills required to run our systems reliably and effectively.”
Taylor added, “In the future, we will need employees with a wide range of skills who can support cross-functional activities such as communications and protection for distribution automation, distributed energy resources, demand response, storage and the IT/OT systems that will support real-time data-driven decision-making and operations.”
History of Convergence
Jim Tarpey, of Grey Hare Advisors, had a different take on those old rules of thumb. Reflecting on his prior 40-year utility career in which he spearheaded strides in smart grid efforts and improvements in cost, reliability and safety, Tarpey says he had the ability to address IT/OT issues within one group, which reduced conflicts and led to significant benefits.
“For us, it took good engineers and good thinking to take the IT tools and refine them, and get our IT and OT systems where they really needed to be — with employees with a long-term commitment,” Tarpey said. “Our group included ex-linemen and ex-field supervisors who came up through the ranks and knew the system and its issues. If a circuit was out, they knew where it was and how to fix it.”
Now, five years later, things are different. Tarpey agrees technology is improving things, but it is an uphill battle. “Improved efficiency led to reduced staffing levels across field, operational and back-office areas, which is generally a good thing,” Tarpey noted. “But when you combine reduced staffing levels with the lower levels of employee retention we see these days — especially in the IT world — it calls into question whether the tremendous amount of data that utility personnel now have at their fingertips will make up for the loss of institutional knowledge and commitment.
“To what extent do you cede control and inherent operational safety of the grid to a machine versus keeping a level of human control?” Tarpey asked. “And what are the reliability impacts when employees no longer see their utility as their lifelong employer — whether on the IT side or the OT side? Solutions may be pushed to be short term, rather than a strategic long-term solution, and require the next guy to implement yet another short-term fix.”
His former utility’s successes with its IT/OT initiatives were facilitated by everyone working together following a long-term overarching strategic plan. In contrast, when IT and OT groups operate independently, it can sometimes lead to conflict and difficulties, an observation echoed by numerous decision makers interviewed by Penton Media.
Turf Wars
Fred Fletcher, who recently retired from his position as general manager at Burbank Water and Power, has seen great improvements in recent years but recalls how different things were prior to 2004, earlier on in the evolution of Burbank’s IT/OT integration improvements. Back then, Burbank had classified certain systems such as SCADA as being purely OT, which enabled operations personnel to have the related IT systems under their own direct control. However, over time, Fletcher admitted, “We had to come clean and recognized it as part of IT.”
Also of interest is the following tidbit, perhaps best kept anonymous, from a former CIO at a large municipal electric utility, who shared about some more direct manifestations of turf wars, where OT personnel “gave up” some IT systems to the IT department’s control, but retained what the CIO called “shadow IT” solutions within the operations department’s control.
Why have these turf wars occurred? Fletcher’s example on the city of Burbank is instructive of many. Back then, Fletcher said, “Some IT personnel did not understand operations and thought they owned the data and thought that the underlying systems on the operations side should be subservient to their IT departmental priorities. Perhaps they thought this because the data from those systems was embedded in their IT systems.”
Because of some work IT personnel needed to perform, an IT department staffer wanted to know when IT could disconnect Burbank’s energy control center from the internet. Fletcher said, “They were stunned when I insisted they couldn’t do that, because they were equally insistent that they had to take it down!”
Ten years later, “Now our OT manager also leads the IT,” Fletcher said. “As it turned out, the OT job is bigger than the IT job, and you don’t need two CTO-level people.”
The Paradox
Are there too many disparate, small pilots going on without any overall design for IT/OT-related R&D work across the electric utility industry generally, thereby making lessons learned a lot more difficult from an industry-wide perspective?
Interesting answers come from Jason Dedrick, a professor at the school of information studies at Syracuse University. Dedrick thinks there is a tendency for each utility to see its situation as unique, which “drives every utility to want to test new technologies or processes in its own environment.”
“While there are certainly venues for sharing lessons learned among utilities, and the lack of direct competition with most other utilities should encourage sharing, I think the sense of uniqueness limits the extent to which utilities will accept lessons from others,” Dedrick said.
A recent IT/OT-related study by Dedrick (supported by the National Science Foundation), which he presented at the last Grid of the Future Symposium in Chicago, Illinois, U.S., referenced the spend on R&D being at a low level of about 0.3%. Dedrick’s presentation, “Adoption of Smart Grid Technologies: Results of a Survey of U.S. Electric Utilities,” also analyzed utility deployment levels across the top grid-side and customer-side technologies, along with associated integrations between the two.
“Because of regulation, it’s probably difficult for utilities to justify more spending on R&D that can’t be added to the rate base, or be used to develop new products or services and increase profits,” Dedrick said. “I’ll be interested to see if this will change in a place like New York, where the REV [Reforming the Energy Vision] is pointing to opportunities for utilities to develop new business models in the future, possibly by deploying new technologies.”
Of additional interest in Dedrick’s study was the fact participants’ highest motivator for adopting smart grids was improved reliability, while the biggest obstacle was technological immaturity, even though some of the main technologies in question are relatively mature, such as SCADA and OMS. In this regard, a key driver is likely “utility personnel reflecting the regulators’ emphasis on prudent/used and useful investments (for IOUs), and the need for coops and municipals to invest in things that have direct benefits for members or citizens,” Dedrick pointed out.
Slowing Down Grid Improvements
While the industry’s engineers are highly capable and can solve a wide range of technological challenges, utilities have had to learn to address a range of sticking points:
- Lack of cultural fit between IT and OT groups within utilities
- Immature market for full usage of new solutions by utilities or their customers
- Lack of prompt, full regulatory enablement to make it possible for the full benefits of new solutions to be deployed in a timely fashion and delivered widely
- Technology lag between different systems (for example, communications platforms, control systems and the T&D assets involved)
- Fundamental need to redesign processes within the utility.
Regarding the impacts of regulatory concerns on grid-edge and smart grid improvements, National Grid’s Kelly said, “In terms of working with the commission’s regulatory processes, a key lesson we’ve learned is that you have to be fairly confident in the product and solution strategy and choices since approvals can take a year or more prior to the installation process getting underway. Things can easily change along the way, so the concept of future-proofing your technology choices or solution design, and introducing flexibility to make changes through market evolution, is an extremely important aspect of it.”
Regulatory approvals are not the only activity causing a lag in the uptake of new IT solutions. As pointed out by Richard Engelbrecht, of Rochester Gas & Electric (RG&E), the mix of substation assets and associated communications and control equipment varies for many reasons. Switchgear, transformer or other major equipment replacements or upgrades at one substation may make associated communications system upgrades worthwhile for that location. However, for other substations that did not need equipment upgrades, Engelbrecht said, “We may be supporting equipment that could be 30 or more years old, for example, legacy installations that still have 1200 baud modems running on old pilot wires in some cases.”
In addition, timing differences sometimes occur during long-term rollouts of OT solutions, where the associated IT platforms have changed midstream, creating another type of IT/OT convergence problem. As Engelbrecht pointed out, “A long-term upgrade with a wide scope could take years. Before the OT upgrade is completed, the associated IT platforms may be nearing obsolescence. Since the project is in progress, the platform lock-in can create an issue matching software requirements to hardware that is available.”
Greater Standardization
Moving toward greater standardization of communications platforms is leading to an associated reduction in IT/OT problems. National Grid’s Kelly reported on how things have changed from a grid perspective over the last few years, for example: “In the old days, you had grid equipment manufacturers trying to sell their various radio-based communications systems. And back then, this may have made sense, since the communications systems were mostly proprietary, were mostly radio based and point to point, and they were still very expensive. But now, the communications system costs have come down, and widespread networked communications platforms are becoming more ubiquitous.
“From an IT/OT perspective — through all of this — those of us on the utility side of it, as grid players, want to get the benefit of standards-based communications solutions allowing us to just connect equipment to a system with minimal overhead,” Kelly explained. “So we’ve had a desire for the grid equipment manufacturers to partner with the communications vendors to provide simple yet integrated communications capabilities with the bandwidth that we need.”
Success Key
Many people said a key to optimizing benefits and minimizing difficulties in the new IT/OT converged world is to keep the big picture in mind. For example, RG&E’s Engelbrecht said, “Every one of the new struggles brings new benefits. A key to success is making sure somebody is engineering the end state; if nobody is doing so, it can lead to a sense of constant chaos.”
Tucson Electric’s Taylor has seen the same issue in his work as CTO. “As the pace of integrating new technologies increases, utilities will explore new business models that will require new regulatory policies,” Taylor said. “Utilities are embracing technology that helps improve service for customers, but it will be some time before we see new regulatory models in response to these changes.”
Burbank Water and Power’s Fletcher suggests an update to an ideal model in circulation since 2008 from EPRI would be a good thing for utilities. “Having a sort of road map of an ideal architecture is really helpful because it provides us with a common way to look at our plans going forward,” Fletcher said. “While the actual model has been going through updates and changes, the ideal model has not been thoroughly updated, and it surely needs to be updated.”
Survey Review of IT/OT Integration and Cybersecurity
- Which of the following best describes your organization?
- What is your primary job function?
- What are your primary areas of responsibility?
- Which of the following IT and OT solutions does your utility currently employ?
- Generally speaking, how would you characterize the overall functionality of your utility’s IT business applications? Of your utility’s OT operational applications?
- Generally speaking, how would you characterize the sophistication of your utility’s cybersecurity solutions?
- How would you characterize your familiarity with IT/OT integration?
- Which of the following statements best reflects the current state of IT/OT integration at your utility?
- How important is IT/OT integration to the success of your utility?
- What are your key challenges with regard to IT/OT integration at your utility?
- Does your utility allow employees working offsite to access operational applications remotely?
- How is field data returned to your utility and how does your utility store it?
- Does your utility have a system in place to automate the process of meeting operational compliance requirements?
- Who is the primary resource leading your utility’s efforts with regard to NERC CIP compliance?
- Where did your utility stand with regard to meeting the NERC CIP v5 Compliance deadline of April 2016?
- What are your utility’s primary concerns with regard to cybersecurity? What keeps you up at night?
