In October 2014, Pacific Gas & Electric (PG&E) opened a US$28.5 million state-of-the-art electric distribution control center in Fresno, California, U.S. When two additional regional control centers open later this year, the utility will be leading the evolution in distribution control center technology and design. Together, the three regional centers will replace a network of 13 local control facilities and enhance the reliability and resiliency of PG&E’s more than 140,000-circuit-mile (225,308-circuit-km) electric distribution system.
What makes these new distribution control centers notable is PG&E developed them with a focus on being able to manage the next-generation grid, which incorporates the two-way flow of energy from renewables as well as traditional sources of energy. PG&E executives have coined this “the grid of things,” a platform to interconnect and enable all of the complex and emerging energy technologies customers are adopting. The utility’s new control center technology enables continuous innovation in the future while ensuring the safety, reliability and resiliency its customers depend on today.
An Early Test
While the Fresno control center is still in its infancy, it already has been put to the test and has started contributing to PG&E’s record responsiveness and reliability. In December, a severe storm hit much of California, ravaging the area of Bakersfield, in particular, in the southern part of PG&E’s service area. As power outages began to develop, the utility’s SmartMeter data poured into a new distribution management system (DMS), which quickly alerted PG&E operators at the Fresno control center to the magnitude and location of each outage.
Within minutes, PG&E was able to redirect the system operator focus into the Bakersfield area using new seamless technology at its operating consoles. This ability to shift jurisdictional focus rapidly allowed operators in Fresno to quickly and accurately direct field crews to outages and potential public safety hazards, expediting service restoration to PG&E customers and maximizing assistance to public service agencies. Despite strong storms like this one, PG&E customers experienced the fewest number of outages in the utility’s history in 2014. In addition, the utility has continued to reduce the average duration of outages, marking its sixth straight year of delivering record electric reliability to customers.
Key Features
The centerpiece of PG&E’s new electric distribution control centers is the updated and enhanced DMS, which facilitated the transition from reliance on paper to consolidated electronic methods. The new DMS incorporates advanced electronic mapping of the service territory with SmartMeter data from specific locations, helping operators pinpoint the exact location of an outage, many times before customers can report it. The maps featured in the DMS use the utility’s geospatial information system, which manages, analyzes and displays all forms of geographically referenced information. In addition, PG&E has plans to further integrate additional smart grid technology with its DMS in the coming years.
The new DMS is supported by improved operator consoles at the control centers, each equipped with a 50-inch (1,270-mm) 4k monitor, as well as an additional four-monitor array that allows for greater visualization of the grid and insight into real-time events affecting the grid. This improved visibility helps to ensure greater oversight and accuracy as operators monitor and operate the system. In addition, each pod of operating consoles has devoted video wall panels that allow for even greater situational awareness as well as collaboration on overall system conditions that might require analysis or action.
Situational Intelligence
The new distribution control centers will use a grid operations situational intelligence (GOSI) system. This advanced data collection and analysis system aggregates hundreds of thousands of data points from internal data sources such as supervisory control and data acquisition (SCADA), SmartMeter, asset and crew information. It also will take in external data sources such as fire, weather and earthquakes. GOSI finds correlations and relationships between those data sources, the distribution network model and geospatial reference points. The system then provides a visual display of contextual information about the events happening and the potential risks to assets operators can use to inform their decision making.
Resiliency
Fault location, isolation and service restoration (FLISR) systems represent the present and future of service reliability at PG&E. This remotely controlled equipment with automated self-healing switches is already installed on nearly 20% of electrical distribution circuits throughout northern and central California. In the event of an outage, these circuits automatically isolate the electrical fault and then restore power through alternative sources to the majority of impacted customers, often within minutes.
Since 2012, when this self-healing technology was first adopted, it has helped PG&E avoid more than 60 million customer outage minutes and saved more than 600,000 customers from a sustained outage (defined as longer than five minutes). The further penetration of this technology in the future will improve these statistics.
Disaster is the true test of system resiliency capabilities. Through seamless technology, all three new centers will have the flexibility to shift full operations to other new regional control centers if support is needed in the event of a major storm or natural disaster. In fact, the design calls for using all three new centers as one virtual center for routine work balancing once the project is completed.
Resiliency also is a feature and function at the facility level. Given the earthquake-prone nature of PG&E’s service territory, it was important the new centers be better able to survive and remain operational after a seismic event. With this in mind, PG&E designed and built the control center buildings not only to meet but exceed seismic standards. A primary feature of the buildings is cross-brace framing, which supports compression and tension forces prevalent in an earthquake.
In addition, at the central control center in Concord, PG&E is using a system in which server room equipment is mounted on seismic isolation devices that dissipate the energy of a seismic event, preventing damage to this critical equipment. The Concord facility also possesses a high reliability standard with dual circuit feeds, dual server rooms, dual network feeds, dual network cables, dual uninterruptible power supply backup systems as well as dual backup generation.
Communication Advances, Too
Cutting-edge Voice over Internet Protocol (VoIP) and Radio over Internet Protocol (RoIP) technology integrates all phone and radio communication lines into a single user interface, called a turret. This system allows for rapid reassignment of operator area responsibility that supports workload balancing, which is especially important during storm events. This further supports PG&E’s one virtual center design concept, critical to the utility’s consolidated control center network.
The new system provides a more stable environment, better connectivity, higher sound quality and more opportunities to better allocate workloads. In addition, PG&E has invested in new hardware to support the VoIP and RoIP system, which features a touch-screen interface, handheld wireless push-to-talk functionality and increased audio speaker functionality.
Collaboration
PG&E’s new electric distribution control centers are the product of years of collaborative research, development and testing with several dedicated partners. From the beginning, PG&E turned to its electrical distribution operators and leaders from the International Brotherhood of Electrical Workers (IBEW) to help design, develop and deploy every aspect of the consolidated control centers in full — from building layout to technology providers.
When the idea for the project was conceived in 2012, PG&E worked with IBEW to select five operators from the utility’s current electric distribution control centers to help drive the development of the new control centers. With their years of technical experience working in control centers and real-world understanding of how new technology and designs would be incorporated into fellow operators’ daily activities, these operators were critical partners.
In addition, PG&E collaborated with IBEW leadership, seeking the union’s institutional knowledge and expertise from across the utility industry. PG&E leadership and IBEW representatives met throughout the design and development process to discuss findings, brainstorm alternatives and determine the best fit for the utility and its operators going forward.
Todd Gracyk, an IBEW union member and 36-year operator with PG&E, was one of those five operators. “It was great because we had an open conduit to our union leadership as well as between the union and the company,” said Gracyk. “We were given every opportunity to voice our opinion and make sure that our ultimate designs were the best, both for the safety and reliability of the system, and for the everyday work of the operators.”
While the utility performed its own initial analysis of potential systems and designs, it also relied on the operators to audit and expand on its vision. The operators began their research by visiting another utility’s consolidated electric distribution control center, providing the opportunity to hold open discussions with operators about the functionality of the centers as well as obtain a hands-on feel for the technology. From these site visits, the operators were able to build on other utilities’ successes and avoid areas that had not proved ideal. The operators also met with vendors and suppliers to find the best partners to build the infrastructure and technology that they determined best fit overall project needs.
“We [the operators] were talking to vendors every day, setting priorities for features and enhancements we wanted to see in our systems,” Gracyk noted. “And, in addition, PG&E challenged us to come up with the best ways to test the technology as it was developed and train our other operator colleagues once we established the new systems.”
The New Way
When the first control center was deployed in Fresno in October, it introduced a whole new way of work for the operators. The center itself was significantly larger in size, employing more operators in one consolidated space than any of the legacy centers. With so many more operators located together, the teams were able to manage planning work better. In fact, as the operators acclimated to the new environment, they found certain operators naturally transitioned into different roles, broadening their skill sets and diversifying their responsibilities.
“In the legacy centers, there were five to six operators working together to do everything at once,” Gracyk said. “In the new center, some operators have become sort of foremen for our teams, steering us through the evolution of processes, and taking control of planning work and performing audits, while others are becoming experts in the traditional operator roles. The ecosystem of the centers is still developing, but so far, the operators are really taking to the work.”
The Fresno center’s deployment was not without challenges, but PG&E’s phased go-live approach allowed the operators to continue to innovate and adjust while ensuring safe and reliable service for customers. “There’s a learning curve for all of this. We all underwent substantial training prior to the go-live, but the nature of the work is that you can’t prepare for everything. The operators have done an excellent job adapting and learning as we go. I look forward to the next set of challenges we will face as we bring the Concord control center live later this year,” Gracyk commented.
Completion in 2015
The second of three centers, the central distribution control center in Concord, is set to open in July 2015. This center will house approximately 90 employees and primarily be responsible for monitoring service to the San Francisco Bay Area and Central Coast regions. The third center is scheduled to open in November. The three centers will work cooperatively to enhance system reliability for PG&E’s nearly 16 million customers throughout northern and central California.
Gary Cassilagio ([email protected]) is the director of PG&E’s business applications organization within distribution operations. He is responsible for distribution grid technology and the lead sponsor for the distribution control center consolidation project. Cassilagio has managed technology integration efforts involving outage management systems, distribution management systems, smart meter, smart grid, SCADA and mobile technology, which have supported PG&E’s overall strategic business goals.
