THE CONCEPT OF MODULAR, PROTECTION, AUTOMATION AND CONTROL (MPAC) BUILDINGS is to provide highly integrated, well-engineered, preassembled control buildings to a site either for retrofitting existing substations or for new substations. A substation automation system is contained within an MPAC building to collect information and provide local control, monitoring and communication to the switching centers.

During the past four years, Pacific Gas and Electric (PG&E; San Francisco, California, U.S.) has completed 10 pilot projects, including the installation of several MPAC buildings. Based on the lessons learned, PG&E is moving forward to further refine the design of the MPAC building projects. In addition, the utility has reviewed in detail the data collection and communication to the switching centers to define the critical data to be shared. In fact, PG&E simplified the system automation architecture within the MPAC building, reducing database complexity, providing faster data updates and streamlining troubleshooting and testing.

Working in partnership with two selected vendors during the initial phase of the project, PG&E was able to clarify various requirements and necessary standards for automation refinements, and is currently using these requirements and standards to refine future MPAC project designs. PG&E is also ramping up the number of installed buildings each year to reach a total of 180 buildings over the next 10 years.

BEGINNINGS

In 1997, PG&E launched a program to design an integrated substation protection, control and monitoring system that could be used for both retrofits and new transmission substations. The design goals were to:

• Conform to open-system architectural and communications standards

• Incorporate the integration of devices and applications

• Provide accurate, timely, reliable and secure high-speed communication between the substation data/information and corporate end users or information systems

• Establish a migration path for the future and easily support device upgrades

• Enable support of outage reporting and response with intelligent feedback using automated applications to eliminate human error

• Support equipment diagnostic applications for maintenance programs.

The initial design was based on a protection scheme used in distribution substations that incorporated two intelligent electronic device (IED) relays. From a protection standpoint, the IED relays provide primary and backup functions. From an automation standpoint, each IED has specific assigned functions. For example, the front panel of one is used as the local manual-control interface to the circuit breaker. The Set A IED has 12 user-programmable pushbuttons programmed to provide functional replacement of traditional control switches for circuit-breaker control, automatic reclosing, relay cutout for Set A and Set B protection functions, and SCADA cutout function through a local/remote switch. The Set A IED also includes 48 user-programmable light-emitting diodes (LEDs) that are driven by standardized relay logic to provide feature status of relay cutout, automatic-reclosing features and equipment-diagnostic alarms. Text used on the pushbuttons and LED labels was also standardized.

The Set B IED relays provide the power-system data for monitoring and support remote control from either the switching center or the substation. In the initial design, an intelligent remote thermal unit (RTU) was used as the interface to the IED relays. The RTU handled communications to the switching center and a substation computer, which provided graphical human machine interface (HMI) in the substation.

EARLY MPACS

In 1998, PG&E worked with a system integrator to retrofit a pilot system into an existing control building. This pilot demonstrated that the concept was sound and satisfied the majority of the objectives. However, it also demonstrated that the effort and cost associated with retrofitting were unacceptable. To eliminate this problem, PG&E worked with the integrator to design, purchase and install a factory-assembled control building that provided all protection, control and monitoring for the substation addition. This addition consisted of a new 230-kV/70-kV transformer with three 230-kV circuit breakers and two 70-kV circuit breakers. While successful, this installation identified the need for upfront training and involvement of all field personnel. Two additional pilots were installed in 2000. Evaluations proved the viability of MPACs and the next phase of the program was launched.

From 2001 to 2003, PG&E firmed up design and supporting standards for the MPAC buildings and systems. The intelligent RTU was replaced with a data concentrator that could survive a rugged substation environment with an embedded processor and no hard disk. The data concentrator also provided enhanced communications capabilities and a secure network interface for relay engineers and others.

The substation HMI software was converted from the third-party supplier to the software being used at the switching centers. This made displays consistent and eliminated the need to develop and maintain two different databases, because everything could be loaded directly into the switching center systems. The substation HMI would poll the data concentrator using DNP3 protocol and would communicate to the switching center using a TCP/IP protocol with three levels of security.

Five MPAC building standards were developed to meet public road restrictions for transportation and different applications (see table). Typically, there would be several modular units at a larger station, each covering a voltage class. The buildings, each interconnected to the others, would be located near the controlled bus section and would have complete control and monitoring of the entire station.

LESSONS LEARNED

During 2003 and 2004, several MPAC buildings were built, shipped and installed in PG&E substations. The easiest lesson learned is that MPAC buildings should be configured and tested at the factory to ensure proper operation of the equipment and systems within the building. Additionally, the relay-setting programming should be standardized for each application to provide consistency from station to station.

PG&E also learned that the data concentrator was the source of several problems. Database programming was time consuming and required a high level of expertise to configure and maintain. Operating reliability was poor, and random power-downs and restarts subjected the switching centers to numerous alarms upon reboot.

Factory-acceptance testing was implemented in early 2005, as was the relay configuration standardization and the factory configuration of equipment. A system architectural redesign was proposed to have the substation HMI server communicate directly with the relays. This alternative was available as a result of two changes to the system architecture. First, the supervisory control and data acquisition (SCADA) program used for the HMI and at the switching station was revised to provide the capability of DNP master protocol. This change enabled the SCADA software to poll the IED devices directly. The second change was the development of a secure, isolated wide area network (WAN) with dedicated routers. The WAN is used solely for SCADA and operational data, and the routers were selected to provide security and user authentication.

These security features eliminated the need to provide access and security functions within the data concentrator. The SCADA system direct-IED poll proposal was successfully tested in the engineering laboratory. A second full-scale test was run on a large MPAC building that was located in a PG&E substation but was not connected to the power-grid equipment. This test was also successful, so future designs will no longer have the data concentrator. The new architecture reduced the site acceptance time from three weeks to less than one week.

PG&E is now comfortable with the design and the performance of the MPAC buildings and sees little need for additional changes in the near term. PG&E is working with its strategic suppliers for systemwide deployment at the rate of 10 or more per year. The last five years have been full of investigations, learning, frustrations and successes. The latter will be remembered the longest.

---

Bernard S. Tatera Jr. received a BSEE degree in 1986 from California Polytechnic State University and a MSEE degree from Santa Clara University in 1995. He started his career with PG&E in 1986 and has held various positions including protection engineer, senior protection engineer, supervising protection engineer and senior automation engineer. From 1993 to 1997, Tatera worked in the Sacramento Municipal Utility District's Energy Operations department in the area of system protection and control. Since his return to PG&E in 1997, Tatera's assignments have included the testing and programming of substation automation projects, and he now works as a consulting engineer in PG&E's System Automation group. He is a registered professional engineer in the state of California and a member of IEEE Power Engineering Society. bst1@pge.com

Lee Smith received a BSEE degree from California Polytechnic State University and a MSEE degree from the University of Pittsburgh. For the last 25 years, Smith has been involved with executive management of automation companies. He is a registered professional engineer in the state of Pennsylvania, a member of IEEE Power Engineering Society (PES) and a participant of IEEE PES Substation Committee — Data Acquisition, Processing and Control Systems Subcommittee. He also currently serves as the president of the DNP users group, and is a former member of IEC T57 Technical Committee — Working Group 10. Lee.Smith@dcsystems.com

ORIGINAL MPAC ARCHITECTURE

The alarm avalanches resulting from the data concentrator resulted in a companywide operational assessment of what data was available from the MPAC substations and what data should be collected and displayed at the substation and at the switching center. The first decision was that both sites should have the same information and displays so that the system operator and substation technician could work from the same picture.

When looking at a large MPAC building, the amount of data available is 35,000 to 40,000 data points, but not all of this information is important to the system operators. The companywide study examined the data points available from each relay model and identified those that should be communicated to the substation HMI and switching center. This information has been used to define “object models” for each relay model for each application (feeder and transformer). These object models are then used to define the databases for the substation and the switching center.

Common displays were also defined during the companywide review; one of these is a relay front-panel display. This display permits the system operator to see the same picture as the technician at the substation. The study assessment was beneficial to all parties and should be performed early in the design phase instead of down the road, but then sometimes people do not buy in until they have a problem to solve.

All of the timed-based automatic-reclosing functions are programmed into the relay logic. Other functions are a combination of both devices. For example, the HMI has a sychroscope display showing the voltage vectors from each side of a circuit breaker. When the vectors are in the same quadrant and approaching each other, the operator can click on the close command and the relay synchronizer function will close the circuit breaker when appropriate. Additional automation functions will probably continue to be defined and developed in the future.

MPAC Building Design Standards Developed to Meet Public Road Restrictions for Transportation

Properties	Small with Battery	Small without Battery	Large with Battery	Large without Battery	Double-Large Long
Use	Double-tapped distribution or transmission	Double-tapped distribution or transmission	Transmission bus section	Transmission	Transmission
Breaker Positions	Up to 10	Up to 22	Up to 11	Up to 26	Up to 56
Number of Racks	12 racks (10 breaker/relay, 2 Comm/HMI)	24 racks(22 line terminal,2 Comm/HMI)	16 racks(9 line terminal, 1 bus tie, 2 bus differential, 2 bus sectionalizing, 2 Comm/HMI)	30 racks (23 line terminal, 1 bus tie, 2 bus differential, 2 bus sectionalizing,2 Comm/HMI)	60 racks (53 line terminal, 1 bus tie, 2 bus differential, 2 bus sectionalizing,2 Comm/HMI)
Rack Bay Length (2)	14 ft 6 inches	26 ft 10 inches	18 ft	35 ft 6 inches	71 ft
Interior Control Room Size	14 ft 6 inches wide by 17 ft 5 inches long	14 ft 6 inches wide by 39 ft long	14 ft 6 inches wide by 31 ft long	14 ft 6 inches wide by 48 ft long	14 ft 6 inches wide by 96 ft long
Battery System
Average System	Site specific	NA	Site specific	NA	NA
Interior Battery Room Size	14 ft 6 inches wide by 12 ft long	NA	14 ft 6 inches wide by 17 ft 3 inches long	NA	NA
Total Building Size	15 ft 4 inches wide by 40 ft long	15 ft 4 inches wide by 40 ft long	15 ft 4 inches wide by 49 ft long	15 ft 4 inches wide by 49 ft long	15 ft 4 inches wide by 98 ft long
Typically there are several modular units at a larger station, each covering a voltage class. The buildings are located near the controlled bus section. Each building has complete control and monitoring of the entire station. Each modular unit is interconnected to the others. Ideally, PG&E sites the building, so the HVAC unit will not be on the southern or eastern exposures, to alleviate direct exposure to sunlight, although the units are designed for direct exposure.

PG&E's “Substation 2020” Program

1997	Electric Transmission Maintenance department launches the Substation 2020 effort to focus on defining, testing and implementing technological advances in transmission substation automation.
1998	First project under Substation 2020 program is completed in California's Northern Sacramento Valley. Adding remote control and monitoring to an existing control room would prove beneficial to operators. A computer with HMI and RTU with peripheral units is added to an existing control building. This effort refocused PG&E's direction toward complete buildings as a more economical solution.
1999	Second project under Substation 2020 program is completed in the Central Coast area. A new control building is built rather than a retrofit. The new transmission control building is separate from the existing distribution-level control building. It provides protection, control and monitoring for a new 230-kV/70-kV transmission transformer with three 230-kV circuit breakers and two 70-kV circuit breakers. Lessons learned from this pilot identified the need for upfront training and field involvement.
2000	Pilot projects at Cortina and Templeton are evaluated and identified to provide automation benefits and the potential for overall project cost reductions.
2001-2003	Working with strategic partners, PG&E developed standards to establish modular protection, automation and control (MPAC) as an option for future control-room replacements.
2003	Two vendors are selected to provide MPAC buildings for current and future projects.
2004	Three MPAC buildings are delivered to PG&E substations, tested and put into service.
2005	Seven additional MPAC buildings are delivered, tested and in service. New system automation architecture is developed and tested.