This was the moment of truth. On May 8, 2016, the Energy Control Center was about to issue the command to close Breaker 4 and energize the first 345-kV transformer at the East 13th Street substation. At 1:58 p.m. the command was sent, Breaker 4 closed, and the new integrated substation automation and protection system (SAPS) was officially on-line.
A few team members momentarily held their breath as the Breaker 4 command was issued. It was Con Edison’s first time installing and energizing the 345-kV Plug and Switch System (PASS) breaker. In addition, the technicians were also installing a new SAPS, which was completely based on an IEC 61850 fiber-optic architecture. After exhaustive testing and double and triple checks, the breaker operation was anticlimactic. The power came back on, and everything worked as expected.
Hardening a Critical Substation
Eighteen months earlier, Sanjay Bose, the central engineering vice president, had given his team the directive to upgrade and modernize the East 13th Street substation from its 1950's vintage relay protection system to a state-of-the-art IEC 61850 system. This was a radical departure from the standard design for systems of this type.
It was also going to be perhaps the largest brownfield implementation of its kind: an IEC 61850 protection and automation system in a high-voltage substation serving a critical and high-density metropolitan area. The East 13th Street substation provides power for 220,908 meter customers in Manhattan, which translates to almost 1 million people. Except for the World Trade Center load pocket, all customers south of about 36th Street depend on this substation for their power.
This was dramatically demonstrated during the onslaught and aftermath of Superstorm Sandy. A picture of Manhattan taken that night shows a dark outline south of 36th Street, where normally one would see a dense concentration of lit up buildings and streets. The East 13th Street substation, which is separated from the East River by only a narrow highway, had suffered the brunt of the storm’s fury. It was flooded by the storm surge and was forced off-line.
As a result of Sandy, Con Edison has launched a massive effort to harden the substation against future flooding events. The utility has raised perimeter wall heights around the substation and deployed movable gates, which can be rolled into position and tightened down to effectively seal the substation perimeter against water ingress. In some instances, Con Edison also elevated equipment above expected flood levels and installed an extensive high-capacity pumping system.
Embracing Challenges
Upgrading the relay protection and substation automation systems was an important component of this comprehensive storm-hardening program. The flooding had not only damaged substation equipment, but it had affected and compromised the copper wiring circuits used for relay protection. Con Edison focused on reducing the system’s vulnerability to future flooding and providing fast restoration for any equipment damaged. Additionally, Bose’s mandate was to find and implement the most advanced solution available.
Although adoption of IEC 61850 has been slow and tentative in the United States, it was evident from the start that this technology offers significant advantages, and it became clear that this was the direction Con Edison should take for this project.
As this was going to be the first system of this type for Con Edison, there were initial concerns that had to be addressed. Interoperability, for example, is touted in all IEC 61850 literature as its biggest selling point. Con Edison’s research on real-world implementations, however, uncovered examples that clearly demonstrated lingering interoperability issues from other utilities. Con Edison also had concerns about system performance in high-stress, high-activity states where the substation network was heavily loaded. The utility also had to develop a new methodology for system testing.
From a project management perspective, the overriding concern was the aggressive schedule imposed by operations. Con Edison needed an unusually long outage to perform all of the scheduled work and had to complete the outage before entering the summer peak load season. Compounding the challenge, the major new substation components — the PASS breakers — were on order to be delivered and had to be installed on schedule. Postponing the outage would have added months at the very least and as much as one year to the overall effort, which was unacceptable.
As a result, the team was committed to a schedule that required the system to be delivered, tested and ready to begin installation in less than 10 months. The contract for the project was awarded to ABB in February 2015. The system passed factory acceptance testing (FAT) in November 2015 and was then delivered to the substation for field installation and testing.
Another major concern was how the system would perform in high activity or stressed conditions. It is critical that the protection GOOSE message timing must not be affected or degraded by lower-priority network traffic. Preliminary calculations showed that network bandwidth utilization, for example, should not exceed 4% to 5% under any but the most extreme circumstances. However, it was essential to demonstrate actual performance during system integration testing at the FAT. For this purpose, the specifications included a requirement to perform a stress test at 30% bandwidth utilization. During the actual test, engineers ran the level of traffic all the way to 60% bandwidth utilization (way beyond any expected real-world conditions) and observed absolutely no impact on the timing of the relay-protection functions.
An added challenge for a brownfield installation is that the substation must provide uninterrupted service to all power customers in the area while undergoing this complete renovation. This greatly increases the complexity of the installation when compared to a greenfield project. For example, a utility must carefully plan the intermediate transitions. In addition, technicians must install interface equipment to provide the connection between the new automation and protection system and the legacy equipment remaining in adjacent sections of the system. Also, the interface equipment must be configured to accommodate adjacent section upgrades.
To minimize the risks of making drastic changes to the system each time a new section is upgraded, the vendor was required to deliver a system that was fully programmed for its ultimate configuration. Also, no modifications to the database, HMI graphics or any other core system functions were permitted once the first stage was commissioned and became operational. Similarly, when cutting in a new upgraded section, no configuration changes were allowed to any IEDs or protection devices installed in previous stages.
Taking a Proactive Approach
Con Edison devised a multi-pronged plan to mitigate these technical and logistical risks. First, the utility decided that even though the project would be awarded to the winning bidder as a turnkey system, Con Edison would be intimately involved from the beginning in all project phases. It would be a cooperative effort with embedded teams from Con Edison working at ABB’s main facilities at various stages, as well as ABB teams working at Con Edison. That way, Con Edison could quickly identify and address design issues and decisions as they arose. Also, the Con Edison team would have a transparent window into every system detail to gain highly detailed knowledge of the system. This provided many advantages, not the least of which was the very efficient level of communications between Con Edison and ABB.
Second, Con Edison required the vendor or integrator to set up an interoperability demonstration early in the project cycle. After procuring the protection relay IEDs and network switches used in the design, a representative number of these devices were connected in ABB’s lab environment and made to communicate with each other as the Con Edison team observed and confirmed reliable interoperation.
Third, Con Edison adopted a prototyping approach. Many of the system components were of identical or similar design. The vendor would build one or two of each of these components, such as outdoor cabinets with protection relays and network switches, and Con Edison’s team would inspect each prototype closely before dozens of them were built.
Testing the System
All of these measures provided a reassuring level of confidence in the design. The biggest perceived risk was that if Con Edison treated this like previous projects, major design or performance issues might be uncovered during the testing phase. This risk was exacerbated by the project schedule pressures. Should this happen, Con Edison would not have time to properly address and resolve those issues. Because of the early and close involvement and collaboration with the vendor, this risk was greatly reduced.
Perhaps more than in any other previous project, the success of the FAT became the central point of focus. The success of the project hinged heavily on how the system would perform when connected together for the first time.
To optimize the schedule and take maximum advantage of the opportunity to work with the actual system during the FAT, Con Edison decided to have the equipment shipped and staged at a warehouse within commuting distance of Manhattan. This allowed Con Edison’s protection system technicians (PSTs) to travel to the FAT warehouse and participate in setting up and running the tests. This proved an invaluable way to provide hands-on training with the system as well as with the new testing methodologies developed specifically for this system. Later, this PST team would be responsible for testing and commissioning the system on site. Through their active participation in the FAT, they gained confidence in the system and in their newly acquired hands-on skills for working with IEC 61850.
Achieving Success
Following many months of tireless work by a dedicated team, Con Edison has now built a completely new control room and installed dozens of equipment cabinets throughout the substation and miles of blown fiber-optic cable. In addition, the utility has tested and retested all components and successfully commissioned the first phase of the installation. Over the next two years, Con Edison plans to complete the three remaining installation and cutover phases.
Today, Con Edison is now the owner and operator of an IEC 61850 SAPS running a critical high-voltage substation. This system provides a new level of storm-hardened ruggedness and reliability to better serve Con Edison’s energy customers. The system has been working flawlessly since the Breaker 4 closing operation on May 8 when it was first officially brought online.
This project required a closer than usual level of cooperation between Con Edison and the vendor. A successful project is highly dependent on the customer and the vendor working as a team and sharing the same goal. Open and honest communications proved to be the key to successful collaboration and a successful project outcome.
Manny Pimenta is a senior engineer for Consolidated Edison. He joined the company in 2012 and has more than 20 years of experience with SCADA systems. He also worked for eight years in the defense industry as a test systems engineer and engineering team leader. He was part of the EPRI IEC 61850 test bed project and participated in EPRI’s Transmission and Substation Area Task Force.
Six Key Features of Con Edison’s Newly Modernized System
1. Two fully separate and independent lines of relay protection. In many previous installations, Con Edison used IEC 61850 in only one of the two lines of protection. For the 13th Street substation, however, the technicians installed two separate and independent LANs, two sets of protection relays, and separate and independent PTs and CTs.
2. Station bus implementation. After considering process bus implementation, Con Edison decided to install the IEC 61850-based substation automation and protection at the station bus level only. In Con Edison’s view, current vendor solutions are relatively recent entries into the process bus market and lack a sufficiently solid record of multiple years of in-service reliability. In the future, Con Edison plans to install and test merging units at strategic points in the substation. Assuming the pilot installations are successful, it will gradually build out the process bus implementation.
3. Unique mechanical lifting mechanisms. These allow some of the protection relay panels to be raised above flood levels in anticipation of an event while remaining fully operational.
4. Data diodes. By preventing remote access to the substation network, the data diodes provide foolproof cybersecurity. They allow for one-way data traffic from the substation to the enterprise LAN.
5. Extensive use of hot swappable solid-state disk drives in RAID configurations. These enhance reliability and rapid restoration.
6. Advanced disturbance monitoring system. This system automatically collects and organizes event files for forensic analysis of fault events.
