Assessing DER network cybersecurity defences in a power-communication co-simulation environment

: Increasing penetrations of interoperable distributed energy resources (DER) in the electric power system are expanding the power system attack surface. Maloperation or malicious control of DER equipment can now cause substantial disturbances to grid operations. Fortunately, many options exist to defend and limit adversary impact on these newly-created DER communication networks, which typically traverse the public internet. However, implementing these security features will increase communication latency, thereby adversely impacting real-time DER grid support service effectiveness. In this work, a collection of software tools called SCEPTRE was used to create a co-simulation environment where SunSpec-compliant photovoltaic inverters were deployed as virtual machines and interconnected to simulated communication network equipment. Network segmentation, encryption, and moving target defence security features were deployed on the control network to evaluate their influence on cybersecurity metrics and power system performance. The results indicated that adding these security features did not impact DER-based grid control systems but improved the cybersecurity posture of the network when implemented appropriately.


Introduction
There is ample evidence from the last decade that many power system networks in the US [1][2][3][4][5][6] and abroad [7] are the target of active cybersecurity reconnaissance and attacks. The most widely discussed attacks are those that caused widespread blackouts in Ukraine in 2015 and 2016 [8,9], but there have been several other disconcerting trends including the increase in operation technology (OT)-focused malware, e.g. Crash Override and Black Energy [10,11], deep reconnaissance into power system networks [12][13][14], and growing willingness to deploy powerful cyber weapons that are affecting critical infrastructure [8,9,15]. Attackers often use myriad techniques to gain footholds in information technology (IT) networks and then pivot to other computers, servers, and networks to exfiltrate sensitive information, monitor operations, or plan for sophisticated attacks [16].
At the same time, penetrations of distributed energy resources (DER) -e.g. photovoltaic (PV) and energy storage systems -in the electric power system continue to grow rapidly on distribution and sub-transmission systems [17,18]. Over the last decade, an increasing number of inverter vendors and aggregators have provided monitoring portals for their customers. Like many other internet of things (IoT) devices, modern DER provide this monitoring or control functionality via proprietary communication protocols. However, these IoT devices now control a substantial portion of the total power production in certain jurisdictions, such as Hawaii and California [19,20].
In 2018, a revision to the US interconnection and interoperability standard, IEEE Std. 1547, required DER equipment to have either an IEEE 2030.5, IEEE 1815 (DNP3), or SunSpec Modbus communication interface [21]. New California Public Utility Commission Electric Rule 21 regulations that went into effect in early 2019 define IEEE 2030.5 [22] as the default application protocol for investor-owned utility communications to DER [23,24]. The adoption of standardised communication protocols is a critical step towards interoperability between power system operators and DER equipment, but a comprehensive national approach to DER cybersecurity is absent.
There are many security requirements for operators of critical infrastructure in the US. Power system operators are required to adhere to the North American Electric Reliability Corporation (NERC) critical infrastructure protection standards which coveramong other things -training, security and information management, perimeter defences, and incident reporting [25]. NERC requirements are reserved for bulk power equipment operating at or above 100 kV, so DER equipment and associated networks are exempt from these requirements. The solar industry and national government understand this gap in power system security and are working to address the requirements by reviewing and updating security requirements in the DER communication protocols [26,27], standing up DER cybersecurity working groups [28], and seeking new security standards for DER devices and networks [29].
There is extensive research that may improve the national DER cybersecurity posture [30]. Generally, utilities principally rely on perimeter defences (e.g. firewall rules) to defend their IT and OT systems and there is little emphasis placed on the holistic network design. In this work, three additional network defence techniques were analysed with respect to power system performance and security trade-offs; network segmentation, encryption, and moving target defence (MTD) were deployed in a virtualised environment to (i) calculate the additional communication latencies associated with these features, (ii) determine the impact these would have for distribution-and transmission-level grid services (e.g. voltage regulation, frequency reserves, protection etc.), and (iii) evaluate any security improvements in the broad areas of confidentiality, integrity, and availability by conducting adversary-based (red team) assessments. This work produced power system performance and cybersecurity metrics to advise the solar and power system industry on the best cybersecurity practices for DER networks. The primary contributions of this work are (i) designing and operating the first PV communications network in a cyber-physical cosimulation environment with real network packets passed between virtualised DER equipment and a DER management system (DERMS), (ii) evaluating network latency for several DER network defence strategies, and (iii) quantifying cybersecurity metrics for defensive strategies with live human-in-the-loop red team assessments.
In the remainder of the paper, Section 2 introduces the cosimulation environment and associated emulation components. Section 3 discusses the additional latency in DER communication networks when applying networking defence technologies. Section 4 covers the red team assessment methodology, limitations, and results for each DER network design. Section 5 provides conclusions on the cyber-physical studies and recommendations for future research.
2 Co-simulation environment SCEPTRE (capitalised, but not an acronym) is a live, virtualised power system and control network co-simulation platform developed at Sandia National Laboratories (Sandia) capable of investigating the trade-offs between power system performance and cyber resilience [31]. SCEPTRE provides a comprehensive industrial control system (ICS) and/or supervisory control and data acquisition modelling and simulation hardware-in-the-loop (HIL) capability that captures the cyber-physical impacts of controls system operations and targeted cyber events. Changes in the network are reflected in the power simulation, and changes in the power simulation are reflected in the communication system, thereby allowing researchers to analyse the complex interactions in a cyber-physical environment. A simplified representation of the co-simulation environment is shown in Fig. 1.

Virtual machines (VMs)
The virtual components in a SCEPTRE model are created and run as VMs using Minimega, an Emulytics™ (Emulation + Analytics) tool that was developed at Sandia for orchestrating distributed VMs and producing host and network emulations. SCEPTRE leverages Minimega's hypervisor capabilities to deploy VMs on compute nodes [32]. A virtual representation of PV inverters was created as SunSpec Modbus remote terminal units (RTUs) using SunSpec models 1, 101, 123, and 126 -containing common, inverter (single phase), immediate controls, and static volt-VAR data (see [33]). These virtual RTUs interface with PV systems represented in the power simulation using ZeroMQ as discussed in Section 2.3.
Along with the PV inverters, networking equipment was also created using emulated switches and routers. A utility advanced distribution management system (ADMS) system was created using a Windows 7 VM running a SunSpec system validation platform (SVP) [34] executable to provide control and monitoring of the DER systems inside the environment.

Network environments
The communication architectures were created in SCEPTRE to include components of a utility-to-DER network. Within the utility subnet, an ADMS -implemented using the SunSpec SVPconducted the volt-VAR shift control algorithm from [35,36] by sending SunSpec Modbus packets through the emulated network. Measurements from the power system were pulled by the 20 DER RTUs and volt-VAR control settings were issued -as requiredonce per second. For these experiments, the following environments were created: (i) A flat network with and without secure shell (SSH) encryption. (ii) A network segmented into three random enclaves with and without SSH encryption between the ADMS and the enclaves. A HIL inverter was added to the network without SSH encryption. (iii) A MTD network without encryption.
These architectures were adapted from the work done in [37] for cybersecurity network architectures in microgrids, but in this case, reflect a DER ADMS control system. An example of flat unencrypted topology is shown in Fig. 1. Appendix A includes the flat encrypted and segmented encrypted topologies. The other topologies are presented in [38].
Internet traffic was generated to simulate normal conditions where other entities are connecting to internet resources. Simulated packets were produced using Protonuke clients and serversstandalone Minimega tools for internet protocol (IP) traffic generation -which support HTTP, HTTPS, SSH, and SMTP communications between VMs within the emulated environment.

Power simulations
SCEPTRE interfaces with and runs several different power simulation programmes (e.g. pypower, PowerWorld, and OpenDSS) depending on the use case. These simulations were coupled to the simulated control network to demonstrate the performance of DER grid-support control functions under different cybersecurity architectures, protocols, and additional security features. For these experiments, the distribution model presented in [35] was used but each of the 750 kW PV sites was assumed to be constructed with ten 75 kW PV inverters. When the DER settings were updated in the RTUs, an internal backend ZeroMQ (or ØMQ) [39] network transferred the new settings to the DER devices in the OpenDSS distribution circuit simulation. Similarly, the status of the power system at the location of the DER was transferred to the RTUs using ØMQ when there was a power simulation update.

Communication latency
When cybersecurity features are added to control networks, there is an increase in communication latency from processing data, additional router/switch hops, firewall rules, exchanging keys, binding certificates, performing encryption, or reconfiguring the system. These operations have the risk of adversely affecting realtime grid operations if the delays are significant. Several experiments were conducted to determine the communication latency associated with adding security features to DER networks.

DER latency impact on power system operations
As noted in Department of Energy's 2017 report on the Modern distribution grid: volume III, the communication timing requirements for DER are on the order of seconds, with typical bandwidth and latency requirements of 10 kbps and 5 s, respectively [40]. These communications requirements represent generalised limits on tolerable latencies between the utility and smart inverters. Prior work on transmission-level and distributionlevel DER control algorithms provided a more detailed view of the relationship between communication latency and performance. It was found the hierarchical volt-VAR shift algorithm was effective with latencies up to 20 s [41] for distribution circuits, whereas the transmission services were severely impacted with lower latencies. Synthetic inertia experienced a loss of machine synchronism defined by rotor angle separation with latencies between 200 and 400 ms (depending on the gain) [42]; communications-enabled fast-acting imbalance reserve was ineffective if the delay is longer than the time to the frequency nadir (e.g. ∼1-10 s depending on system inertia) [43]; and communications-enabled DER droop control experienced oscillations with latencies of 110-400 ms (depending on the gain) [44]. These findings all indicate the control algorithm will lose effectiveness with increasing latency, leading to a range of potential problems. Therefore, the selection of cybersecurity defences must not substantially extend communication times.

Communication latency studies
Communication latency is a combination of the encryption time, a number of network hops, communication media, and device read/ write times. While small improvements in communication time can be made with optimisation of encryption algorithms, new router technologies, and faster memory read/write times, these are likely to be minor; DER generate low-priority internet communications with cost-competitive communication interface boards. Like many other IoT technologies, there is little incentive for DER vendors to invest in performance improvements. In this section, DER latency is dissected to show the addition of security features only increases the latency by a few percents. Therefore, the addition of security features must not substantially impact grid services and, wherever possible, should be added to DER communication networks to improve the cybersecurity posture of the power system. The following sections investigate latency from non-security factors (geographical separation, physical media, and device read/write times) and for security features (encryption, segmentation, and MTD).

Geographical separation and physical media:
Phasor measurement unit (PMU) messages between Albuquerque, NM and several geographically distributed locations within the continental United States were used to understand latency impacts of distance and communication media. PMU transit times to Albuquerque were calculated using the global positioning system (GPS) timestamp and GPS time at the receiver. The results for communication transit times from sites in Las Cruces, NM (310 km), Pullman, WA (1570 km), and Lubbock, TX (460 km) are shown in Table 1. The connection to Texas was over a dedicated fibre line with minimal network hops, which minimised average communication time. While fibre and copper communications are both extremely fast, fibre has less signal loss, allowing for much longer runs and fewer hops [45]. Conversely, the network routes from PMUs in NM and WA had more routers and switches in the path which slowed transfer times. In general, these results show the architecture (switch and router hops) and communication medium (copper versus fibre) impact data-in-flight times more than geographic separation [46].

3.2.2
Device read/write times: 1000 Modbus read and write times were collected for two commercially available residentialscale DER devices and one controller HIL (CHIL) device [47] at the Distributed Energy Technologies Laboratory at Sandia National Laboratories. The SunSpec SVP was used to calculate mean, μ, and standard deviation, σ, for read and write operations. The results are shown in Table 2. Inverter 1 had a large standard deviation for both read and write times. It is not clear if there were internal communication checks or other inverter processes that slowed the responses. Like inverter 1, inverter 2 -the CHIL device -had a direct Modbus/transmission control protocol (TCP) connection over one network hop but responded much faster to both read and write requests. The connection to inverter 3 included an ethernetto-serial converter in the path to translate Modbus/TCP to serial Modbus. This added an additional delay due to the conversion processing -possibly accounting for some of the larger average communication times for reads and writes. It is believed the  variations observed in these results are because the inverters used different protocol stacks, processor hardware, and scheduling techniques for I/O tasks.

Encryption latency:
As per the IEEE 2030.5 and IEEE 1815 requirements, public-key infrastructure will be used to encrypt data between the utility and aggregators or DER devices. To authenticate entities, digital certificates -defined by the X.509 standard and registered to a certificate authority -are bound to the asymmetric key of the entities. Authenticated endpoints undergo key exchange to settle on a mutual symmetric key for bulk encryption.
The time to perform the encryption in the devices is highly dependent on the hardware. It is important to note both the value and limitations of the latency results obtained from an emulated system [48], because the absolute latency values are not representative of hardware implemented in the field. However, the relative impacts of applying additional security mechanisms are illustrative and help to provide scale, i.e. calculation speed depends on the resource hardware, processor speed, and available memory. IEEE 2030.5 specifies a cipher suite that uses transport layer security (TLS) 1.2, elliptic-curve Diffie-Hellman ephemeral key exchange, elliptic-curve digital signature algorithm signature authentication, and advanced encryption standard-128 (AES-128) bulk encryption. Specifically, the AES-128 CCM8 algorithm operates with 128-bit keys and data blocks in counter mode with cipher block chaining message authentication code (MAC)providing simultaneous encryption and authentication known as authenticated encryption with associated data and producing an authentication tag of 8 bytes [49].
In the co-simulation environment, the authentication and key exchange process were handled by SSH bump-in-the-wire (BITW) devices. Owing to the limitations of the co-simulation environment, tests were conducted using RSA for message signing/ authentication and AES with CTR/GCM modes for symmetric encryption. Extensive cryptographic benchmarking has been conducted with Crypto++, including throughput and key setup times [50]. The symmetric encryption ciphers employed by SunSpec RTUs are listed in Table 3 along with the mean round trip times (RTTs) experimentally observed in SCEPTRE for each cryptographic algorithm with TLS transport security. Cipherspecific RTT results are shown in Fig. 2. The encryption process increases the RTT by 1.67-2.05 ms over the unencrypted transfer. While this represents up to an 85% increase in latency, is it only a small increase in the total time to communicate with DER devices. As shown in Table 3 and Fig. 2, increasing AES key lengths decrease the mebibyte/s throughput because there are more cryptographic processing rounds. AES/GCM and ChaCha20-Poly1305 are authenticated encryption mode ciphers, so the algorithms perform more work than CTR mode but GCM is efficient and parallelisable so it has similar RTTs to the CTR modes. ChaCha20-Poly1305 is an efficient stream cipher that produces the quickest RTT but has the lowest bandwidth in the cited benchmarks.

Network topology:
A SCEPTRE experiment was created to calculate the increased latency associated with adding network segmentation. The main difference in the topologies was the addition of an extra hop required to break the DER control network into multiple segments. A round trip time (RTT) for the segmented DER network and the flat topology was calculated by pinging the DER from the utility Windows VM. The results for more than 10,000 individual measurements showed the average RRT for the flat network to be 1.56 ms, but the segmented network was 1.82 ms on average [46].

Moving target defence (MTD):
MTD is a class of technologies that dynamically modify a system environment to create uncertainty for adversaries by overlaying another control network on the publicly addressable one. MTD leverages softwaredefined networking (SDN) to randomise network parameters (IP addresses and ports) and communication paths. It is possible to randomise IP addresses and port numbers at fixed intervals or in response to detected network activity -i.e. dynamic defence. Randomising IP addresses at a configurable frequency supports evading adversarial discovery. This is meant to thwart the ability of an adversary to conduct reconnaissance and establish communications between devices on the network [51]; MTD has been proven to be effective at increasing the resilience of grid-wide area networks against certain types of attacks [52].
An example of this technology is shown in Fig. 3. On the left is a utility subnet consisting of an ADMS, geographical information system, and DERMS. On the right, is a collection of DER in a campus or utility/commercial site on a single switch. There is an 'IP generator' computer in the bottom that sends the new IP addresses to the switches in front of actual DER or computation devices. The MTD changes the IP addresses of these switches but the utility-owned and DER nodes retain static IP addresses. The actual implementation would likely require multiple MTD subsystems that independently reconfigure the IP addresses of the utility subnet and DER devices. Since this technology requires a seperate network to be overlaid on the publicly-addressable one, it is likely that DER would require a cellular modem or other out-ofband communication technology to be included in the MTD/SDN overlay.
In prior work, the communication latencies for various MTD modes were determined for different randomisation time periods; it was found that MTD increased the average latency by less than 1 ms but caused slightly higher dropout rates (∼1 dropout per 33.3 s with IP randomisation every 3 s) [51]. Other approaches to MTD, such as path randomisation, may increase latency more. A 11.73 ms increase in RTTs for path randomisation was reported by Chavez et al. [53].

Latency observations
Based on the results for network segmentation, encryption, MTD, geographical separation, and DER read/write times, some observations can be made about the impact of the control system when adding security features. In general, large geographic distances have the possibility of adding 50-100 ms of latency for utility-to-DER communications due to the additional networking equipment (routers and switches) between endpoints. DER read and write times vary widely; they can be 1 s or larger in some situations. In contrast, network segmentation adds <1 ms, encryption adds on the order of 3-5 ms of additional latency and MTD adds 1 ms. Therefore, for the proposed cybersecurity features, it is not believed that they will impact the grid-support service performance since they only contribute a minor percentage of the total latency between the utility and DER.

Red team assessments
Red teaming is defined as an authorised, adversary-based assessment conducted to strengthen defences through awareness of the potential device or system vulnerabilities. The assessment combined practices from multiple sources: Sandia's Information Design Assurance Red Team, NIST's guide to ICS security guidelines, best cyber security practices, and collective expertise regarding the DER devices and network. These guides informed the methodology that was used for the assessment of the network environments. The rules of engagement were limited to the SCEPTRE experiment network and the HIL device. For each of the environments, the red team assessment focused on identifying and compromising the PV inverters by turning them off, as well as disrupting network communications and modifying grid-supported functions (e.g. freq-watt, volt-Var, and power-factor).
The two scenarios investigated on the segmented networks for this assessment were (i) Public network attacker (outsider) access: This is an intruder who does not have access to the DER device but does have access to one of the internet service provider (ISP) routers. This also implies that the intruder is on the perimeter network. (ii) Local attacker (insider) access: The intruder is on the DER home area network with a foothold on any of the network enclaves or subnets.
The adversary is a computer running either a Kali Linux 64-Bit OS or a Window 32-Bit OS with third-party tools. The Kali Linux VM is shown in Fig. 1. The network mapper Nmap and OpenVAS vulnerability scanning tools were used to map the network, provided IP identification, detect open ports, host fingerprinting, and discover vulnerabilities on the devices in the network. Packet sniffers -Tcpdump and Wireshark -were used to capture packets and interpret the traffic. SunSpec dashboard application and Simply Modbus monitoring software were used to craft specific protocol traffic to the target devices and replayed using Netcat or Python scripts. To eavesdrop on traffic, the network security tool Ettercap was used. Scripts to modify and drop traffic was written using its filter compiler, etterfilter. Hping3 and Flood_router6 were used to deny services/resources to legitimate users.
The SCEPTRE experimental environment testbed contained real and simulated components that are representative of a DER communication system. The simulated inverters were Linux-based, unhardened, and network-connected, much like many commercial DER devices on the market. The simulated routers ran valid routing/firewalling services, but not on actual hardware.
The Emulytics challenges for the red team included a reduction in attack surfaces, no human elements, limitations in hardware, software, and firmware diversity, and limited emulated system complexity to subvert. The biggest challenge was found to be the interactions between the backend processes -SCEPTRE, Phēnix, Minimega, and OpenDSS -because they are a disparate set of tools not originally designed to seamlessly interface together in real-time. It was common that the environment needed to be reinitialised to complete all the assessments.
The flat topology was designed to represent a network where the utility communicated to the inverters in a large area network. The segmented topologies included the same utility to inverter interactions but have added security features controlled by a system owner inside their network. The MTD network employed dynamic configuration to obfuscate network and routing parameters. The network topologies for these experiments, though simplified and contrived, are still representative of some real-world DER control networks.

Assessment approach
The following red team assessments were conducted on each DER control reference architecture: (i) Network reconnaissance: this phase involved the intruder actively gathering information about the vulnerabilities of the target system. This yielded network information, including IP addresses, MAC addresses, open ports, slave IDs, vulnerable services, and operating systems. (ii) Fabrication: this network attack inserted or maliciously replayed fake messages on the network to investigate the confidentiality and integrity of data transfer between the utility and the RTUs. (iii) Interruption: this network attack generated a deluge data transmission to render the system unavailable to legitimate users. This test investigated the availability of the RTUs and utility to operate under a denial-of-service (DoS) attack. (iv) Interception: under this attack, data transmissions were eavesdropped, maliciously dropped, delayed or altered while in transit from the utility to the RTUs and vice versa. This test investigated the confidentiality and integrity of data transfers under a man-in-the-middle (MITM) attack.

Experimental results
Red team assessment observations and challenges from each SCEPTRE environment are summarised below. Further details of the exposed vulnerabilities are provided in [38].

Flat network without encryption: Observations:
reconnaissance enabled mapping of the network. The routers and inverters were susceptible to DoS attacks. MITM attacks between each inverter and the corporate utility router were possible. Fabricated data was easily replayed to modify grid-support functions on the inverters.
Challenges: None. This environment was the baseline for the assessments. Challenges: on a BITW encryption setup, an attacker intercepting traffic between the BITW endpoints will only see encrypted traffic across any potential attacker-controlled parts of the network, but this challenge was not encountered due to misconfiguration of the encryption tunnel that exposed unencrypted data in some communication paths.

Segmented network without encryption:
Observations: the red team was provided two access points, one on the ISP router's subnet (outsider access) which was bereft of inverters and the other access on one of the subnets with a subset of the inverters. Reconnaissance was successful from both access points. From the outsider access, MITM was unsuccessful because there were no addressable inverters. Attempts to pivot using the password-less SSH boxes and deploy MITM were unsuccessful due to Linux package dependencies on the air-gapped SCEPTRE network, i.e. the needed tools could not be loaded on the SSH VMs without tearing down and rebuilding the Emulytics environment. MITM was only successful within one local subnet or enclave. However, DoS and replay attacks were successful from both access points.
Challenges: from an outsider position on an emulated network, it was not a target-rich environment. Pivoting into subnets with targets was difficult when hosts did not have the human element and OS vulnerabilities seen in the real world.

Segmented encrypted network:
Observations: the red team was provided the same two access points described above. Encrypted tunnels to the utility (corporate network) were created to each segmented unencrypted subnet using BITW SSH gateway hosts. Reconnaissance confirmed the encryption tunnel was again misconfigured, with the inverters immediately connected to the ISP router rather than being located behind the SSH box. Placing the inverters behind SSH boxes would have ensured the encryption of data between the inverters and the gateway, thereby eliminating some attack types on the same subnet as the inverters. While MITM was still an available attack when the adversary was on the DER subnet, an outsider without the ability to pivot and deploy tools remains excluded from this attack vector. Again, DoS and replay were successful from both access points.
Challenges: no unique challenges were introduced in this topology.

Segmented, unencrypted network with HIL DER:
Observations: only the outsider access was granted to the adversary in this topology. The HIL device was not on the same subnet as the adversary. Reconnaissance, replay, and DoS attacks were successful. MITM attack was unsuccessful because there were no inverters in the same subnet with the adversary.
Challenges: the HIL inverter was known to have grid-support functions and communicate with user datagram protocol (UDP). However, while attached to the Emulytics environment, the HIL could not be commanded with Netcat UDP packets and the red team did not discover whether this was due to the Emulytics platform translating all traffic through protocol buffers or due to other network effects. Python UDP communication still succeeded in replaying fabricated data.

Flat MTD without encryption:
Observations: MTD provided a couple of features that initially inhibited red team traction. Vulnerable switch proprietary protocols running on default switch configurations were exploited for reconnaissance resulting in very large area network information, SDN controller IP addresses, and open ports. DoS attacks on the switch were successful but MITM was not successful.
Challenges: the MTD environment was built with SDN concepts inside an Emulytics platform. This platform is also built on rapid prototyping models of SDN, causing a fusion of certain network surfaces that would have been separated in the real world. For instance, a real MTD system would protect the applications and application plane communications with the interceding control plane, leaving the controller and control plane communications as a new attack surface. Conflation of the Emulytics platform and the MTD environment may have contributed to difficulties defining what element was in scope and what new attack surfaces were available.
Finally, the common observation and challenge evident in all the topologies was the abbreviated set of DER Modbus registers in the virtual devices. This artificially limited the attack surface of the simulated inverters.

Summary
In the assessments of each of the network topologies, the team identified vulnerable areas that could be exploited -mostly due to flaws in system configurations and network implementation. To quantify the impact of the red team on the virtualised communication networks, a scoring rubric was created loosely based on prior assessment work with military microgrids [54][55][56].
The findings of this assessment are summarised in Fig. 4. For the confidentiality, integrity, availability (CIA) columns, a scale of 1 to 5 was created to categorise the risk levels. A score of 1 indicated a low risk to all the devices and a score of 5 indicated a high risk to most of the devices. Scores between 2 and 4 indicated an increasing number of compromised devices or system risk. Total scores were summed for a security risk score between 3 and 15. For this defined range, scores between 3 and 4 were assigned a low risk, scores between 5and 9 as medium risk, and scores between 10 and 15 were assigned as high risk, based on the red team impact on CIA. This was completed for the theoretical security posture provided by the defensive components and the as-built system the red team assessed and quantified, as indicated in Fig. 4.
Unfortunately, it is difficult to calculate a power system risk using red team scores from the penetration tests [56,57] because a single vulnerability can compromise the entire DER network and drastically impact the power system. Typically, enterprise cyber teams will establish scoring rubrics to track their cybersecurity posture over time using tools like ATT&CK [58], Nessus/common vulnerability scoring system (CVSS) [59], or STRIDE [60]. In this work, scores were generated for barebones environments and strictly represented a gradation of security practices indicating the quality of defences and number of ICS assets that were subverted [61], but not a measure of time/effort required to affect OT network or power systems operations. The scores were partially advised using the CVSS methodology that ranks vulnerabilities based on ease and severity of exploitation [62]. The following rubric was used for the CIA scoring: •  Fig. 4, clearly proper security feature implementation is essential to gain the advantages offered by these technologies. In the case of implementation errors or oversights -as were common in these assessment environments -minimal additional effort was required by the adversary to sidestep the defences and subvert the DER control network. Any implementations that did not receive scores of 1 across the board should be considered exploitable and/or at risk for disruption. Thus, all topologies the red team assessed represent some power system risk.
Based on the results of the assessments, the following recommendations are provided: (i) DoS attacks are difficult to prevent (as evidenced by the March 2019 attack on sPower [63,64]). Aggregators/utilities should regularly patch their networking equipment and implement firewall whitelists to mitigate these attacks. (ii) Segmentation makes it difficult for the adversary to move between subnets. Flaws in system configuration and networking implementation enabled manipulation of all DER devices. (iii) Implementing the right encryption tunnel between DERMS and DER drastically reduces the risk of replay and MITM attacks. (iv) It is important that developers add layers of defence by reviewing and pushing secure code to applications to prevent common attacks.
(v) MTD has the potential to drastically improve security for DER networks, but this is still an area of research.

Conclusions
This work studied the trade-offs between communication quality of service (QoS) metrics and cyber resilience for a grid service provided by a DER control network. To effectively provide grid services (e.g. voltage regulation, frequency reserves, protection etc.), certain tolerances for latency, networking dropouts, and communication availability were previously determined. Using those communication requirements as a reference, the impact of network segmentation, encryption, and MTD was calculated. It was found that each of the three security features generates minimal increases in latency compared to unsecured topologies, and therefore would not adversely impact grid control operations, while substantially increasing the theoretical cybersecurity posture of the control network. Also, while this additional security comes with some operational overhead for maintaining the necessary infrastructure, it is believed the security benefits outweigh those costs. It is noted from the red team assessments, however, that improper implementation of the security features will only provide a false sense of security and dedicated adversaries will find ways to evade these defences.
Further work is recommended to assess the security features of other communication protocols, such as IEEE 2030.5, for a better understanding of the security posture in soon-to-be-fielded DER control environments. The inclusion of aggregator services in the DER networks should also be considered for future SCEPTRE red team assessments, as these are commonly used to pass control and measurement data to/from utilities. Lastly, the full lifecycle of DER operations should also be analysed from a security perspective and include establishing firmware update procedures, patching requirements, and recommended maintenance schedules.

Appendix: network topologies
The construction of the networking topologies in SCEPTRE required programming different custom-built VMs running opensource software and emulators. The flat, encrypted environment, shown in Fig. 5, established an encrypted tunnel between the utility corporate network subnet (192.168.0.0/24) and the DER OT network (75.75.128.0/24) with two Vyatta routers running AutoSSH. This was intended to prevent the red team from seeing plaintext DER network traffic from their connection at the ISP router. However, the red team was still able to issue Modbus commands to the DER devices from their location on the network. Fig. 6 shows the segmented, encrypted topology with three DER enclaves that exchange encrypted data with the utility DERMS. In this topology, no firewall rules were implemented at the segment boundaries, so the red team was once again able to issue commands directly to the DER but unable to see the utilityto-DER communications in the other segments. The addition of strict firewall rules would greatly improve the cybersecurity resilience of these topologies.