Hierarchical control system for a flexible microgrid with dynamic boundary: design, implementation and testing

The design, implementation, and testing of a control system for a flexible microgrid (MG) is presented in this study. The MG controllers can be implemented in a real-world MG with multiple smart switches, photovoltaic panel system, and battery energy storage systems (BESSs). With the benefits from smart switches, the MG has unique characteristics such as dynamic boundary and flexible point of interconnection (POI) concepts. To control such a unique MG and realise the dynamic boundary, an MG central controller and two types of local controllers are implemented. Compared to the MG with fixed boundary, the MG with dynamic boundary can have smaller BESS capacity, better utilisation of renewable energy, and multiple POI options. Also, compared with IEEE Std 2030.7–2017, the topology identification and active and reactive power balance functions are newly designed to realise the dynamic boundary concept. The planned islanding and reconnection functions are modified to realise the flexible POI concept. These functions are introduced including the software architecture, cooperation, and interaction among them. Finally, a hardware-in-the-loop testing platform based on the Opal-RT real-time simulator is set up to verify the performance, realisation of the dynamic boundary, and flexible POI concepts with four comprehensive test scenarios.


Introduction
Owing to the increasing deployment of distributed energy resources (DERs), distribution system resiliency under extreme weather, and power supply needs for remote communities, microgrids (MGs) have been actively researched [1]. Based on the definition by the U.S. Department of Energy [2], an MG should have (i) clearly defined electrical boundaries; (ii) a control system to manage and dispatch resources as a single controllable entity; and (iii) installed generation capacity that exceeds the critical load. From this definition, it is possible to combine multiple renewable DERs and smart switches into the MG design, real-time control, and implementation of MG controllers [3].
In the previous MG design research, the MG boundaries are usually fixed and typically determined based on reliability [4] and energy supply adequacy [5,6]. An MG with fixed boundary usually installs multiple DERs including battery energy storage systems (BESSs) and stochastic renewable energy sources such as photovoltaic (PV) panels and wind turbines. With the development of the smart switch, power grid monitoring, and communication technologies, it is possible to deploy smart switches between load sections in an MG. With the help of smart switches, the MG can choose the load sections inside the MG autonomously. Thus, the boundary of the MG can extend or shrink dynamically, i.e. the dynamic boundary concept [7][8][9]. With dynamic boundary, one of the benefits is that the capacity of the BESS in the MG can be designed much smaller than the MG with fixed boundary. This is because the MG with dynamic boundary can shrink the boundary when renewable DERs cannot support non-critical loads so that the BESS only needs to provide power to the critical load instead of the whole MG. With a BESS with smaller capability, the MG cost can be lower [10].
In addition, due to the stochastic power generated by renewable DERs, it is natural to design an MG with dynamic boundary concept according to the generated power from renewable DERs and remaining state of charge (SoC) of the energy storage systems [11]. With dynamic boundary, the MG may extend the boundary when the power generated from renewable DER is high and vice versa. In this case, renewable DERs can be more fully utilised so that there is no wastage of renewable energy.
Last but not the least, with dynamic boundary, an MG would have multiple point of interconnection (POI) options to choose instead of a fixed POI location which typically yields a more flexible and resilient system. Dynamic boundary provides more degrees of freedom in MG design and real-time control aspect so that the MG becomes much more flexible and reconfigurable. Kim et al. [12] and Du et al. [13] have utilised dynamic boundary concept to design the network reconfiguration control technology on distributed grid and MG, respectively.
On the other hand, the dynamic boundary concept poses challenges to the design, implementation, and testing of MG controllers. There are many commercial MG controllers available in the market. Kowalczyk et al. [14] have investigated the MG controllers such as Schweitzer Engineering Laboratories MG systems and MGC600 from Asea Brown Boveri. In addition, MG controllers are also available from Siemens [15], General Electric (GE) [16], Eaton [17], and Spirae [18]. These existing commercial MG controllers share similar functions, e.g. energy storage systems integration and controls, demand management, renewable and load forecasting etc. Furthermore, several functions, e.g. feeder shedding based on generator overload instead of under frequency and generation unit commitment, are exclusively available in certain products.
In addition, there are some university designed MG controllers implemented and tested in real MGs. A plug-and-play control system has been designed and tested in Consortium for Electric Reliability Technology Solutions MG laboratory test bed for more than ten years [19]. Illinos Institute of Technology has designed, implemented and tested a networked MG system with multiple MGs [20]. University of California, Irvine, has also developed and tested its MG on campus [21]. Most [22]. However, there are no functions supporting the dynamic boundary concept mentioned in [22]. This paper aims, to discuss the controller design, implementation, and testing with dynamic boundary and flexible POI concept. The difficulty of design, implementation, and testing of an MG control system with dynamic boundaries is that a unique software architecture including innovative functions and cooperation between functions is required to realise dynamic boundary and flexible POI concepts. The major contribution of this paper is to discuss the unique designed functions, modified functions, and their coordination to realise dynamic boundary concept in a MG control system. Also, to implement and test the MG controllers with dynamic boundary and flexible POI concepts, a National Instruments (NI) compact RIO-based MG central controller (MGCC) and two local controllers (LCs) are designed and introduced in this paper. This paper focuses on the design, implementation, and testing of the MG with dynamic boundary and verifies the benefits from the dynamic boundary and flexible POI concepts. Since designing the MG controller is the key to implementing the dynamic boundary and flexible POI, it is necessary to discuss the function blocks, the controller architecture, and interaction among function blocks in MG controllers. In this paper, a real-world MG system is first introduced with circuit model and dynamic boundary concept. Then, the structure and function blocks of MG controllers are introduced followed by the architecture, cooperation, and interactions. Finally, the implementation and test of such MG controllers are verified through a Hardware in the Loop (HIL) test bench, i.e. using Opal-RT (Brand [23]). Four different MG scenarios are designed and tested which verify the functionality, the dynamic boundary, and the flexible POI concepts. The highlights of this paper are listed as follows: • The topology identification function is designed to automatically identify the MG topology based on smart switch status so that the MG boundaries can be identified from the main grid. • The active and reactive power (PQ) balance function is designed to automatically extend and shrink the MG boundary considering load shedding, load restoration, BESS antiovercharging, BESS anti-overdischarging, and PV curtailment.
• The planned islanding and reconnection functions are modified to island from the main grid and reconnect back to the main grid on multiple POI locations. • The controller software architecture and function interactions are newly designed to implement and realise the dynamic boundary and flexible POI concepts.

MG system
In this research, one of the unique benefits comes from the real MG system with dynamic boundaries implemented by Electric Power Board (EPB) of Chattanooga. This MG consists of a BESS, PV generation, and smart switches implemented near the Chattanooga airport [24]. As shown in Fig. 1, the MG is designed as the area inside the yellow dash lines. This MG consists of a PV generation area with nearly 2 MW peak power, a BESS with a capacity of 600 kWh, a critical load area (serving some portions of the Chattanooga airport), and multiple non-critical load areas with ∼2.85 MW. In addition, three traditional POIs are shown in the black circles which can connect to different main feeders. Note that the smart switches are implemented between load sections which cannot be observed in this figure. In this section, the circuit model, the dynamic boundary, and flexible POI concepts will be introduced.

Circuit model
For the purpose of simplification, the real MG is simplified into a 14 bus system, shown in Fig. 2. Here the load sections are represented by arrows named Load-2 to Load-11; three POIs are named as G-1, G-6, and G-14; BESS and PV are BESS-13 and PV-12. Meanwhile, the smart switches are represented by the bus number between them, such as 'SW0102' means the smart switch between G-1 and Load-2. The critical load is represented by Load-10 which is directly connected to the BESS and PVs so that it can be powered on once the BESS or PV is powered on. This is because the priority of the critical load is much higher than the non-critical ones.
On the other hand, the non-critical loads are connected to the MG through smart switches which can be optionally powered on and off. Since the smart switches control strategy is not the focus  of this paper, it is briefly discussed in the function block PQ balance inside the MGCC. In addition to the three POIs, all smart switches can be POIs, i.e. the MG can be separated from the main feeder through every smart switch.

Dynamic boundary and flexible POI concepts
In a previous research, the boundaries of the MG are usually fixed because of hardware limitations and the commercial customer power supply requirements [12]. With the benefits from EPB MG, it becomes possible to implement MG controllers with dynamic boundaries and flexible POI concepts. The dynamic boundary is based on the application of smart switches. With smart switches, the boundary of the MG can be extended or reduced. The first benefit of the dynamic boundary is that the BESS in an MG with a dynamic boundary can be much smaller than an MG with a fixed boundary. This benefit has been discussed in [10]. In addition, there is no wastage of renewable energy in an MG with dynamic boundary. An example extending and reducing boundaries of the proposed MG is shown in Fig. 2. As shown in Fig. 2, the MG is represented in green lines while the boundary smart switches are in red. In the midnight case, without the PV power support, the MG shrinks the boundary to guarantee the power supply of the critical load (Load 10). On the other hand, in the morning case, with PV power supply, it can be observed that through extending operation the boundary of the MG covers Load-7, Load-8, and Load-3. In this case, the power from the PV is not wasted but utilised to cover some noncritical loads. In addition, the dynamic boundary can have benefits in the MG for the islanded state, the islanded to grid (I2G) connected state, the planned grid to islanded (PG2I) state, and the grid-connected state. The highlights of the dynamic boundary concept in a MG are listed as follows: • The BESS capacity of the MG with dynamic boundary can be much smaller than the MG with a fixed boundary. • In the islanded state, the boundary of the MG can be extended or reduced automatically according to the PV active power and the SoC of the BESS, so that no renewable energy is wasted and the power supply for the critical load power is guaranteed. • In the I2G connected state, the MG can choose an arbitrary smart switch to be the POI through which the MG can connect to the main feeder. Therefore, the POI can be flexible in the MG. • In the PG2I state, the MG can be islanded from the main feeder at a designated smart switch. During the islanding process, the active and reactive power flow through that smart switch can be minimised. • In the grid-connected state, the MG can change the POIs through islanding and being reconnected to another main feeder without losing power to the critical load.

MG controllers
The relationships among main feeder supervisory control and data acquisition (SCADA), MGCC, LCs, smart switches, grid connectors, BESS, and PVs are summarised in Fig. 3. It can be observed that the main feeder SCADA is the top-level commander that will communicate with MGCC, smart switches, and grid connectors through distributed network protocol (DNP3). Meanwhile, the MGCC is the commander for an independent MG which focuses on system-level control and energy management function blocks. It will exchange real-time data with LCs and smart switches through DNP3. Finally, the LCs will directly monitor the PV and BESS status such as PV irradiance and battery SoC and control the working status of PV and BESS. Note that for simplicity purposes, in the HIL experiment, the MGCC would directly communicate with the smart switches and grid connectors so that the SCADA will not be modelled in HIL experiment.

MG central controller
The software logic of the MGCC has two major steps, i.e. the model management and other function blocks. In the model management, the MGCC will gather the MG constant parameters from comma-separated values files and generates related global variables. In the second step, thanks to the benefits of LabVIEW real-time module, function blocks can work in parallel, i.e. working in independent while loops. The inputs and outputs of function blocks are utilising shared variables and global variables while the variables inside function blocks are utilising local variables. Therefore, each function block could work independently. Once the inputs are updated, they can generate related outputs in real-time.
The benefits of this overall architecture are listed as follows: • The function blocks are working in parallel and independently so that they are more flexible and easier to test. • The inputs and outputs of the function blocks are clear and easy to change.
There are 14 function blocks developed in the MGCC and 6 function blocks in one LC. The relationship among function blocks is shown in Fig. 4. Owing to the page limitation, only 6 function blocks in MGCC will be introduced in this section because they are closely related to the dynamic boundary and flexible POI concepts.
Other function blocks are the normal ones required by IEEE Std 2030.7-2017. For reference purposes, the preliminary functions of all function blocks in MGCC are given in Table 1.

Finite state machine:
The finite state machine (FSM) function block determines the MGCC state based on the flags and events feedback from all other functions. The FSM will also generate related enable/disable signals for function blocks. Since the dynamic boundary and flexible POI concepts require the MG to be islanded or grid-connected, 5 FSM states are the islanded (ISD), I2G, grid connected (GC), unplanned grid to islanded (UPG2I), and PG2I states. The invoked enable/disable signals for MGCC functions under these five states are shown in Table 2.

Topology identification:
The topology identification function block can monitor the current smart switch statuses through the communication function block and then generates the MG real-time topology matrix. The MG topology matrix is utilised by PQ balance and reconnection function blocks. The flow chart of the topology identification is shown in Fig. 5. Basically, the topology identification utilises a Kruskai's algorithm-based searching method [7] in graph theory to generate the MG topology. The loads and sources separated by smart switches can be treated as nodes while the smart switches are treated as lines. The first step for the topology identification is to determine the independent connected-graphs given the smart switches status from communication function. These connected-graphs can be labelled as grid areas. The next step is to determine the boundary switch for each grid area and check whether the connected nodes are energised. The energised node can be determined by checking the power/voltage on both sides of the smart switch. If the power/ voltage of the other side of the smart switch is almost zero, it can be treated as the boundary node. The final step is to list all smart switches on the grid area boundary to be critical and other switches as non-critical (Note that the critical switch can be found through the power/voltage check on both sides of the smart switch.). These critical switch topology matrices will be sent to reconnection function while the non-critical ones will be sent to PQ balance function.

PQ balance:
The PQ balance function block provides a smart switch control strategy for balancing the active and reactive power of load sections within the MG during ISD state and the I2G state. In this paper, a rule-based real-time planning and monitoring strategy is applied in the PQ balance function block to balance the active power of load sections. The objective function of the PQ balance can be described as follows: subject to: where P i L s are the load powers inside current MG boundary, P j PV s are the PV powers inside current MG boundary, n is the load number inside current MG boundary, m is the PV number inside current MG boundary, P b C, max is the maximum BESS charging power, and P b D, max is the maximum discharging power of the BESS. The rule-based real-time planning and monitoring strategy has five subfunctions, i.e. load shedding, load restoration, antiovercharging, anti-overdischarging, and PV curtailment functions.
As shown in Fig. 5, the flow chart gives the basic rules of the PQ balance strategy where ∑ P i PV is the active power of the PVs, ∑ P j L is the total active power of load sections, and P Imb, Max is the maximum imbalance active power that is allowed before load shedding or load restoration command is given. First, the PQ balance will read the current MG topology given from topology identification. In addition, the power flow data will also be read in from communication function. Based on the read in data, the PQ balance can determine whether load restoration, load shedding, or PV curtailment is necessary. If the PQ balance determines it is necessary to do the load restoration or load shedding, an alternative generating-based algorithm will be utilised to generate all possible alternative MG boundaries and choose the optimal solution to extend or shrink the boundary [25]. In addition, the PQ balance will also monitor the SoC of the BESS. If the SoC of the BESS reaches the maximum threshold, (2) will be replaced by On the other hand, if the SoC of the BESS reaches the minimum threshold, (2) will be replaced by

Energy management:
The energy management function block focuses on day-ahead economic power dispatch under GC state and the smart switch coordination under PG2I state (Planned Islanding). As shown in the flow chart given in Fig. 5, with the forecasting active power from PV and load forecasting function blocks (The PV/load forecasting algorithm is designed based on [26]. The average accuracy of the one day forecasting results is around 90 to 95%.), the energy management function block can make general scheduling/dispatch strategies to reduce the electric bill in grid-connected mode, and to provide energy surety to the critical load under islanded state. The objective function for the energy management under grid-connected mode is subject to: where P i, t L are the forecast load power at time t, P j, t PV are the forecast PV power at time t, P b, t is the BESS power at time t, α t is the price of electricity at time t, SoC b, t is the SoC of the BESS at time t, P b, t C is the BESS charging power at time t, P b, t D is the BESS discharging power at time t, E b is the capability of the BESS, η C is the BESS charging efficiency, η D is the BESS discharging efficiency, SoC b min and SoC b max are the minimum and maximum SoC of BESS, respectively. Since (5) and all constraints are linear, a mixedinteger linear programming solver is implemented to solve this optimisation problem. To eliminate the influence from the large forecasting error, two limits are added before the optimisation solver starts where P i L, min are the minimum load forecasting power, P i L, max are the maximum load forecasting power, P j PV, min are the minimum PV forecasting power, and P j PV, max are the maximum PV forecasting power.
Under PG2I state, the energy management objective is to minimise the power flow through the POI (Planned Islanding). To realise this target, the energy management function will first read in the current MG topology, smart switch status, the source power, and the BESS power. Then, it will ask the PQ balance for the candidate smart switch combination. Given the candidate smart switch control signals from PQ balance, the energy management function block will schedule the BESS active power and smart switch coordination to minimise the active/reactive power flow through the POI. Finally, if it is necessary, the energy management function may curtail the PV output power.

Reconnection:
The reconnection function block focuses on reconnecting the islanded MG back to the main grid. This function block is designed and implemented in both MGCC and BESS LC. The flow chart for both parts is given in Fig. 5 where the part in MGCC is called reconnection and the part in LC is called resynchronisation. It can be observed that through the commands from the reconnection in MGCC, the resynchronisation will try to minimise the voltage difference until the MG is able to reconnect back to grid. The unique design of this function is that the topology identification will give the critical switch topology matrix to the reconnection function so that the reconnection can be done on the boundary smart switch. In this case, the reconnection can be utilised on multiple POI location instead of a fixed location.

Communication: The communication between MGCC and
LCs is utilising DNP3 [27]. DNP3 is widely utilised in electric and water companies. Since EPB SCADA system is using DNP3, the communication link between MGCC and LCs here is also using DNP3.

Local controllers
The LCs have a similar software structure as MGCC with fewer function blocks, shown in Fig. 4. The main purposes of the LC controllers are to communicate and control the voltage, frequency, and active/reactive power of the PV and BESS. The preliminary functions of the function blocks in LCs are shown in Table 3. Since the function blocks in LCs are simple and straightforward, a detailed discussion is not given in this paper.

Function block cooperation and interactions
Based on the discussion in the FSM function block, multiple function blocks can work simultaneously under one state. Thus, the cooperation and interactions among functions are critical to implement the dynamic boundary and flexible POI concepts.

ISD state:
Under ISD state, the most critical function block is the PQ balance since it can implement dynamic boundary through controlling the smart switches. As shown in Fig. 4, the FSM function block provides the enable signal when MG moves to ISD state; the topology identification function block provides the current MG topology; the communication function block provides the PV, BESS, and smart switch status. After the PQ balance algorithm ends, as shown in Fig. 5, the new smart switch control signals will be given to communication function block and then close/open the smart switches.

I2G state:
Under I2G state, most of the function blocks under ISD state are still enabled. However, the critical function block is the reconnection function block because it can implement flexible POI concepts. As shown in Fig. 4, the PQ balance function block will generate the candidate smart switch location to the reconnection function block. With the current MG topology and the candidate smart switch location, the reconnection function block can determine smart switch control signals to the communication function block.

PG2I state:
Under PG2I state, to achieve a smooth transfer from GC to ISD state, the energy management function block will generate both BESS active power signal and smart switch control signals with the help from PQ balance function block. As shown in Fig. 4, the PQ balance function block will generate the candidate smart switch control signals and then the energy management function block can determine the POI location and active/reactive power from the BESS. Finally, the energy management function block will generate the BESS active power signal and smart switch control signals to the communication function block.

GC state:
Under the GC state, the critical function block is the energy management function block. As shown in Fig. 4, based on the forecasting one day ahead PV and BESS power profile, the energy management function block can optimise the BESS and PV active power. Therefore, an economic power dispatch can be achieved under GC state. Since the energy management is not related to dynamic boundary concept in GC state, it will not be tested in the experiment. Another benefit from dynamic boundary under GC state is to change POI point without disconnecting power to the critical load section. The procedure of changing POI point is to move into PG2I state, I2G state, and GC state in sequence. This test requires the function blocks discussed in both PG2I and I2G.

Hardware in loop-based test
To test and verify the performance of the proposed MG controllers, a real-time HIL test utilising Opal-RT is developed and implemented. As shown in Fig. 6, the three NI-CompactRIOs are MGCC, PV LC, and BESS LC. The OPAL-RT and three NI-CompactRIOs are connected through a network switch inside an experiment demo cabinet. All the DNP3-based communication are implemented through this local internet. Besides the cabinet, a personal computer provides a user interface and models the commands from the distribution management system. Note that the real-time data including the frequency, voltage, and active power can be recorded into the FTP inside NI-CompactRIOs. The NI-CompactRIOs are chosen as general-purpose controllers which are widely utilised in electric vehicles [28], wireless power transfer systems [29] etc.
Under the OPAL-RT simulation environment, the MG feeder circuit is emulated in SimuLink. The control signals of critical circuit parts, e.g. smart switches, PV, and BESS are received through DNP3-based communication with LCs and MGCC. Thanks to the HIL simulation, it is possible to evaluate the performance of the controllers before implementing it into a real MG.

Hardware in loop experiment
The overall EPB feeder circuit in SimuLink is shown in Fig. 7. Similar to the MG model discussed in Section 2.1, the feeder circuit has exactly the same topology. The corresponding load sections, grid interfaces, smart switches, the PV, and BESS are labelled. To generate a more 'real world' feeder, the load sections here are utilising real-world dynamic loads, i.e. the load power would dynamically change following predefined profiles. For the PV, similar to load sections, two different real-world PV active power profiles are available. Additionally, the user can also manually tune the PV active power for testing purposes.

ISD state scenario
Under the ISD state scenario, the dynamic boundary concept will be verified through different PV active power conditions. The test procedure can be described as follows: FSM begins under ISD state with the SoC of the BESS to be 50% and then manually increase the PV active power from 1 MW (half of the maximum power) to 2 MW (maximum power) and back to 0 MW. The active power of the loads, BESS, and PV are shown in Fig. 8.
It can be clearly observed that by manually changing the PV active power (marked with green arrow), the boundary of the MG will extend (SW0203 and SW0304 are closed) and reduce (SW0203, SW0304, SW0307, SW0709, and SW1011 are opened) accordingly.

I2G state scenario
Under the I2G state scenario, flexible POI concept will be verified through reconnecting to the main feeder with arbitrary smart  Fig. 9, after reconnection command is sent out, the SW0102 is closed. Then, once the voltage, frequency, and angle differences are smaller than a predefined threshold value, the POI will be closed and thus the MG reconnects back to grid. Finally, the rest of the load sections will be powered on. The frequency, voltage, and angle difference of the POI (SW0203) are shown in Fig. 9. The experiment results verify that arbitrary smart switch can be the POI and the reconnection process is stable for voltage, frequency, and angle.

PG2I and UPG2I states scenario
Under PG2I state scenario, the comparison between PG2I and UPG2I is given to further verify the benefit of the flexible POI concepts. In this case, the FSM begins under the GC state with the SoC of the BESS to be 50% and PV active power following the predefined power profile. Then the MG can move into ISD state through either the PG2I state or the UPG2I state. In PG2I state, due to the benefits from dynamic boundary and flexible POI concepts, every smart switch can be the POI. Thus, the active and reactive power flow through the POI can be stable (active power gap is 0.0056 p.u. and reactive power gap is 0.0036 p.u.) through scheduling the BESS active power and smart switch coordination. As shown in Fig. 10, with the cooperation between the PQ balance and energy management, when the POIs are opened, the frequency and voltage are stable. On the other hand, in UPG2I state, since there is no scheduling operation, the POI is the grid interface bus and the active/reactive power flow through the POI has a large gap. As shown in Fig. 11, the active/reactive power flow through SW0406 has a large gap (active power gap is 1.73 p.u. and reactive power gap is 0.70 p.u.) during islanding operation.

GC state scenario
Under the GC state scenario, another benefit from dynamic boundary and flexible POI concepts, i.e. the POI of the MG can be changed without powering off the critical load section, is verified. As shown in Fig. 12, the MG has reconnected back to the grid through G-6 at first and during GC state, all smart switches are closed. Once the POI change command is sent out, it will move into PG2I state and open SW0406, SW0203, and SW0304. After successfully islanding, the MG will wait for the reconnection command. Finally, the MG reconnects back to grid through G-14 and finishes the POI change (from G-6 to G- 14). Note that the critical load is always powered on during these operations.

Conclusions
This paper discusses the software level design, implementation, and testing of MG controllers with dynamic boundary and flexible POI concepts. The major function blocks in both MGCC and LCs are introduced. Then, the cooperation and interaction among function blocks to implement and realise dynamic boundary and flexible POI concepts are discussed. Finally, in the HIL test, four test scenarios have successfully verified the functionality, dynamic