Energy aware adaptive resource management model for software ‐ defined networking ‐ based service provider networks

Service provider networks (SPNs) seek to implement the resource management mechanisms called load balancing and energy saving to satisfy today's networking demands. Software ‐ defined networking (SDN) allows SPNs to perform agile and efficient networking operations. It is still an open issue on how to implement these resource management mechanisms in SDN ‐ based SPNs. An energy ‐ aware adaptive resource management model (ERM) for SDN ‐ based SPNs is proposed. Load balancing aims to distribute overall network traffic volume as fairly as possible among available links. Energy ‐ saving aims to turn off as many network devices and links as possible. Unlike the works in the literature, our model utilises the trade ‐ off between load balancing and energy efficiency. To do so, the controller establishes paths among edge pairs. The controller aggregates traffic load between edge pairs to as few links as possible based on a pre-defined load level. Then, unused network elements switch to sleep mode. In the case of load balancing, the controller distributes traffic load among active network resources. Experimental results show that ERM can successfully increase both energy ‐ saving and resource utilization even under heavy traffic loads.


| INTRODUCTION
Advancements in both wired and wireless networks, the proliferation of mobile devices and applications have changed the way we use the Internet usage. Within the last 2 decades, there have appeared applications and services such as e-commerce, on-line gaming, social media, cloud services, and the Internet of Things (IoT). Accordingly, the volume of Internet traffic has increased, and Internet traffic characteristic has differentiated dynamically [1,2]. Due to the lack of smart and autonomous systems, service providers (SPs) usually serve their network resources such as bandwidth in full capacity to satisfy diverse user demands and accommodate traffic bursts. Thus, network devices such as routers and switches are active all the time, even if network traffic is relatively low compared to peak times [3]. This results in 2%-7% of the World's electricity consumption in Information and Communication Technologies (ICTs) [4,5]. Besides, the energy consumption of ICTs has been growing 3% every year [6]. Thus, SPs need smart network management systems that not only increase resource utilization but also decrease energy consumption.
Software-defined networking (SDN) [7] is a new networking paradigm that emerged to satisfy today's networking demands. It separates control and data planes to provide centralised management, virtualisation, agility, programmability, and efficiency. These abilities allure the attention of SPs since they aim to increase resource utilization, decrease operational costs, and protect the environment. SDN-based resource management models proposed in the literature either perform traffic load balancing to increase resource utilization or energysaving both to decrease costs and protect the environment. The works that focus on load balancing distribute the network traffic as fairly as possible among links. Others that focus on energy saving aggregate network traffic to put as many links and switches as possible into sleep mode. Therefore, these two operations are usually considered as opposite. To the best of our knowledge, our work is the first work that focuses on utilising the trade-off between these two resource management mech-anisms in SDN-based networks. Apart from this, the SDN controller may become a bottleneck in terms of decision making and signalling as the network size, and the number of flows increases. It is still an open issue on how to perform these operations scalably in SDN-based Service Provider Networks (SPNs).
In this work, we propose an energy-aware adaptive resource management model (ERM) that utilises the trade-off between energy saving and load balancing. The proposed model is constructed upon our previous works [8,9]. Briefly, pre-established multi-paths between edge (ingress-egress) nodes abstract the physical complex network into a simple virtual one. Upon that, the controller performs routing, admission control, signalling, and load balancing in a scalable manner by taking paths into account. Apart from the works in the literature, we focus on utilising the trade-off between energy-saving and load balancing in the proposed model. To the best of our knowledge, this is the first work in literature that energy-saving and load balancing operations work in harmony. To make this happen, we introduce a trade-off value that determines the importance of these two operations against each other. If trade-off value favours energy-saving, the controller aggregates edge traffic into fewer paths and puts unutilised network elements such as links and switches into sleep mode. If the trade-off value favours load balancing, the controller saves less energy and focuses on distributing the traffic among paths. In line with the explanations above, we summarise the contributions of the paper below.
� To the best of our knowledge, this is the first work in the literature that balances the trade-off between load balancing and energy saving. Depending on the traffic volume of the network, the network administrator can adjust the level of energy saving. � Adaptive energy saving and load balancing mechanisms are proposed to provide scalable resource management. � The novel load balancing mechanism works in harmony with the proposed energy-saving mechanism. To make this happen, the controller balances traffic of edge pairs among active paths. � The controller can communicate with data plane elements over the modified OpenFlow [10] protocol to sleep and wake up them. � The controller performs routing, admission control, signalling, and resource management operations in a scalable manner exploiting pre-established multi-paths. � The proposed model can be applied to non-SDN-based SPNs by replacing switches on the edge with SDN-capable ones.
The rest of this work is organised as follows. Section 2 discusses related works. Section 3 describes background information about the proposed model. Section 4 presents design and implementation details of the proposed model. Section 5 presents the experimental results and the evaluation. Finally, the work is concluded in Section 6.

| RELATED WORKS
This section briefly explains the related works regarding to energy saving in SDN-based networks. Basically, there are two approaches that have been adopted for energy efficiency in SDN-based networks. These are link rate adaptation (LRAA) and power-down approaches. In the first approach, the link rate is adapted according to the traffic load. In the power down approach, ports, line cards, and integrated chassis of routers and switches are either turned off or put into sleep mode. The LRAA approach contributes less energy saving compared to the power down approach [11,12]. However, the power down approach causes routing oscillation and delay [13]. There are also some works that exploit both approaches.
The idea in Reference [14] is to reroute flows on existing paths to adjust link loads in a way that LRAA-based energy saving is maximised. In the scope of this study, the idea mentioned above is defined as a Mixed Linear Integer Problem (MILP). Then, three greedy algorithms and one genetic algorithm-based heuristic algorithm (GA) are proposed for redirecting flows over existing paths. Experimental results show that GA outperforms greedy algorithms. In this work, several paths are computed per pair before rerouting, whenever the controller executes proposed algorithms. Thus, the control plane and signalling scalability are limited. Although two real networks are used for assessment of the proposed algorithms, real traffic trace is not used. In addition, there are also no admission control and resource management in terms of load balancing mechanisms.
The following works utilise PDA to perform energy saving. The work [15] provides a 0-1 Integer Linear Programing (ILP) model that maximises energy saving in a global manner by taking energy consumption of integrated chassis, line cards, and ports of routers into account. It proposes two greedy algorithms namely Alternative Greedy Algorithm (AGA) and Global Greedy Algorithm (GGA) for energy saving. These two algorithms mainly reroute flows on different paths when the network has a relatively low traffic load. This work has routing and signalling scalability issues. The controller performs rerouting per flow. Besides, it communicates with all the switches along the new path per flow.
The work [16] combines IEEE802.az, that is energy efficient Ethernet and SDN-based Segment Routing (SR) to save energy. SDN controller computes link costs based on two metrics, which are EAGER and CARE. The EAGER metric takes link utilization into account. The CARE metric, on the other hand, takes both link utilization and congestion threshold into account. Two energy saving algorithms proposed in the work are tunnelling TE (TTE) and Nontunnelling TE (NTE), respectively. In TTE, the controller performs path computation per source and destination pairs in the first phase. Then, the controller establishes SR tunnels. In the final phase, the controller puts unused links into sleep mode. In NTE, the controller computes paths and decides links that will be put into sleep mode. In the second phase, there is no tunnel establishment. The controller only puts the designated links into sleep mode. Finally, controller computes ÇELENLIOĞLU AND MANTAR -89 ECMP routes between source and destination pairs using awake links. This work in fact has a similar idea that we proposed in our preliminary work [9]. Routing and signalling scalability are achieved via pre-determined paths. However, there is no contribution regarding admission control. Although it is said that load sharing is performed, authors do not mention how this is achieved.
Software Defined Green Traffic Engineering (SDGTE) [17] framework minimises active links and switches in backbone networks without knowing the future traffic demand. Authors provide an ILP definition of energy efficient routing problem. Whenever a flow arrives, the controller performs ILP based routing computation. Apart from energy-efficient routing, SDGTE reroutes flows both in under-utilised and over-utilised links to minimise energy consumption further and to reduce link utilization. In under-utilization case, links whose utilization is below the predefined threshold are determined, and flows on those links are rerouted. Then, the controller puts these links into sleep mode. In the latter case, the controller identifies the over-utilised links and re-routes the flows on those links to reduce link utilization. Apparently, the controller performs routing computation per flow whenever a new flow comes. Besides, the controller also performs re-routing in case of over and under link utilization.
Authors in [18] propose a multi-objective routing approach in a multi-controller control plane. To achieve these objectives, a multi objective evolutionary algorithm, called SPEA2 is developed. SPEA2 performs routing in a way that energy consumption and traffic delay is minimised without degrading the performance. Meanwhile, it takes both controller to switch and switch to switch loads into account. Whenever the controller receives a flow request, SPEA2 calculates a path taking data and signalling load into account. Subsequently, the controller establishes a path for the requested flow. The control plane in this work has multiple controllers. Thus, it satisfies routing and signalling scalability at some level.
Green Application Layer is an ETSI standard [19], which is a framework for exchanging information between control and data planes. The work [20] integrates GAL with SDN to exchange information regarding power management of data plane entities with a controller. This model adopts both PDA and LRAA. In that sense, it proposes an ILP model for the allocation of resources optimally based on network load and actions of flow tables. Experimental network in this study is small. The controller may become the bottleneck as network size, number of flows, and actions in flow tables of switches increase. Besides, the signalling scalability issue appears as flows are rerouted.
Authors in [21] aims to minimise active links and adapt discrete link rates to traffic load for saving energy. Firstly, they provide a mixed-integer programing definition of the problem. Secondly, they propose a heuristic algorithm, which identifies most energy consuming flows and reroutes them on alternative paths for reducing energy consumption. First, the proposed algorithm computes the shortest path for each flow. Subsequently, it calculates the energy consumption of the whole network. The value found at this stage is the upper bound for energy consumption. In the third stage, the proposed algorithm removes each flow one by one to identify its impact on energy consumption. To do so, a weighted graph is generated for each flow. Finally, the controller computes k-shortest paths and the with the least energy consumption is selected as a route. Obviously, the scheme has routing and signalling scalability issues since the controller performs per flow routing and rule installation.
Researchers in [22] propose two algorithms for allocation of Virtual SDN (VSDN) in a reliable and energy efficient manner. Relative Disjoint Path (RDP) generates two trees based on a redundancy factor. Then it merges the two trees to obtain a graph where links and nodes exist in both. In this way, there is only one path if the redundancy factor is 0. Similarly, there are two disjoint paths if the redundancy factor is 1. Otherwise, one or more links are shared among two paths. State-Aware of Bandwidth and Energy Efficiency (SA-BEE) algorithm allocates VSDNs based on available bandwidth and energy consumption factor. It adaptively increases or decreases energy consumption factor based on the network state at first. Then it generates a weighted graph for a source node to all other nodes. Finally, it looks for lower energy consuming paths between the source node and all other nodes.
In GreSDN [23], the controller performs energy-efficient routing without re-routing of flows. It maintains two topologies. The first topology is the physical topology, which contains all the links and switches. The second topology is the virtual topology, which contains only awake links. The controller performs per-flow routing bearing the two topologies in mind. If there is an inactive link along the path, the controller sends a signal to the corresponding switch to awake the link. Meanwhile, the device management module within the controller periodically checks the state on links. If a link is not used, it puts the link into sleep mode. The authors propose two routing algorithms; Constant Weight Greedy Algorithm (CWGA) and Dynamic Weight Greedy Algorithm (DWGA). Both algorithms perform path computation on two graphs. These graphs are generated per request. At the time of graph generation, the controller removes any link where the summation of the current load with requested bandwidth exceeds its capacity on both physical and virtual graphs. Finally, CWGA and DWGA perform routing computation based on static and dynamic link costs, respectively. Apparently, the controller performs per-flow routing. Thus, it suffers from a severe routing scalability issue. Besides, the controller must communicate with all the switches along the path during the path establishment process. This also results in the signalling scalability issue.
In the work [24], authors propose a heuristic algorithm (ETALSA) for energy saving via only powering up/down links. ETALSA that runs on a controller takes energy prices into account apart from other works. During the execution of ETALSA, the controller iteratively selects switches from the one with the highest energy cost to the lowest. Then, it computes the utility value per link, which is connected to the selected switch. The utility value is computed based on the connectivity of switches, traffic demand, and energy prices.
Finally, the controller powers a link down based on the utility value if there becomes less energy consumption. This work resembles our preliminary work [9] in a way that preestablished multi-paths exist both for scalability and resource management. The main differences are that the work [24] powers only the links down and it takes energy prices into account.
Consequently, the works in the literature mostly perform either load balancing or energy saving. Our work differentiates itself from the works in the literature, performing energy saving in compliance with load balancing.

| BACKGROUND
ERM relies on pre-established paths (PEPs) that connect ingress and egress nodes. The controller performs resource management operations, which are energy-saving, load balancing and path capacity updates, based on PEPs. Routing, admission control, and signalling scalability are also achieved in favour of PEPs as presented in our previous work [8]. Apart from the previous work, we propose ERM for SDN-based SPNs. The proposed model exploits the trade-off between energy-saving and load balancing. We propose a novel energysaving algorithm that is adaptive to the traffic load. Additionally, previously proposed load balancing and resizing algorithms are revised so that controller performs these operations in harmony with the energy-saving operation.
For the purpose of clarification, Tables 1 and 2 summarise main acronyms and notations used in this work, respectively.

| Architectural overview
The proposed model is illustrated in Figure 1. The data plane resides at the bottom and consists of provider edge and core switches. Ingress SDN switch (ISS) and Egress SDN switch (ESS) are the switches where traffic enters and leaves the network, respectively. An edge switch is both ISS and ESS at the same time because traffic is bidirectional. In addition to edge switches, there are core switches, which we name Core SDN switches (CSSes). A set of CSS connects edge switches and solely perform packet forwarding. In the upper layer, the proposed controller resides. It is logically centralised and mainly responsible for resource management (i.e. load balancing, active capacity resizing, and energy saving), routing, and admission control operations. To do so, the controller abstracts the single physical network into multiple virtual networks using PEPs between edge switches. All operations are performed based on these paths. The two layers in Figure 1 communicate with each other via a signalling protocol (e.g. OpenFlow [10]).
The proposed model has PEPs between each edge pair (EP), such as ISS 1 -ESS 1 in Figure 1. These paths are virtual and simplify the complex physical network. A simpler virtual network consists of edge switches and PEPs. The controller performs resource management operations, routing, admission control, and signalling based on the virtual network. Briefly, controller aggregates flow into less number of paths adaptive to the traffic load and then puts unused links and switches into sleep mode for energy saving. It also equalises the cost of paths that belong to the same EP to achieve load balancing. Controller, additionally, adjusts the capacity size of PEPs concerning traffic load to improve resource utilization more.
Notice that the controller does not perform routing and admission control operations per flow. In case of routing, the controller simply assigns a flow to a path. In the case of admission control, the controller decides based on the state of paths, which are stored in both link state database (LSDB) and path state database (PSDB). PEPs also allow the controller to manage the whole network with less signalling. This is because the controller does not communicate with each node along the path per flow in routing, admission control and resource management operations. It only communicates with edge switches, instead. Therefore, scalable routing, admission control and signalling are achieved in favour of PEPs.

| Network abstraction
The physical network is represented by a graph G = (V, E). Network administrator makes the decision of which pair has how many PEPs. In the scope of this work, we assume that PEPs are already exist in the network. Paths between EPs can be established using Virtual LAN (VLAN), MPLS or SR. In compliance with this, we do not develop any multi-path computation algorithm as there are already several algorithms used in the literature, such as [25][26][27]. |P s,d | per (V s , V d ) is defined by network administrator. Note that PEPs can be established regarding to any cost value such as residual bandwidth, latency, etc.
Notice that any two paths where each of which belongs to a different pair may share one or more physical links. We name such paths and pairs as neighbour paths and pairs, respectively. For the sake of simplicity, the controller virtualises links based on the number of neighbour paths per link. We denote a virtual link which serves to a path P s;d k as L s;d;k i . Capacity of a physical link, denoted as φ(E i ), is portioned out to its virtual links based on the load of neighbour paths. In compliance with this, a path capacity φðP s;d k Þ becomes the capacity of its virtual link which has the minimum capacity (minðφðP s;d;k i ÞÞ). For example, there are two paths on the physical link E 6 in Figure 1

| Controller design
The controller has two databases, four routing, admission control and resource management modules and three helper modules. The two databases are maintained in the background without interfering operations performed by modules. Routing, admission control and resource management modules have main tasks to handle but they can cooperate with other helper modules.
LSDB keeps track of physical network elements that are links and switches. Some of them are identifier of E i , corresponding switch datapath identifiers (DPID) and port numbers that E i connects, maximum transfer rate, power consumption and link load. Note that link load is not a static value unlike others because it changes in time regarding to the network load. PSDB to keep track of PEPs and other information regarding to them. This database is the virtualized version of the physical network. Controller mostly use this database for network management purposes. PSDB contains both static and temporary data. Some of the static information are path identifier (PID), datapath identifiers of edge nodes (i.e. ISS and ESS), links that form the path. Some of the dynamic data are path capacity and path load.
Routing and admission control module (RACM) allows flows to be routed over the associated edge nodes. In routing, the controller assigns the incoming flow to a path of the corresponding pair. In admission control, the controller checks available resources before making a decision regarding the routing. This way, it prevents the overloading of resources. If there is an available resource, the controller performs routing of the flow and follows the standard routing procedure.

ÇELENLIOĞLU AND MANTAR
Otherwise, it simply rejects the flow. Load balancing module (LBM) basically equalises the cost of paths per pair. At first, it computes the path costs of the corresponding pair and calculates the equalisation cost. Afterwards, it shifts flow in overloaded paths to under-loaded paths. Path resizing module (PRM) updates the capacity of paths depending on loads of neighbour paths and pairs. Whenever load of path exceeds a certain threshold, the resizing procedure is invoked. Energy saving module (ESM) aggregates flows into less number of paths. Then, it puts network elements that are not needed into sleep mode.
Path management module (PMM) establishes, updates or removes paths PEPs. Information collection module (ICM) collects data from underlying switches and performs necessary calculations on them to extract the information required for routing, admission control, load balancing, capacity resizing, and energy-aware resource management. The controller maintains all the information in databases. Inter controller communication module (ICCM) is responsible for communicating with other controllers in the control plane. This module prevents the network from a single point of failure problem. When the single controller stops working, all the routing, admission control, and resource management operations stop. In this work, we do not address this issue as there are proposed works mentioned in [28] regarding this. A simple solution for our model is that there can be two or more controllers within the control plane. One of them would be the master, and the other controllers, a.k.a. backup controllers, periodically check if the master controller is alive. In case of failure of the master controller, one of the backup controllers takes over. As the proposed model maintains network state in databases, the backup controller can directly connect to LSDB and PSDB to manage the whole network. Thus, there is a minimal overhead of taking control in the case of controller failure.

| Energy saving
There occurs a trade-off between optimal energy saving and computation overhead. In today's highly dynamic Internet traffic, the controller may become a bottleneck while computing optimal energy saving. Additionally, the optimal energy state may not last long due to the traffic fluctuations. Thus, we prioritise scalability over optimal energy saving. To do so, our controller makes EP-based energy-saving computation. Upon completion of pair-based optimization, the controller aggregates pair traffic into designated paths. Subsequently, it deactivates idle links and switches in the whole network. This local energy saving converges to a global energy saving in time.
To be clearer about how energy can be saved with traffic aggregation over less number of paths, consider the network illustrated in Procedure 1. The paths P 1;1 2 and P 2;1 1 are neighbours because they share the links E 6 and E 9 . Assuming that these paths have large enough capacity to carry their pair loads, the controller aggregates whole network traffic to P 1;1 2 and P 2;1 1 . Then, it deactivates the remaining four core switches and six links to save energy.
The first step of energy-aware path management is determination of the most energy efficient subset of paths of a pair P s,d while total capacity of P s,d (φ(P s,d )) is greater than or equal to its load (ϑ(P s,d )). Time complexity of the brute force solution [generation of all combinations (θ(2 n )) and searching for the best combination (θ(n))] are exponential. In this regard, we propose a polynomial time algorithm called energy-aware path management algorithm (EPMA) 1.
Let  . PP is a predefined utilization ratio (e.g. 70%). It allows the controller to aggregate the desired amount of load to the path as much as possible. For instance, defining PP as 70% forces the controller to make the path utilization around 70% but at the same time prevent it from overutilization. According to the Equation 1, PDV (P s;d k ) changes exponentially from the distance between path utilization and PP. The reason is to force the controller to reach the desired utilization as fast as possible.
Time complexity of EPMA is Oðn � |φðP s;d Þ − ϑðP s;d Þ|Þ. Notice that we design EPMA for a single pair. Therefore, total time complexity for the whole network becomes Oðn � |φðP s;d Þ − ϑðP s;d Þ| � BÞ where B is number of pairs. As soon as the controller determines future active and passive paths, it may need to shift flows from the paths that will be passive to the paths that will remain active or just have been activated. In such cases, the controller determines current and new states of the paths first. To be clear, there are four states which are defined in Table 3. If there are paths in State 1, the controller shifts the flows in these paths to the paths that are either in State 0 or State 2.
After the shifting process, the controller tries to deactivate all the network devices along the paths that do not share switches and links along their way with other paths. Since the whole path is passive, energy-saving increases dramatically. If this is not the case, the controller tries to deactivate the switches and links along its way that does not carry traffic. Before deactivating a switch, the controller makes sure that all the links coming and leaving the switch does not carry traffic. In addition to this, two new configuration values can be added to the configuration field of OFPT_PORT_MOD message for activating or deactivating ports.
EPMA finds the optimum path combination per pair assuming that all the paths are active. However, some of the paths may be in passive mode in reality and their capacity may be taken by neighbour paths. To handle this problem, ESM computes future capacities of passive paths because a passive path can be selected as a result of EPMA. To do so, it calculates the unused capacity of each link by subtracting load from its capacity and shares this unused capacity equally among the passive paths that the link serves. The capacity of the passive path becomes the capacity of the link which has the minimum capacity compared to other links that serve the same passive path. Let us show this procedure in an example for clarification purposes. Suppose that ESM executes EPMA for the pair ISS 1 -ESS 1 as illustrated in Figure 1. Also, suppose that capacities of all links are 1Gbps and paths except P 1;1 2 and P 2;1 1 are active. The controller must find the future capacity of Path2 in case EPMA selects it. There are one, two and two passive paths on links E 2 , E 6 and E 9 , respectively. Load of E 2 , E 6 and E 9 are all 0 because P 1;1 2 and P 2;1 1 do not carry traffic. Unused capacities of E 2 , E 6 and E 9 are 1 Gbps for each. In this case, capacity share of P 1;1 2 for the links E 2 , E 6 and E 9 are 1, 0.5 and 0.5 Gbps, respectively. The minimum of these capacities is 0.5 Gbps. Therefore, the future capacity of P 1;1 2 becomes 0.5 Gbps.

| Adaptive load balancing
Procedure 2: Active balancing algorithm The controller performs load balancing adaptive to the traffic for two reasons. First of all, the controller has limited computational resources. Thus, computationally intensive actions may degrade the performance of the controller such as optimal (network-wide) load balancing. Second, optimum load balancing may not last a long time because traffic is highly dynamic. Due to the aforementioned reasons, the controller balances the load of a pair among its active paths by equalising their costs (i.e. load, utilization, congestion level). Although pairs are virtual and individual, pair-based load balancing converges to network-wide load balancing. This is because cost computation is based on the cost of physical links.
Active balancing algorithm is presented in Algorithm 2. In the first step of load balancing, the controller calculates the cost of active paths and pair equilibrium cost (μ(V s , V d )). The pair equilibrium cost is the cost of all the paths converges to it after the load balancing process. As a path consists of links, we define path cost as the summation of costs of links which form the corresponding path. We denote the cost of a path P s;d k and cost of one of its link E s;d;k i as λðP s;d k Þ and λðE s;d;k i Þ, respectively. In the second step, the states of paths are identified based on μ (V s , V d ). A path whose cost is greater than μ(V s , V d ) is classified as overloaded. Similarly, a path whose value is less than μ (V s , V d ) is classified as under-loaded. If a path has the same cost as μ(V s , V d ), it is already balanced. In the third step, the algorithm obtains information such as ISS, ESS, and PID regarding each flow in unbalanced paths. Finally, some of these flows are shifted from overloaded paths to under-loaded paths. The controller selects which flow to be shifted in a greedy manner (e.g. heavily to lightly loaded flow). Flow shifting occurs via updating the PID of flows defined in forwarding rules in edge switches.
The computation of pair equilibrium cost takes Θ(n). Sorting flows by size takes Oðf � logf Þ where f is number of flows. Shifting flows to underutilised paths takes Oðn � logf Þ. Thus, time complexity of the active load balancing algorithm is Oðn � ðf � logf þ n � f ÞÞ in the worst case where all paths are active. The controller performs load balancing for all pairs. Therefore, time complexity for overall network load balancing becomes OðB � n � ðf � logf þ n � f ÞÞ.

| Active capacity resizing
Traffic load of some active paths may be higher than other active paths even when load balancing exists. This is because pair based load balancing takes some time to balance overall network load. As paths are actually virtual, and their capacities are re-sizeable. Dynamically adjusting active path capacities prevents the network from congestion and increases resource utilization. Note that, this process is performed on PSDB, not in data plane. For this reason, the controller does not explicitly alter rule of paths on switches.
As aforementioned, neighbour paths may share physical links. A physical link is split to one or more virtual links where each of which serves to a different path of different pairs. We Although the computation of new path capacity is simple, the controller iterates over all virtual links and paths twice. Thus, the time complexity of the resizing algorithm becomes Oðð|E| � |P|Þ 2 Þ. This indicates that resizing operation is a heavy process. For this reason, PRM does not perform resizing frequently unless the overall network is heavily loaded (e.g. 80% of its total capacity). In a heavier loaded network (e.g. 95% of its total capacity), the controller performs consecutive resizing operations because the threshold is exceeded even after the resizing operation. This infers that active capacity resizing must be avoided in such cases. To handle this problem, the controller stops resizing if the number of consecutive resizing operations in a predefined time interval exceeds a threshold (e.g. 3). In this case, PRM backs off so that ESM takes over to expand the capacity, if possible. Therefore, PRM and ESM work in harmony to increase resource utilization. Procedure 3: Active capacity resizing algorithm

| EXPERIMENTAL RESULTS
In this section, we present the experimental results of our model. The proposed model is implemented using Floodlight [29] as a controller, Mininet [30] as a network emulator, Open vSwitch [31] as SDN capable switch, TG [32] as a traffic generator, and OpenFlow [33] as a signalling protocol. Several scenarios have been created and various tests have been carried out in order to evaluate the proposed model using different topologies and traffic tracks [8].
The test scenarios are performed to analyse the effect of path number and traffic load on energy-saving performance, the effect of node connectivity on energy-saving performance, and the comparison of the proposed model with existing work in terms of energy saving.

| Effects of path number per pair and traffic load on energy saving performance
In this test case, we have investigated the energy-saving performance of the proposed model for a various number of paths per pair under light, moderate, and heavy traffic loads. Path number per pair is 2 (P2) and 5 (P5). Net-M is chosen as the test topology. There are three test runs. During each run, active link and switch ratios are recorded, and the average of each run is calculated. Figures 2 and 3 illustrate energy-saving performance of P2 and P5 in terms of link and switch.
We have observed that active link ratio drops down to around 50%-60% in P2 for all traffic loads as illustrated in Figure 2. This experiment results in lower power consumption in P5 for light and moderate traffic loads since the active link ratio drops down to around 35%-45% at the end of the experiment. Note that, the energy-saving performance of both P2 and P5 is the closest to each other under heavy traffic load because the controller needs more resources to carry the heavy load. Also, notice that the active link ratio of P2 decreases to around 60%-70% in almost 50 s compared to P5. This ratio does not change much since then. The reason is that the controller has less choice in terms of path number per pair. However, the energy-saving performance of P5 is better after 300 s. The difference between the active switch ratio of P2 and P5 is much more apparent as shown in Figure 3 for all traffic loads. P2 shows poorer performance even under light traffic load. In case of heavy traffic load, the controller fails to make any switch passive. However, this is not the case for P5. The active switch ratio of P5 drops down to around 60% for light and moderate traffic loads. Although its energy-saving performance under heavy traffic load is not as good as others, the controller successfully makes a few amount of the switches passive.

| Effect of connectivity on energy saving performance
In this test case, we have investigated the energy-saving performance of the proposed model for different topologies. The average degree of connectivity for NSFNET, Net-M, and Net-L are 3, 6, and 3.53, respectively. Path number per pair is two, and a moderate traffic load is applied. There are three test runs. During each run, active link and switch ratios are recorded, and the average of each run is calculated.
We have observed that the degree of connectivity does not affect the energy-saving performance of the proposed model on links as illustrated in Figure 4. The active link ratio in all topologies is close to each other. On the other hand, this is not the case for switches. NSFNET has the best performance among other topologies because paths in this topology have more common links and switches. Net-L has the worst performance among others because paths in Net-L are more diverse than others due to the high connectivity. This test case shows that the establishment of paths affects the energy-saving performance of the proposed model.

| Comparison of energy saving performance
In this test case, we have investigated the energy-saving performance of the proposed model by comparing it with the approach defined in Reference [16]. In our model, as aforementioned, we define path energy consumption as the multiplication of PNR and PDV. In Reference [16], authors define energy consumption based on link utilization and a congestion threshold, that is 80%. We call this model the CARE model. In this test case, we run EPMA both for the proposed method and CARE model. Path number per pair is 2 (P2) and 5 (P5). Net-M is chosen as the test topology. Moderate traffic is applied in all three test runs. During each run, active link and switch ratios are recorded and the average of each run is calculated. Figures 5 and 6 illustrate energy-saving performance of the proposed and CARE models in terms of link and switch.
On one hand, the two models have almost the same performance for P2 in terms of active link ratio case as depicted in Figure 5. On the other hand, the proposed model outperforms the CARE model in the P5 case. Similarly, the results of the active switch ratio test resemble active link ratio test. The two models have almost the same performance for P2, and the proposed model outperforms the CARE model for P5.

| Comparison of load balancing performance
In this test case, we have investigated the load balancing performance of the proposed model by comparing it with the previous model SRRM [8]. In this model, load balancing is performed among active paths. Path number per pair is 2 (P2) and 5 (P5). Net-M is chosen as the test topology. Moderate traffic is applied in all three test runs. During each run, load rates are recorded and the average of each run is calculated. Figure 7 illustrates load balancing performance of both models.

ÇELENLIOĞLU AND MANTAR
In both P2 and P5, the utilization of links is higher in ERM compared to SRRM. This is because the same load is distributed over less capacity due to passive paths in ERM.
Additionally, utilization increases in line with energysaving. In previous test cases, we have observed that energysaving performance increases conforming to path number per pair. Hence, load utilization of P2 7a is lower than P5 7b. The trade-off between energy saving and load balancing indicates that high energy saving may cause over-utilization. Thus, this may result in significant network congestion.
Finally, this test case shows that an adequate path number per pair and PDV selection allows efficient usage of network resources in terms of both energy saving and resource utilization.

| CONCLUSION AND FUTURE WORKS
To put in a nutshell, this study proposes an ERM for SDNbased SPNs. In the proposed model, multi-paths are preestablished between each EP. This turns a complex physical network into a simple virtual one. The controller performs energy-saving, load balancing, and capacity updates based on this network. To save energy, the controller aggregates flows that belong to an EP into less number of paths, while taking neighbour paths and load level into account. Pair based power saving converges to global power saving in time. Apart from energy saving, the controller also performs adaptive load balancing by equalising path costs of individual pairs and adjusts path capacities based on network congestion even among active paths for further resource utilization.
In future work, we aim to increase the energy-saving performance of the proposed model using deep learning techniques. Additionally, we also aim to increase load balancing performance by estimating incoming traffic.