Energy efficient temporal load aware resource allocation in cloud computing datacenters

Abstract

Cloud computing datacenters consume huge amounts of energy, which has high cost and large environmental impact. There has been significant amount of research on dynamic power management, which shuts down unutilized equipment in a datacenter to reduce energy consumption. The main consumers of power in a datacenter are servers, communications network and the cooling system. Optimization of power in a datacenter is a difficult problem because of server resource constraints, network topology and bandwidth constraints, cost of VM migration, the heterogeneity of workloads and the servers. The arrival of new jobs and departure of completed jobs also create workload heterogeneity in time. As a result, most of the previous research has concentrated on partial optimization of power consumption, which optimizes either server and/or network power consumption through placement of VMs. Temporal load aware optimization, minimization of power consumption as a function of time has vastly been studied. When optimization also included migration, then solution had been divided into two steps, in the first step optimization of server and/or network power consumption is performed and in the second step migration of VMs has been taken care of, which is not an optimal solution. In this work, we develop joint optimization of power consumption of servers, network communications and cost of migration with workload and server heterogeneity subject to resource and bandwidth constraints through VM placement. Optimization results in an integer quadratic program (IQP) with linear/quadratic constraints in number of VMs assigned to a job on a server. IQP can only be solved for very small size systems, however, we have been able to decompose IQP to master and pricing sub-problems which may be solved through column generation technique for systems with larger sizes. Then, we have extended the optimization to manage temporal heterogeneity of the workload. It is assumed that time-axis is slotted and at the end of each slot jobs makes probabilistic complete/partial release of the VMs that they are holding and there will also be new job arrivals according to a Poisson process. The system will perform re-optimization of power consumption at the end of each slot that also includes the cost of VM migration. In the re-optimization, VMs of unfinished jobs may experience migration while new jobs are assigned VMs. We have obtained numerical results for optimal power consumption for the system as well as its power consumption due to two heuristic VM assignment algorithms. The results show optimization achieves significant power savings compared to the heuristic algorithms. We believe that our work advances state-of-the art in dynamic power management of datacenters and the results will be helpful to cloud service providers in achieving energy saving.

Introduction

The datacenters have been growing exponentially and together with that their power consumption. The energy consumption results in high operational cost and large impact on the environment. It is expected that the electricity demand for datacenters to rise more than 66% over the period 2011–2035 [1]. As a result, there has been significant research on how to reduce power consumption of the datacenters. The main consumers of power in a datacenter are servers, communications network and the cooling system. It has been determined that an idle server consumes about 70% of its peak power [2]. Dynamic power management together with server consolidation has been used to reduce power consumption by temporarily shutting down servers when they are not required. Server consolidation refers to migration of VMs to as few servers as possible so as to prevent underutilization of the servers. However, server consolidation is challenging because energy cost of migration and, if not carefully done, network communications cost may rise. Server consolidation may result jobs being assigned VMs from multiple servers, which may increase communication traffic between VMs. Thus it is important that optimization of power consumption includes servers, network communications and cost of migration. It has been determined that network accounts for at least 20% of the energy consumption of a cloud computing center and it may rise upto 50% under light job loading, which is typical of the data centers [3]. Since dynamic power management turns off the idle servers, it also reduces power consumption of the cooling system.

The optimization of power consumption also needs to take into account heterogeneity of the workloads and servers. Cloud workloads often have very large variations in their resource requirements, arrival rates and execution times. Cloud centers also have heterogeneity in their servers. In time, datacenters update the configuration of their resources and upgrade the processing capabilities, memory and storage spaces. They also construct new platforms based on the new high performance servers while the older servers are still operational. The heterogeneity of both servers and workloads increases complexity of the optimization of power consumption.

In this paper, we developed joint optimization of power consumptions of the servers, network communications and cost of VM migration with workload and server heterogeneity subject to server resource and network bandwidth constraints. We assumed a hierarchical two-tier datacenter network, though the work can be easily extended to higher tier networks. Optimization results in an integer quadratic program (IQP) with linear and quadratic constraints in number of VMs assigned to a server. This IQP problem is NP-hard [4] and it can only be solved for very small size systems. Due to similarity between our optimization problem and cutting stock problem, we utilized column generation (CG) technique to solve this optimization problem for larger systems. Then, we have extended our solution to handle temporal heterogeneity of the workload due to arrival and departure of the jobs. We assumed that the time-axis is slotted and at the end of each slot jobs are completed either partially or fully and new jobs arrive to the system according to a Poisson process during each time slot. In the partial completion a job releases each of its VMs according to independent Bernoulli trials, while in full completion each job departs from the system according to independent Bernoulli trials. Thus at the end of each slot, the workload load of the system consists of new arriving jobs during the present slot and unfinished jobs from the previous slots. We determine new VM placement by solving this optimization problem that also allows migration of the VMs of unfinished jobs in the system. VMs migrate if the energy savings outweigh cost of the migration. Management of VM migration requires addition of new constraints to the optimization. The main contributions of our work is as follows,

• It formulates joint optimization of server, network and migration power consumption with bandwidth constraints for a given network topology. It performs power optimization and VM migration simultaneously. The optimization problem is expressed as an IQP with quadratic constraints, which can only be solved for very small size systems.

• We have been able to cast this optimization problem into an integer linear programming (ILP), which may be solved through column generation technique for larger size systems. It appears that this is the first application of the column generation technique to the solution of the optimization of power consumption problem in cloud computing systems.

• The work incorporates temporal variation of the workload to the optimization, which allows general arrival and departure of the jobs as a function of discrete-time. This enables re-optimization of the power consumption at the discrete-time instants.

The remainder of this paper is organized as follows:Related work is presented in section 2 and system model in section 3. Section 4 presents IQP modeling of the optimization problem and the section after CG modeling of the problem. Section 6 develops the probabilistic extension of the model. Section 7 presents the temporal load aware formulation of the optimization problem. Section 8 discusses optimization structure and complexity mitigation and the section following that presents numerical results regarding the analysis in the paper. Section 9 discuss the details of assumptions made in this research. Finally, section 10 presents conclusions of the paper.

Related work

In this section, we will present a survey of the related work on the dynamic power management in cloud computing centers. The previous work on dynamic power management may be classified into two as with or without power optimization and the first case may be further subdivided into two depending on whether or not optimization is joint over the servers and network power consumption. Classification may also include other parameters such as workload and server heterogeneity awareness and VM migration. Almost all of the previous works present heuristics rather than solving the optimization problem due to its complexity and then, they perform simulation to determine accuracy of the proposed heuristic.

First, we describe the previous work on dynamic power management without power optimization, which simply turns off idle servers to conserve power consumption. In [5], the effectiveness of dynamic power management in data centers had been investigated using M/M/k queuing model with matrix analytic technique. In [6], this analysis had been extended to the heterogeneous workload case.

Next we explain the previous work on dynamic power management with optimization of network power consumption, which is also referred to as traffic aware VM placement. In [7], an ad-hoc framework has been proposed which minimizes energy consumption of the datacenter network. The framework consists of two steps, and it assumes that the traffic patterns of the jobs are known. In the first step, VM assignment is done in a manner that the traffic in the network is reduced. In the second step, energy-efficient routing of the traffic is carried out that minimizes the number of active switches. In [8], it has been observed from real datacenter network traces that traffic demands of different flows do not peak at exactly the same time. As a result, [8] proposed monitoring of the traffic flows in the network and their consolidation into a small subset of links and switches periodically and shutting down of unutilized switches for energy saving.

Next, we describe previous work on dynamic power management with joint optimization of server and network power consumption. In [4], VM placement problem taking into account server operation and network communication costs had been studied. There is a trade-off between physical machine (PM) cost and network cost where the PM cost is minimized if minimum number of servers is active. However, this may result in jobs being assigned VMs from multiple servers, which increases the network cost. The work proposes an algorithm minimizing the network-cost with fixed PM-cost. The proposed algorithm doesnot consider resource constraints of the servers, network topology and bandwidth constraints of the links. In [9] also VM placement minimizing power consumption has been studied. The work considers both server and communications network power consumptions as well as bandwidth constraints of the links. Server power consumption is assumed to be function of CPU operating frequency. Network infrastructure is assumed to have a hierarchical tree topology and following the optimization, idle servers and switches are turned off. They prove that job load of the servers should be balanced to achieve minimum server power consumption. Starting from this result, they propose a heuristic to assign the VMs to servers, which assigns VMs with high communication requirements among them to the same server. The work assumes that servers are homogeneous and doesnot consider resource constraints of the servers. The joint server and network power consumption optimization has also been studied in [10]. They proposed a unified model that combines server and network optimization by converting the VM assignment to a routing problem. However, optimization problem hasnot been solved due to its complexity, and instead the network is divided into clusters, which are optimized in parallel. The assignment of VMs to the servers and flows to the links in clusters are performed using a heuristic. The [11] also studied the joint optimization of server and network power consumption. They formulated the problem as an integer programming problem, proved that it is NP-hard and then proposed two greedy algorithms for VM scheduling.

Next, we explain previous work on heterogeneity aware dynamic power management. As mentioned earlier, due to inevitable platform upgrades or enhanced hardware resources, cloud platforms gradually become heterogeneous over time, which makes the VM placement problem more complex. In [12], the impact of hardware heterogeneity on the performance of public clouds had been investigated. During a two-year period, the activities of datacenters (DCs) are measured to establish some useful performance benchmarks that might affect the dynamic resource allocation in cloud DCs. Then these benchmarks, such as for CPU performance and network communication overhead, are utilized to evaluate the impact of heterogeneity on the performance of cloud computing centers. In [13] also heterogeneity of workloads and PMs have been considered. According to their resource demands and performance requirements jobs have been divided into classes, and similarly servers have been grouped based on their platform ID and capacities for different resources. Then, heterogeneity aware resource monitoring and management system dubbed “Harmony” was proposed to perform dynamic capacity provisioning that minimizes the total energy consumption and scheduling delay considering heterogeneity as well as reconfiguration costs. The work assumes that a job will always be placed on a single server, as a result the optimization doesnot include network communications cost.

Next we describe the previous work on VM migration aware dynamic power management. In [14], an algorithm named as Peer VM Aggregation (PVA) has been proposed to migrate VMs of a job with high communication demands to the same server in order to reduce network utilization and power consumption. Simulation results show that average network utilization is reduced by %25. In [2], a two stage VM placement algorithm minimizing power consumption with migration has been proposed. In the first stage, VM placement is determined by solving a bin packing problem that minimizes power consumption. In the second stage, VM migration is applied at job departure points from the system that adapts the VM placement according to the released resources. Both stages of the problem are formulated as mixed integer linear programming (MILP) problem. While the work includes resource constraints of the servers in the optimization, but not network communications cost.

The work in this paper combines several of the above optimization problems together, therefore its results are more comprehensive and reliable. Our work jointly optimizes power consumption of servers and communications network and it includes both workload and server heterogeneity, resource constraints of the servers, network topology and link bandwidth constraints. Our work allows optimization to be done at discrete-time instants as the time evolves and some jobs depart and new ones arrive. It also models the VM migration and its cost, which enables adjustment of VM placement that re-optimizes power consumption under the new workload. In the previous work, optimization involving migration was performed in two steps, the first step performing VM placement that minimizes power consumption and the second step performing individual VM migration if it is cost effective. Clearly, this is not optimal because of partitioning of the problem into two separate sub-problems and piecemeal migration. Our work also includes the time-dimension in the optimization, which is absent from the previous work.

System model

In this section, we will present model of the system under consideration for optimization of power consumption through placement of VMs for the jobs. A datacenter consists of servers and communications network that provides connectivity among the servers and they are the main consumers of power in the system. Power consumption of a datacenter depends on its architecture, and in this work, we assume a hierarchical architecture, which is one of the commonly used topologies in the datacenters. It is assumed that the datacenter consists of a collection of Performance Optimized modular Data Centers (PoD). Each PoD consists of a number of racks and each rack contains a collection of servers. In Fig. 1, we show a typical two-tier datacenter network [15, 16], which has servers housed in a rack connected to a Top-Of-Rack (TOR) switch. The TOR switch provides connectivity among the servers of a rack and also connects the rack to the Core Switch (CS) of its host PoD. Core switches depending on the datacenter topology such as clique or fat-tree [15] may have different types of connectivity that provides varying amounts of bandwidths for communications among the PoDs. In this work, we assume that connectivity of core switches has the mesh topology.

Fig. 1

Hierarchical Datacenter Architecture

The main activities resulting in power consumption are processing of the jobs by the servers and the communications between the servers. A job may be served by multiple VMs, which may be located on different servers. A job will have communications demand, when it is assigned VMs on different servers. The magnitude of this demand between two servers will be assumed to be proportional to the product of the number of VMs assigned to that job on the two servers. We assume that servers will be in one of two states, either on or off state. A server will be in the on state if it has at least one VM assigned to one of the jobs and otherwise it will be in the off state. An on server will consume constant power and an off server zero power.

We include server and workload heterogeneity in the model. We assume that a datacenter has T types of servers, where each server type is determined by the amount of different types of resources that it contains. A server type may have K different types of resources such as bandwidth, storage, CPU and memory. The amount of resources owned by a server of each type are given by a unique resource vector. We let M_t denote number of type t servers in the datacenter, and m’th type t server as m_t ∈ {1_t, …, M_t} with t ∈ {1, …, T}. Power consumption of an on type t server will be denoted by Q_t. We assume that a server may have R different VM configurations. Each VM configuration is determined by the amount of different types of resources that it is allocated. We let ${i}_r^k$ denote the type k resource requirement of a type r VM. We assume that there are H types of jobs, where each job type requires a random number of VMs from a group of VM types. Each job type has a different mix of VM types and a geometrically distributed service time in number of slots with a different parameter. We let N_h denote number of type h jobs in the datacenter, h ∈ {1, …, H}, and ${v}_{n_h}^r$ denote the number of type r VMs that job n_h requires, n_h ∈ {1_h, …, N_h}. Let also N denote total number of jobs in the datacenter, then, $N=\sum \limits_{h=1}^H{N}_h$.

The optimization problem also includes communication network bandwidth constraints to prevent traffic congestion. We assume that, there is no communication congestion between the servers located in the same rack because they are connected to their ToR switch with high capacity links. The communications congestion may occur either in the (TORS-CS) links or in PoD links (CS-CS). We assume that a ToR switch will be turned off if none of the servers in that rack are being utilized. Similarly CS in a PoD will be turned off if all the servers connected to its racks are off. We note that an on switch consumes a constant power plus load dependent variable power; the former will be referred to as static and the latter as dynamic power respectively. We will let ${PS}_{\ell, e}^{ToRS},{PS}_{\ell}^{CS}$ denote static power consumption of a ToR switch on the e’th rack of PoD ℓ, and CS switch in PoD ℓ respectively. Similarly, we will let ${PD}_{\ell, e}^{ToRS},{PD}_{\ell}^{CS}$ denote dynamic power consumption of these switches for per bit transmission rate. We also let PW_NIC denote the dynamic power consumption at the network interface card (NIC) of a server for per bit transmission rate.

The notation introduced in the above as well as others for this optimization problem has been summarized in Table 1. From this table,

$$ {M}_t={\sum}_{\ell =1}^L{\sum}_{e=1}^{d_{\ell }}{M}_{\ell, e}^t $$

(1)

$$ {M}_{\ell, e}={\sum}_{t=1}^T{M}_{\ell, e}^t $$

(2)

$$ {a}_{\ell, e}=\left\{{1}_{\ell, e},\dots, {m}_{\ell, e},..{M}_{\ell, e}\right\} $$

(3)

where m_{ℓ, e} denotes the m’th server on the e’th rack of PoD ℓ. The total power consumption of the datacenter will be minimized if the job load is served by minimum number of servers and each job is assigned VMs from as few servers as possible. In the next two sections, we will model the optimization problem first using IQP and then CG technique.

Table 1 Summary of notation

Modeling of the optimization problem with Integer Quadratic Programming (IQP)

In this section, we will model optimization problem of the system described in the previous section as an integer quadratic programming (IQP). The power consumption of a datacenter consists of static and dynamic power consumptions of the switches, dynamic power consumption of the interface cards and power consumption of the servers.

We first determine the dynamic power consumption due to communications of two VMs. Let ${P}_{m_t,{m}_{t^{\prime}}^{\prime}}^{n_h}$ denote total dynamic communication power consumption between two VMs located on servers m_t, ${m}_{t^{\prime}}^{\prime }$ and serving job n_h, then it is given by,

$$ {P}_{m_t,{m}_{t^{\prime}}^{\prime}}^{n_h}=\left\{\begin{array}{c}0, if\ {m}_t={m}_{t^{\prime}}^{\prime}\\ {}\ \\ {}{\vartheta}_{n_h}\left(2{PW}_{NIC}+{PD}_{\ell, e}^{ToRS}\right), if\ {m}_t,{m}_{t^{\prime}}^{\prime}\in {a}_{\ell, e},{m}_t\ne {m}_{t^{\prime}}^{\prime }\ \\ {}\ \\ {}{\vartheta}_{n_h}\left(2{PW}_{NIC}+{PD}_{\ell, e}^{ToRS}+{PD}_{\ell}^{CS}+{PD}_{\ell, {e}^{\prime}}^{ToRS}\right), if\ {m}_t\in {a}_{\ell, e},{m}_{t^{\prime}}^{\prime}\in {a}_{\ell, {e}^{\prime }},e\ne {e}^{\prime }\ \\ {}\ \\ {}{\vartheta}_{n_h}\left(2{PW}_{NIC}+{PD}_{\ell, e}^{ToRS}+{PD}_{\ell}^{CS}+{PD}_{\ell^{\prime}}^{CS}+{PD}_{\ell^{\prime },{e}^{\prime}}^{ToRS}\right), if\ {m}_t\in {a}_{\ell, e},{m}_{t^{\prime}}^{\prime}\in {a}_{\ell^{\prime },{e}^{\prime }},\ell \ne {\ell}^{\prime }\ \\ {}\ \end{array}\right\} $$

(4)

In the above, it has been assumed that communication power consumption between two VMs assigned to a job depends on the type of job but not on the types of VMs. As may be seen, this power depends on the location of the servers housing the VMs and on the data rate, which depends on the job type.

As defined in Table 1., the scheduling variable ${x}_{r,{n}_h}^{m_t}$ denotes number of type r VMs on the server assigned to serve job n_h and connectivity variable ${\overset{\sim }{x}}_{n_h}^{m_t}$ denotes total number of VMs assigned to job n_h on the m^th type t server assigned to serve job n_h, where, ${\overset{\sim }{x}}_{n_h}^{m_t}=\sum \limits_{r=1}^R{x}_{r,{n}_h}^{m_t}$. We would like to determine optimal values of the scheduling variables ${x}_{r,{n}_h}^{m_t}$that minimizes the datacenter power consumption. Next let us define the binary variables, ${y}_{m_t}$ to denote on or off status of m^th type t server, η_{ℓ, e} status of the ToR switch serving to rack e on PoD ℓ as active or not, and ξ_ℓ status of the CS serving PoD ℓ as active or not. Then from the notation introduced in Table 1.,

$$ {y}_{m_t}=\left\{\begin{array}{c}1\kern3em {\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\overset{\sim }{x}}_{n_h}^{m_t}>0\ \\ {}0\kern3em {\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\overset{\sim }{x}}_{n_h}^{m_t}=0\ \end{array}\right. $$

(5)

$$ {\eta}_{\ell, e}=\left\{\begin{array}{c}\ 1\kern3em {\sum}_{t=1}^T{\sum}_{m_t\in {a}_{\ell, e}}{y}_{m_t}>0\ \\ {}0\kern3em {\sum}_{t=1}^T{\sum}_{m_t\in {a}_{\ell, e}}{y}_{m_t}=0\ \end{array}\right. $$

(6)

$$ {\xi}_{\mathrm{\ell}}=\left\{\begin{array}{cc}1& {\sum \limits}_{e=1}^{d_{\mathrm{\ell}}}{\eta}_{\mathrm{\ell},e}>0\\ {}0& \mathrm{otherwise}\end{array}\right.\kern4em \forall e\in \left\{1,\dots, {d}_{\mathrm{\ell}}\right\},\forall \mathrm{\ell}\in \left\{1,\dots, L\right\}. $$

(7)

Then, the optimization problem for minimization of total power consumption is given below,

$$ \mathit{\operatorname{Min}}\left[{\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\sum}_{t=1}^T{\sum}_{t^{\prime }=1}^T{\sum}_{m_t={1}_t}^{M_t}{\sum}_{m_{t^{\prime }}={1}_{t^{\prime}}}^{M_t}\ {P}_{m_t,{m}_{t^{\prime}}}^{n_h}\left({\overset{\sim }{x}}_{n_h}^{m_t}{\overset{\sim }{x}}_{n_h}^{m_{t^{\prime}}^{\prime }}\ \right)+{\sum}_{\ell =1}^L\left({\xi}_{\ell }{PS}_{CS}^{\ell }+{\sum}_{e=1}^{d_{\ell }}{\eta}_{\ell, e}{PS}_{ToR}^{\ell, e}\right)+{\sum}_{t=1}^T{Q}_t{\sum}_{m_t=1}^{M_t}{y}_{m_t}\right] $$

(8)

ST. (5), (6), (7),

$$ {\overset{\sim }{x}}_{n_h}^{m_t}={\sum}_{r=1}^R{x}_{r,{n}_h}^{m_t}\forall {n}_h\in \left\{1,\dots, {N}_h\right\},{m}_t\in \left\{{1}_t,\dots, {M}_t\right\} $$

(9)

$$ {\sum}_{t=1}^T{\sum}_{m_t={1}_t}^{M_t}{x}_{r,{n}_h}^{m_t}\ge {v}_{n_h}^r\forall r\in \left\{1,\dots, R\right\},{n}_h\in \left\{{1}_h,\dots, {N}_h\right\},h\in \left\{1,\dots, H\right\} $$

(10)

$$ {\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\sum}_{r=1}^R{x}_{r,{n}_h}^{m_t}{i}_r^k\le {c}_t^k\forall k\in \left\{1,\dots, K\right\},{m}_t\in \left\{{1}_t,\dots, {M}_t\right\} $$

(11)

$$ {\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\vartheta}_{n_h}{\sum}_{m_t\in {a}_{\ell, e}}\left[{\sum}_{\ell^{\prime }=1}^L{\sum}_{e^{\prime }=1}^{d_{\ell^{\prime }}}{\sum}_{m_{t^{\prime}}^{\prime}\in {a}_{\ell^{\prime },{e}^{\prime }}}\left({\overset{\sim }{x}}_{n_h}^{m_t}{\overset{\sim }{x}}_{n_h}^{m_{t^{\prime}}^{\prime }}\ \right)-{\sum}_{m_{t^{\prime}}^{\prime}\in {a}_{\ell, e}}\left({\overset{\sim }{x}}_{n_h}^{m_t}{\overset{\sim }{x}}_{n_h}^{m_{t^{\prime}}^{\prime }}\ \right)\right]\le {S}_{\ell, e}\forall e\in \left\{1,\dots, {d}_{\ell}\right\},\forall \ell \in \left\{1\dots L\right\} $$

(12)

$$ {\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\vartheta}_{n_h}{\sum}_{e=1}^{d_{\ell }}{\sum}_{m_t\in {a}_{\ell, e}}{\sum}_{e^{\prime }=1}^{d_{\ell^{\prime }}}{\sum}_{m_{t^{\prime}}^{\prime}\in {a}_{\ell^{\prime },{e}^{\prime }}}\left({\overset{\sim }{x}}_{n_h}^{m_t}{\overset{\sim }{x}}_{n_h}^{m_{t^{\prime}}^{\prime }}\ \right)\le {CP}_{\ell, {\ell}^{\prime }}\forall \ell, {\ell}^{\prime}\in \left\{1\dots L\right\},\ell \ne {\ell}^{\prime } $$

(13)

We note that m_t ∈ {1_t, …, M_t} stands for ∀t ∈ {1, …, T}.

In the objective function, the first term corresponds to the total dynamic communication power consumption in the datacenter. Second term represents the static part of communication power consumption and finally the last term corresponds to the power consumption of the servers. Constraint group (10) ensures that VM requirements of each job are satisfied and group (11) guarantees that resource demands of jobs scheduled on a server do not exceed that server’s resource capacities. The constraints (12) and (13) ensure that bandwidth demands do not violate the capacities of TORs to CS and CS to CS links respectively. In these constraints, as defined in Table 1., S_{ℓ, e}, ${CP}_{\ell, {\ell}^{\prime }}$ denote capacities of TORS to CS and CS to CS links respectively.

From the Eqs. (8–13), the optimization problem is in the form of Integer Quadratic Programming (IQP) in the scheduling variables ${x}_{r,{n}_h}^{m_t}$. However, from the definitions of the variables ${y}_{m_t}$, η_{ℓ, e}, ξ_ℓ given in Eqs. (5–7), the IQP problem has other nonlinear constraints. Next, we would like to convert the nonlinearities due to ${y}_{m_t}$, η_{ℓ, e}, ξ_ℓ into a form with linear constraints, which will make the problem simpler. This can be achieved by replacing each of the equations in (5–7) by a pair of constraints as follows,

$$ {\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\overset{\sim }{x}}_{n_h}^{m_t}-{y}_{m_t}\ge 0 $$

(14)

$$ \theta {y}_{m_t}-{\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\overset{\sim }{x}}_{n_h}^{m_t}\ge 0 $$

(15)

$$ {\sum}_{t=1}^T{\sum}_{m_t\in {a}_{\ell, e}}{y}_{m_t}-{\eta}_{\ell, e}\ge 0\forall e\in \left\{1,\dots, {d}_{\ell}\right\},\forall \ell \in \left\{1\dots L\right\} $$

(16)

$$ \theta {\eta}_{\ell, e}-{\sum}_{t=1}^T{\sum}_{m_t\in {a}_{\ell, e}}{y}_{m_t}\ge 0\forall e\in \left\{1,\dots, {d}_{\ell}\right\},\forall \ell \in \left\{1\dots L\right\} $$

(17)

$$ {\sum}_{e=1}^{d_{\ell }}{\eta}_{\ell, e}-{\xi}_{\ell}\ge 0\forall \ell \in \left\{1\dots L\right\} $$

(18)

$$ \theta {\xi}_{\ell }-{\sum}_{e=1}^{d_{\ell }}{\eta}_{\ell, e}\ge 0\forall \ell \in \left\{1\dots L\right\} $$

(19)

Thus we replace Eq. (5) with inequalities (14, 15). Definition in (5) implies that $\sum \limits_{h=1}^H\sum \limits_{n_h={1}_h}^{N_h}{\overset{\sim }{x}}_{n_h}^{m_t}$ = 0 ⇔ ${y}_{m_t}$ = 0 and $\sum \limits_{h=1}^H\sum \limits_{n_h={1}_h}^{N_h}{\overset{\sim }{x}}_{n_h}^{m_t}$ > 0 ⇔ ${\mathrm{y}}_{m_t}$ = 1. From these observations, the inequalities in (14, 15) follow, where θ denotes a very large integer number. The correspondence between (6) and (16, 17) and between (7) and (18, 19) may be established similarly. In the remainder of the paper, these pairs of constraints will be referred to as “positive integer to binary linear conversion constraints” (IBLC).

As a result, our optimization problem has been converted to IQP with linear and quadratic constraints given by (8, 19). This optimization problem is NP hard and it can only be solved for very small size systems using the branch and bound technique [17].

Modeling of the optimization problem with column generation (CG) method

In this section, we will apply column generation technique to obtain another solution to our problem, which may be used with larger size systems. This technique originally had been applied to cutting-stock problem, which consists of cutting a set of available stock lengths to meet customer orders for items in required lengths and quantities with the objective of minimizing the wasted material [18]. Distinct combination of items in length and quantities cut from a stock length is called a pattern. In column generation approach, the optimization problem is divided into two types of sub-problems referred to as restricted master and pricing problems [18]. The Restricted master problem (RMP) determines if the explored patterns satisfy the job demand constraints. The pricing problem finds a new pattern to feed the RMP. The objective function of the pricing problem is in fact the reduced cost coefficient of the RMP. The RMP and pricing problems collaborate until reduced cost coefficients (objectives) of the pricing problems become negative indicating optimal solution has been reached. In our problem, there will be T pricing problems, one for each server type. RMP is in the form of integer linear program (ILP) and pricing problems are combinatorial optimization problems. RMP is solved using continuous relaxation, which at the end requires integer rounding of the results. There has been some work for application of column generation technique in quadratic programming [19,20,21].

In relation to the cutting-stock problem, server types and jobs are similar to stock and item lengths respectively, however server types and jobs have multiple resource constraints, while stocks and items have length as the only limit factor. Further, we have a complicated objective function compared to cutting stock problem. Let us define a pattern as a distinct combination of the number of VMs from each type of VMs that a server can accommodate. Let j_t denote such a pattern and J_t total number of patterns available for a type t server, then j_t ∈ {1_t, …, J_t}. At the end of the solution, each active server is assigned one of these patterns. The new introduced notation may also been found in Table 1. Let ${x}_{r,{n}_h}^{j_t}$ denote number of type r VMs that has been assigned to job n_h in a server with pattern j_t and similarly, ${\overset{\sim }{x}}_{n_h}^{j_t}$ denote total number of VMs assigned to job n_h at a server with pattern j_t. Then, we have the following equality between the two variables,

$$ {\overset{\sim }{x}}_{n_h}^{j_t}={\sum}_{r=1}^R{x}_{r,{n}_h}^{j_t} $$

The column vector ${\boldsymbol{a}}_{{\boldsymbol{j}}_{\boldsymbol{t}}}={\left({x}_{1,{n}_1}^{j_t},..,{x}_{1,{n}_H}^{j_t};..;{x}_{r,{n}_1}^{j_t},..,{x}_{r,{n}_H}^{j_t};..;{x}_{R,{n}_1}^{j_t},..,{x}_{R,{n}_H}^{j_t}\right)}^{\boldsymbol{Tr}}$will denote j’th pattern of type t server.

Let us define binary variable ${y}_{m_t}^{j_t}$ to denote whether or not a given type t server has pattern j_t, then,

$$ {y}_{m_t}^{j_t}=\left\{\begin{array}{c}1\kern1.25em \mathrm{if}\ {m}^{\prime}\mathrm{th}\ \mathrm{server}\ \mathrm{of}\ \mathrm{type}\ t\ \mathrm{is}\ \mathrm{active}\ \mathrm{and}\ \mathrm{has}\ {j}_t\ \mathrm{pattern}.\\ {}0\kern2em \mathrm{otherwise}\ \end{array}\right. $$

(20)

Next, let ${m}_{\ell, e}^{j_t}$ denote number of active type t servers with pattern j_t in the e^th rack of PoD ℓ,

$$ {m}_{\ell, e}^{j_t}={\sum}_{m_t\in {a}_{\ell, e}}{y}_{m_t}^{j_t} $$

(21)

Finally, the state of rack e on PoD ℓ as active or not is determined as,

$$ {\eta}_{\ell, e}=\left\{\begin{array}{c}\ 1\kern2em {\sum}_{t=1}^T{\sum}_{j_t={1}_t}^{J_t}{m}_{\ell, e}^{j_t}>0\ \\ {}0\kern6em otherwise\ \end{array}\right. $$

(22)

We note that total dynamic communication power consumption between two VMs located on servers m_t, ${m}_{t^{\prime}}^{\prime }$ and serving job n_h is still given by Eq. (4). Then, the optimization problem for the RMP is given by,

$$ \mathit{\operatorname{Min}}\left[{\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\sum}_{t=1}^T{\sum}_{t^{\prime }=1}^T{\sum}_{m_t={1}_t}^{M_t}{\sum}_{m_{t^{\prime }}={1}_{t^{\prime}}}^{M_t}\ {\sum}_{j_t={1}_t}^{J_t}{\sum}_{j_t\prime ={1}_{t\prime}}^{J_{t\prime }}{P}_{m_t,{m}_{t^{\prime}}}^{n_h}\left({y}_{m_t}^{j_t}{\overset{\sim }{x}}_{n_h}^{j_t}{y}_{m_{t\prime}}^{j_{t\prime }}{\overset{\sim }{x}}_{n_h}^{j_{t^{\prime}}^{\prime }}\ \right)+{\sum}_{\ell =1}^L\left({\xi}_{\ell }{PS}_{CS}^{\ell }+{\sum}_{e=1}^{d_{\ell }}{\eta}_{\ell, e}{PS}_{ToR}^{\ell, e}\right)+{\sum}_{t=1}^T{Q}_t{\sum}_{m_t=1}^{M_t}{\sum}_{j_t={1}_t}^{J_t}{y}_{m_t}^{j_t}\right] $$

(23)

ST. (18), (19)

$$ {\displaystyle \begin{array}{c}{\tilde{x}}_{n_h}^{j_t}={\sum}_{r=1}^R{x}_{r,{n}_h}^{j_t},\kern1em \forall {n}_h\in \left\{1,\dots, {N}_h\right\},{j}_t\in \left\{{1}_t,\dots, {J}_t\right\}\\ {}{m}_{\mathrm{\ell},e}^{j_t}={\sum}_{m_t\in {a}_{\mathrm{\ell},e}}{y}_{m_t}^{j_t}\kern1em \forall {j}_t\in \left\{{1}_t,\dots, {J}_t\right\},t\in \left\{1,\dots, T\right\},\forall e\in \left\{1,\dots, {d}_{\mathrm{\ell}}\right\},\forall \mathrm{\ell}\in \left\{1\dots L\right\}\\ {}{\sum}_{t=1}^T{\sum}_{m_t={1}_t}^{M_t}{\sum}_{j_t={1}_t}^{J_t}{y}_{m_t}^{j_t}{x}_{r,{n}_h}^{j_t}\ge \kern0.5em {v}_{n_h}^r\kern0.75em \forall r\in \left\{1,\dots, R\right\},{n}_h\in \left\{{1}_h,\dots, {N}_h\right\},h\in \left\{1,\dots, H\right\}\end{array}} $$

(24)

$$ {\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\vartheta}_{n_h}{\sum}_{t=1}^T{\sum}_{m_t\in {a}_{\ell, e}}{\sum}_{j_t={1}_t}^{J_t}\left[{\sum}_{\ell^{\prime }=1}^L{\sum}_{e^{\prime }=1}^{d_{\ell^{\prime }}}{\sum}_{t^{\prime }=1}^T{\sum}_{j_{t\prime }={1}_{t\prime}}^{J_{t\prime }}{\sum}_{m_{t^{\prime}}^{\prime}\in {a}_{\ell^{\prime },{e}^{\prime }}}\left({y}_{m_t}^{j_t}{\overset{\sim }{x}}_{n_h}^{j_t}{y}_{m_{t\prime}}^{j_{t\prime }}{\overset{\sim }{x}}_{n_h}^{j_{t^{\prime}}^{\prime }}\ \right)-{\sum}_{m_{t^{\prime}}^{\prime}\in {a}_{\ell, e}}\left({y}_{m_t}^{j_t}{\overset{\sim }{x}}_{n_h}^{j_t}{y}_{m_{t\prime}}^{j_{t\prime }}{\overset{\sim }{x}}_{n_h}^{j_{t^{\prime}}^{\prime }}\ \right)\right]\le {S}_{\ell, e}\forall e\in \left\{1,\dots, {d}_{\ell}\right\},\forall \ell \in \left\{1\dots L\right\} $$

(25)

$$ {\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\vartheta}_{n_h}{\sum}_{e=1}^{d_{\ell }}{\sum}_{t=1}^T{\sum}_{m_t\in {a}_{\ell, e}}{\sum}_{j_t={1}_t}^{J_t}{\sum}_{e^{\prime }=1}^{d_{\ell^{\prime }}}{\sum}_{t^{\prime }=1}^T{\sum}_{m_{t^{\prime}}^{\prime}\in {a}_{\ell^{\prime },{e}^{\prime }}}{\sum}_{j_{t\prime }={1}_{t\prime}}^{J_{t\prime }}\left({y}_{m_t}^{j_t}{\overset{\sim }{x}}_{n_h}^{j_t}{y}_{m_{t\prime}}^{j_{t\prime }}{\overset{\sim }{x}}_{n_h}^{j_{t^{\prime}}^{\prime }}\right)\le {CP}_{\ell, {\ell}^{\prime }}\forall \ell, {\ell}^{\prime}\in \left\{1\dots L\right\},\ell \ne {\ell}^{\prime } $$

(26)

$$ {\sum}_{j_t={1}_t}^{J_t}{m}_{\ell, e}^{j_t}\le {M}_{\ell, e}^t $$

(27)

$$ {\sum}_{t=1}^T{\sum}_{j_t={1}_t}^{J_t}{m}_{\ell, e}^{j_t}-{\eta}_{\ell, e}\ge 0\forall e\in \left\{1,\dots, {d}_{\ell}\right\},\forall \ell \in \left\{1\dots L\right\} $$

(28)

$$ {\theta \eta}_{\ell, e}-{\sum}_{t=1}^T{\sum}_{j_t={1}_t}^{J_t}{m}_{\ell, e}^{j_t}\ge 0\forall e\in \left\{1,\dots, {d}_{\ell}\right\},\forall \ell \in \left\{1\dots L\right\} $$

(29)

We note that in the above optimization problem, scheduling and connectivity variables ${x}_{r,{n}_h}^{j_t},{\overset{\sim }{x}}_{n_h}^{j_t}$ are treated as constants. In the objective function, (23), the first term corresponds to power consumption of the interface cards and dynamic power consumption of active switches due to communication load, second term to static power consumption of active switches and the third term to power consumption of active servers. Constraint (24) ensures satisfaction of the VM requirements of the jobs. The constraints (25) and (26) ensure that bandwidth demands of the jobs do not violate the capacities of the TORS to CS links and CS to CS links respectively. Constraint (27) ensures that the demand for servers donot exceed the available server capacity. Constraints (28) and (29) are IBLC for the variable η_{ℓ, e} defined in (22). However, the variables ${y}_{m_t}^{j_t}$and ${y}_{m_{t^{\prime}}^{\prime}}^{j_{t^{\prime}}^{\prime }}$defined in (20) and their product makes the objective function and the constraints (25, 26) nonlinear. Let us define the following binary variable in order to remove this nonlinearity,

$$ {\varphi}_{m_t,{m}_{t^{\prime}}^{\prime}}^{j_t,{j}_{t^{\prime}}^{\prime }}={y}_{m_t}^{j_t}{y}_{m_{t^{\prime}}^{\prime}}^{j_{t^{\prime}}^{\prime }} $$

(30)

Then,

$$ {\varphi}_{m_t,{m}_{t^{\prime}}^{\prime}}^{j_t,{j}_{t^{\prime}}^{\prime }}=\left\{\begin{array}{c}1\kern2em if\ {y}_{m_t}^{j_t}={y}_{m_{t^{\prime}}^{\prime}}^{j_{t^{\prime}}^{\prime }}=1\\ {}0\kern4.2em otherwise\ \end{array}\right. $$

(31)

The binary multiplication in the above can be linearized through the following constraints,

$$ {\varphi}_{m_t,{m}_{t^{\prime}}^{\prime}}^{j_t,{j}_{t^{\prime}}^{\prime }}\ge {y}_{m_t}^{j_t}+{y}_{m_{t^{\prime}}^{\prime}}^{j_{t^{\prime}}^{\prime }}-1 $$

(32)

$$ {\varphi}_{m_t,{m}_{t^{\prime}}^{\prime}}^{j_t,{j}_{t^{\prime}}^{\prime }}\le {y}_{m_t}^{j_t},{\varphi}_{m_t,{m}_{t^{\prime}}^{\prime}}^{j_t,{j}_{t^{\prime}}^{\prime }}\le {y}_{m_{t^{\prime}}^{\prime}}^{j_{t^{\prime}}^{\prime }}\ {\varphi}_{m_t,{m}_{t^{\prime}}^{\prime}}^{j_t,{j}_{t^{\prime}}^{\prime }}\ge 0 $$

(33)

Thus the constraints (32, 33) need to be added to the above optimization problem given in (23–29).

Next, we present the T pricing problems one for each server type. The pricing problem for server type t attempts to introduce the new pattern ${\overset{\sim }{j}}_t$ to the RMP through solution of the following optimization problem,

$$ \mathit{\operatorname{Max}}\ {\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\sum}_{r=1}^R{u}_{r,{n}_h}^t\left({x}_{r,{n}_h}^{{\overset{\sim }{j}}_t}\right) $$

(34)

$$ ST.{\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}{\sum}_{r=1}^R{x}_{r,{n}_h}^{{\overset{\sim }{j}}_t}{i}_r^k\le {c}_t^k\forall k\in \left\{1,\dots, K\right\} $$

(35)

where ${x}_{r,{n}_h}^{{\overset{\sim }{j}}_t}$ represents number of type r VMs assigned to job n_h by pattern ${\overset{\sim }{j}}_t$. The pricing problem’s objective function is the reduced cost function of the RMP with respect to server type t and ${u}_{r,{n}_h}^t$ coefficients denote the values of the dual variables of the RMP for type t server. Constraint group (35) ensures that resource constraints of the servers are satisfied.

As shown in Fig. 2, in the column generation technique, RMP and pricing problems are solved iteratively. In each iteration, following solution of the pricing problems, the pattern of the server type t with the highest objective function value is introduced to the RMP. The iterations continue, as long as there are reduced cost functions with positive values. The algorithm terminates when all the reduced cost functions become negative and as a result no new pattern is introduced to the RMP. However, the obtained solution corresponds to the continuous relaxation of the problem, and therefore the results need to be rounded into integer values, which will be dealt in a later section.

Fig. 2

Column Generation Implementation of the Optimization

We note that CG gives us a linear solution of the problem, which reduces solution’s complexity but this is achieved at the expense of substantial increase in number of variables and constraints [22].

Probabilistic model

In the previous sections, we have assumed deterministic traffic rates for communications between VMs and constant power consumption for active servers; however, in practice these quantities are random and vary as functions of time. In this section, we extend the optimization problem of the previous sections to a more realistic model, where VM communication rates and server power consumption are considered as random variables.

First, we assume that the data rate between two VMs serving to a type h job, ${\vartheta}_{n_h}$, is a random variable. As a result, bandwidth constraints given in (12, 13) for IQP and (25, 26) for CG become probabilistic. For example (12) and (25) may be expressed as,

$$ \mathit{\Pr}\left({\sum}_{h=1}^H{\vartheta}_{n_h}{\sum}_{n_h={1}_h}^{N_h}{\varPsi}_{\ell, e,{n}_h}>{S}_{\ell, e}\right)\le p $$

(36)

where ${\varPsi}_{\ell, e,{n}_h}$ for IQP and CG are given by,

$$ {\varPsi}_{\ell, e,{n}_h}={\sum}_{m_t\in {a}_{\ell, e}}\left[{\sum}_{\ell^{\prime }=1}^L{\sum}_{e^{\prime }=1}^{d_{\ell^{\prime }}}{\sum}_{m_{t\prime}^{\prime}\in {a}_{\ell^{\prime },{e}^{\prime }}}\left({\overset{\sim }{x}}_{n_h}^{m_t}{\overset{\sim }{x}}_{n_h}^{m_{t^{\prime}}^{\prime }}\ \right)-{\sum}_{m_{t^{\prime}}^{\prime}\in {a}_{\ell, e}}\left({\overset{\sim }{x}}_{n_h}^{m_t}{\overset{\sim }{x}}_{n_h}^{m_{t^{\prime}}^{\prime }}\ \right)\right], for\ IQP $$

(37)

$$ {\Psi}_{\ell, e,{n}_h}={\sum}_{t=1}^T{\sum}_{j_t={1}_t}^{J_t}\left\{\left[{\overset{\sim }{x}}_{n_h}^{j_t}{\sum}_{\ell^{\prime}\in \left\{1,..L\right\},\ell \ne {\ell}^{\prime }}{\sum}_{e^{\prime }=1,}^{d_{\ell^{\prime }}}{\sum}_{t^{\prime }=1}^T{\sum}_{j_{t^{\prime}}^{\prime }={1}_{t^{\prime}}}^{J_{t^{\prime }}}\left({\sum}_{m_t\in {a}_{\ell, e}}{\sum}_{m_{t^{\prime}}^{\prime}\in {a}_{\ell^{\prime },{e}^{\prime }}}{\varphi}_{m_t,{m}_{t^{\prime}}^{\prime}}^{j_t,{j}_{t^{\prime}}^{\prime }}\right){\overset{\sim }{x}}_{n_h}^{j_{t^{\prime}}^{\prime }}\right]+\left[{\overset{\sim }{x}}_{n_h}^{j_t}{\sum}_{e^{\prime}\in {b}_{\ell },{e}^{\prime}\ne e}{\sum}_{t^{\prime }=1}^T{\sum}_{j_{t^{\prime}}^{\prime }={1}_{t^{\prime}}}^{J_{t^{\prime }}}\left({\sum}_{m_t\in {a}_{\ell, e}}{\sum}_{m_{t^{\prime}}^{\prime}\in {a}_{\ell^{\prime },{e}^{\prime }}}{\varphi}_{m_t,{m}_{t^{\prime}}^{\prime}}^{j_t,{j}_{t^{\prime}}^{\prime }}\right){\overset{\sim }{x}}_{n_h}^{j_{t^{\prime}}^{\prime }}\right]\right\},\mathrm{for}\ \mathrm{CG} $$

(38)

In the above the objective is to keep probability of link congestion below a threshold value of p.

As in [23], we assume that total traffic rate follows a Gaussian distribution, which from the Central Limit Theorem remains an accurate model even if the individual flows are non-Gaussian [24, 25]. Next, we assume that mean and standard deviation of ${\vartheta}_{n_h}$ are given by λ_h and σ_h respectively, then the constraint (36) may be expressed as,

$$ {\sum}_{h=1}^H\left({\lambda}_h{\sum}_{n_h={1}_h}^{N_h}{\varPsi}_{\ell, e,{n}_h}\right)+\zeta \sqrt{\sum_{h=1}^H{\sigma_h}^2{\left({\sum}_{n_h={1}_h}^{N_h}{\varPsi}_{\ell, e,{n}_h}\right)}^2}\le {S}_{\ell, e} $$

(39)

where ζ = Φ⁻¹(1 − p) and Φ⁻¹ is the inverse function of the normal CDF. From [25], the LHS may be bounded, which reduces the above constraint to,

$$ {\sum}_{h=1}^H\left[\left({\lambda}_h+\zeta {\sigma}_h\right){\sum}_{n_h={1}_h}^{N_h}{\varPsi}_{\ell, e,{n}_h}\right]\le {S}_{\ell, e} $$

(40)

In the previous sections, we assumed that power consumption of an on type t server is a constant denoted by Q_t. In fact, power consumption is random and depends on processing utility, I/O, load, memory usage etc. Let q_t denote power consumption of a type t server, then from [23], q_t has a general probability distribution, which varies in the range [0.5Q_t, Q_t] with mean and standard deviation denoted by ω_t, δ_t respectively. It is better to avoid high power consumption at the rack level in order to prevent system failure [26]. As a result, we introduce the following probabilistic constraint,

$$ \mathit{\Pr}\left({\sum}_{t=1}^T{q}_t{\sum}_{j_t={1}_t}^{J_t}{m}_{\ell, e}^{j_t}>{PR}_{\ell, e}\right)\le p $$

(41)

where PR_{ℓ, e} denotes the power supply of rack e on PoD ℓ. As before, from the central limit theorem we assume that the total power consumption at the rack level has a Gaussian distribution. Similar to the analysis for Eq. (36), Eq. (41) can be linearized as follows,

$$ {\sum}_{t=1}^T{\omega}_t\ {\sum}_{j_t={1}_t}^{J_t}{m}_{\ell, e}^{j_t}+\zeta {\sum}_{t=1}^T{\delta}_t{\sum}_{j_t={1}_t}^{J_t}{m}_{\ell, e}^{j_t}\le {PR}_{\ell, e} $$

(42)

This completes the extension to a probabilistic model with random server power consumption and link utilization. Thus new optimization problem also includes constraints (42) on rack’s power consumption and on link utilization (25) is replaced by (40).

Temporal load aware VM placement

In this section, we will study VM placement with optimization of power consumption as a function of time, which will also be referred to as dynamic job scheduling. As a result, it will be assumed that time-axis is slotted and VMs are assigned to jobs in units of slot times. We will assume that arrival of jobs to the system is according to a Poisson process, though the analysis is applicable to other arrival processes. The new arriving jobs during the present slot and leftover jobs from the present slot will be scheduled for service in the next slot. We will consider two types of service disciplines, a job either releasing its assigned VMs simultaneously or individually according to Bernoulli trials at the end of each slot. In the former case, a leftover job will require full complement of its VMs and in the latter case a subset of the VMs it’s currently holding. At the beginning of the next slot, the system will schedule the new arriving jobs and the leftover unfinished jobs from the previous slot such that power consumption is minimized. For the scheduling of leftover jobs, there are two options depending whether or not VM migration is allowed. If VM migration is allowed, then leftover jobs are scheduled like the new jobs, on the other hand, if no migration is allowed then the new jobs can only be scheduled to VMs not utilized by the leftover jobs. As a result of migration, the system may end up in a state that consumes less power, however, migration has communication and processing overhead that optimization needs to take into account. Let G_r denote normalized power consumption cost of migration of type r VMs, which from [27] may be determined as follows,

$$ {G}_r=\left( VM\ Migration power Consumption\left( Source+ Destination\right)\right)\times \frac{Migration Duration\ }{Time Slot Duration} $$

Optimization will allow VM migration if power saving due to migration offsets the cost of migration. As a result, the optimization may result in partial VM migration.

Since jobs release their VMs according to the Bernoulli trials, number of leftover jobs to the next slot will be a random variable with Binomial distribution. However, to make the analysis tractable we will assume that number of leftover jobs is a constant given by the mean of the Binomial distribution. Let ${N}_h^{\prime }$ denote number of the type h leftover jobs from the current slot and N_h total number of jobs to be scheduled in the next slot, which include both leftover as well as new arriving jobs. We note that ${N}_h\ge {N}_h^{\prime }$ and ${n}_h\in \left({1}_h,..,{N}_h^{\prime },..,{N}_h\right)$ and the first ${N}_h^{\prime }$ jobs in the set correspond to the leftover jobs from the current slot. Next, we will develop both dynamic IQP and CG models for re-optimization of power consumption.

Dynamic IQP model

Let us consider n_h^th job, which is in the system in the current slot and will continue to receive service in the next slot. Let ${x^{\prime}}_{r,{n}_h}^{m_t}$, ${x}_{r,{n}_h}^{m_t}$ denote the number of type r VMs assigned to this job over the m^th type t server during the current and next slots respectively. Based on the new notation introduced in Table 2, we define the following binary variable,

$$ {\beta}_{r,{n}_h}^{m_t}=\left\{\begin{array}{c}\ 1\ \mathrm{if}\ {x}_{r,{n}_h}^{m_t}-{x^{\prime}}_{r,{n}_h}^{m_t}<0\\ {}\ 0\kern2em \mathrm{otherwise}\ \end{array}\right. $$

(43)

Table 2 Additional notation

The value of ${\beta}_{r,{n}_h}^{m_t}$ shows whether type r VMs required by job n_h have migrated or not. In the case of VM migration from this type of server, then ${x}_{r,{n}_h}^{m_t}<{x^{\prime}}_{r,{n}_h}^{m_t}$ and as a result ${\beta}_{r,{n}_h}^{m_t}$ will have a nonzero value and in all other cases a zero value. The objective function of this optimization problem is given by,

$$ \mathit{\operatorname{Min}}\ \left[(8)+{\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h^{\prime }}{\sum}_{r=1}^R{\sum}_{\ell =1}^L{\sum}_{e=1}^{d_{\ell }}{\sum}_{t=1}^T{\sum}_{m_t\in {a}_{\ell, e}}{G}_r{\beta}_{r,{n}_h}^{m_t}\left|{x}_{r,{n}_h}^{m_t}-{x^{\prime}}_{r,{n}_h}^{m_t}\right|\ \right] $$

(44)

where absolute value of $\Big({x}_{r,{n}_h}^{m_t}-{x^{\prime}}_{r,{n}_h}^{m_t}$) corresponds to number of VM migrations. In the above, migration of a VM will be allowed if it results in power saving larger than cost of migration. Q_t in (8) also is considered as a linear function of total number of ${\overset{\sim }{x}}_{n_h}^{m_t}$s to better approximate the dependence of the utilization of the server in power consumption.

Job scheduling without VM migration can be achieved by assigning to G_r a very large value, which prevents migration as its cost cannot be offset by any amount of power saving. As a result, unfinished jobs will preserve their VM assignments. Finally, we have to add the following constraints into the optimization problem in order to linearize Eq. (43),

$$ {x}_{r,{n}_h}^{m_t}-{x^{\prime}}_{r,{n}_h}^{m_t}+\theta {\beta}_{r,{n}_h}^{m_t}<1 $$

(45)

$$ {x}_{r,{n}_h}^{m_t}-{x^{\prime}}_{r,{n}_h}^{m_t}+\theta {\beta}_{r,{n}_h}^{m_t}\ge 0 $$

(46)

where (45), (46) are ∀r ∈ {1, …, R}, m_t ∈ {1, …, M_t}, t ∈ {1, …, T}, n_h ∈ {1_h, …, N_h}, h ∈ {1, …, H}.

Dynamic CG model

Next, we consider the dynamic CG model. Assume that n_h^th job is in the system in the current slot and will continue to receive service in the next slot. Let ${x^{\prime}}_{r,{n}_h}^{j_t}$, ${x}_{r,{n}_h}^{j_t}$ denote number of type r VMs assigned to this job over the j_t’th pattern during the current and next slots respectively. Similarly, ${\varphi^{\prime}}_{\ell, e}^{f,{j}_t}$, ${\varphi}_{\ell, e}^{f,{j}_t}$ are binary variables indicating whether f’th type t server on rack e in PoD ℓ is active and has pattern j_t during the current and next slots respectively. In this model, we define binary variable ${\beta}_{\ell, e,r,{n}_h}^{f,t}$ to show whether or not r type VMs required by job n_h have migrated or not from a server as follows,

$$ {\beta}_{\ell, e,r,{n}_h}^{f,t}=\left\{\begin{array}{c}\ 1\ if\ {\sum}_{j_t={1}_t}^{J_t}\left({x}_{r,{n}_h}^{j_t}{\varphi}_{\ell, e}^{f,{j}_t}-{x^{\prime}}_{r,{n}_h}^{j_t}{\varphi^{\prime}}_{\ell, e}^{f,{j}_t}\right)<0\\ {}\ 0\kern4em otherwise\ \end{array}\right. $$

(47)

We note that the summation in the above allows the use of a different pattern at the server as long as it preserves the number of VMs assigned by the original pattern to this job. The objective function of this optimization problem is given by,

$$ \mathit{\operatorname{Min}}\ \left[(23)+{\sum}_{h=1}^H{\sum}_{n_h={1}_h}^{N_h^{\prime }}{\sum}_{r=1}^R{G}_r{\sum}_{\ell =1}^L{\sum}_{e=1}^{d_{\ell }}{\sum}_{t=1}^T{\sum}_{f=1}^{M_{\ell, e}^t}{\beta}_{\ell, e,r,{n}_h}^{f,t}{\sum}_{j_t={1}_t}^{J_t}\left|{x}_{r,{n}_h}^{j_t}{\varphi}_{\ell, e}^{f,{j}_t}-{x^{\prime}}_{r,{n}_h}^{j_t}{\varphi^{\prime}}_{\ell, e}^{f,{j}_t}\right|\ \right] $$

(48)

As in the previous subsection, job scheduling without VM migration can be achieved by setting G_r to a very large value. Finally, similar to the previous subsection, we have to add the following constraints to the problem in order to linearize (47),

$$ {\sum}_{j_t={1}_t}^{J_t}\left({x}_{r,{n}_h}^{j_t}{\varphi}_{\ell, e}^{f,{j}_t}-{x^{\prime}}_{r,{n}_h}^{j_t}{\varphi^{\prime}}_{\ell, e}^{f,{j}_t}\right)+\theta {\beta}_{\ell, e,r,{n}_h}^{f,t}<1 $$

(49)

$$ {\sum}_{j_t={1}_t}^{J_t}\left({x}_{r,{n}_h}^{j_t}{\varphi}_{\ell, e}^{f,{j}_t}-{x^{\prime}}_{r,{n}_h}^{j_t}{\varphi^{\prime}}_{\ell, e}^{f,{j}_t}\right)+\theta {\beta}_{\ell, e,r,{n}_h}^{f,t}\ge 0 $$

(50)

where (49), (50) are $\forall r\in \left\{1,\dots, R\right\},f\in \left\{1,\dots, {M}_{\ell, e}^t\right\},t\in \left\{1,,,T\right\},{n}_h\in \left\{{1}_h,\dots, {N}_h\right\},h\in \left\{1,\dots, H\right\}$.

Optimization structure and complexity mitigation

In this section, we consider initialization of the optimization and rounding of the solution of relaxed problem to integer values.

CG initialization

We use offline initialization to reduce computation time for the solution of the optimization problem. Without initialization, in the first iterations, the RMP does not contain adequate columns to provide beneficial dual information to pricing sub-problems [28]. An appropriate initialization helps to reduce number of iterations of the solutions of RMP and pricing problems through introduction of optimal patterns, which are patterns that maximize resource utilization of active servers. Using the notation introduced in Table 2, we define the initialization (optimization) problem as follows,

$$ \mathit{\operatorname{Max}}\ {\sum}_{r=1}^R{x}_{r,h}^{{\overset{\sim }{I}}_t}{i}_r^k $$

(51)

$$ ST.\kern0.5em {\sum}_{\begin{array}{c}r=1\\ {}\nabla r\in {R}_h\end{array}}^R{x}_{r,h}^{{\overset{\sim }{I}}_t}{i}_r^{k^{\prime }}\le {c}_t^{k^{\prime }}\forall {k}^{\prime}\in \left\{1,\dots, K\right\} $$

(52)

where, R_h denotes the set of VM types available to type h jobs. Solving this problem for each {k, t, h} results in the best Υ_t patterns for different types of jobs. Then, for a type t server we will have Υ_tHK initial patterns. To obtain ${x}_{r,{n}_h}^{{\overset{\sim }{I}}_t}$s, which are introduced in the previous sections and are related to the initial pattern ${\overset{\sim }{I}}_t$, ${x}_{r,h}^{{\overset{\sim }{I}}_t}$ is assigned to a type h job while other jobs are set to zero. Hence, for each ${x}_{r,h}^{{\overset{\sim }{I}}_t}$ variable there would be N_h different patterns. Thus, initial number of patterns for server type t will be equal to ${\sum}_{h=1}^H{N}_h{\mathrm{Y}}_tK$. So in the proposed initialization, we may have separate candidate patterns for each job. Then through collaboration of the pricing problems and RMP, new patterns, that consider different jobs in a server will be introduced by pricing problems.

Heuristic rounding termination algorithm

As mentioned earlier, LP problem (solvable in polynomial time) has less complexity compared to ILP problem (NP-hard optimization problem). In the CG solution of our optimization problem, RMP has been formulated as a LP and pricing problems as ILP type. As a result, we need to determine the optimal ILP solution of the RMP after the solution of the relaxed LP. Typically, this is done through the branch and bound algorithm [18], which is time consuming, as a result, we propose a heuristic method that satisfies the scheduling time constraint [28, 29]. The proposed method will round up and down the values of the scheduling variables, ${m}_{\ell, e}^{j_t}$, of the LP solution [30, 31]. This operation will be carried out after ${m}_{\ell, e}^{j_t}$ have been sorted according to their priorities. ${m}_{\ell, e}^{j_t}$s more likely to be rounded down will be given higher priority. Following this operation, it is possible that all the servers of a rack will become inactive in which case TOR switch serving to that rack will be turned off to save power.

First, let us define s_{ℓ, e} as the set of scheduling variables for the rack e on PoD ℓ,

$$ {s}_{\ell, e}=\left\{{m}_{\ell, e}^{j_t},{j}_t\in \left\{{1}_t,\dots, {J}_t\right\},t\in \left\{1,\dots, T\right\}\right\} $$

and define set S as the set with its elements given by the subsets s_{ℓ, e} as given below,

$$ S=\left\{{s}_{\ell, e}|{1}_{\ell}\le e\le {d}_{\ell },1\le \ell \le L\right\} $$

Next, we split S into two mutually exclusive subsets,

$$ S=\left\{{S}_1,{S}_2\right\} $$

where S₁ consists of all s_{ℓ, e} that have their scheduling variables with values strictly less than one and S₂ otherwise. The elements of S₁denote potentially inactive racks, while the elements of S₂ active racks. From the above definition, elements of S₁are given higher priority than S₂ in the rounding operation. Next, we explain the steps of the proposed heuristic, we note that Step i) applies only to S₁, while the remaining steps apply both to S₁ and S₂.

Step i) Sort the elements of set S₁ according to the number of active servers in a rack, $\sum \limits_{t=1}^T{\sum}_{j_t={1}_t}^{J_t}{m}_{\ell, e}^{j_t}$, in ascending order with the first element of S₁ having the least number of active servers. We note that the order of the elements of S₂ isnot significant for rounding operation.

Step ii) Sort the scheduling variables in each s_{ℓ, e} subset according to server type efficiency and resource scarcity. First, we explain how to determine server type efficiency. Depending on the job load some resources become critical and may become performance bottleneck [32, 33]. As a result, we first sort resources according to their criticalities. For a given job load, let L^kdenote the total demand for type k resource,

$$ {L}^k=\sum \limits_{h=1}^H{\sum}_{n_h={1}_h}^{N_h}\sum \limits_{r=1}^R{v}_{n_h}^r\ {i}_r^k\forall k\in \left\{1,\dots, K\right\} $$

Then, the resource types may be ordered according to their criticality using the following formula,

$$ {\mathit{\max}}_k\frac{L^k}{\sum \limits_{t=1}^T{M}_t{c}_t^k} $$

(53)

Thus higher is the ratio of total demand to total amount of that resource in the datacenter, then higher will be the criticality of that resource. Next, we define efficiency of a server type with respect to resource type k as the ratio of (${c}_t^k/{Q}_t$) with higher value indicating higher efficiency. Then, we order server types according to their efficiency for the critical resource. In the case of a tie, server efficiencies with respect to second critical resource will be used to break down the ties and so on and so forth. System will prefer to use the server types with higher efficiencies. The scheduling variables in each s_{ℓ, e} subset will be sorted in ascending order according to the efficiency of their server types.

Step iii) Sort the scheduling variables with common server type in each s_{ℓ, e} subset according to pattern efficiency:

The patterns of each server type will be sorted in ascending order according to their resource utilization $\sum \limits_{h=1}^H\sum \limits_{n_h={1}_h}^{N_h}\sum \limits_{r=1}^R{x}_{r,{n}_h}^{j_t}\ {i}_r^k$.

Step iv) Apply the rounding down operation. Following the completion of sorting, all the ${m}_{\ell, e}^{j_t}$s within the set S have been assigned priority with the first element of the set having the highest priority in rounding down operation. Initially, we round up all the ${m}_{\ell, e}^{j_t}$ variables with non-integer values. Then, rounding down operation is applied from the highest to lowest priority ${m}_{\ell, e}^{j_t}$s one by one. In this operation, each ${m}_{\ell, e}^{j_t}$ is decremented by one if the demand constraints are not violated, $\sum \limits_{\ell =1}^L\sum \limits_{e=1}^{d_{\ell }}\sum \limits_{t=1}^T\sum \limits_{j_t\in {J}_t}\left(\ {x}_{n,r}^{j_t}\right){m}_{\ell, e}^{j_t}<{v}_n^r$.

The complexity order of the proposed algorithm may be approximated as,

$$ O\left({\sum}_{t=1}^T{M}_t{J}_t\left(\mathit{\min}\Big(\mathit{\log}\left({\sum}_{t=1}^T{M}_t{J}_t\right), NR\right)\right) $$

where, M_t, N, R and J_t are number of type t servers, number of jobs, number of VM types and number of patterns of type t servers respectively. $\left(\sum \limits_{t=1}^T{M}_t{J}_t\Big(\mathit{\log}\left(\sum \limits_{t=1}^T{M}_t{J}_t\right)\right)$ is due to the sorting part and $NR\sum \limits_{t=1}^T{M}_t{J}_t$ is due to the demands for constraints part.

Numerical results

In this section, we present some numerical results regarding the analysis in the paper. Numerical results plot a performance metric for assignments of VMs to new arriving jobs either at an empty or non-empty system that optimizes power consumption. In an empty system, all the VMs are available, while in a non-empty system some of the VMs are occupied by the jobs already in the system. In the non-empty case, a performance metric is plotted as a function of discrete-time and new jobs arrive to the datacenter according to a Poisson process with parameter λ and VMs of the jobs in the system are released according to independent Bernoulli trials.

We compare performance of our optimal VM placement algorithm with that of two heuristic VM scheduling algorithms to be referred to as deterministic and random. The deterministic algorithm is similar to the scheduling scheme proposed in [34] that assigns a job to the PoD and rack with the smallest index number, which has enough idle resources to serve the job. In the random algorithm, each VM of a job is placed to a randomly chosen rack of a PoD with enough idle resources given that communication demand does not violate the link capacities; otherwise a new rack is randomly chosen for the placement of VM.

IBM ILOG CPLEX version 12.4 on a machine at 3.4 GHz(core i7) with 32GB RAM is used as a platform to solve the optimization problem. We solve the optimization problem using both IQP and CG techniques. IQP technique provides exact solution but is applicable to only small size systems, while CG is applicable to systems with large sizes but has rounding approximation. As a result, we test the accuracy of the CG technique against the IQP at the end.

We assume a datacenter with the hierarchical topology shown in Fig. 1. We presume that the datacenter has 4 PoDs and each PoD having 12 racks. In consonance with [35], we assumed that each rack contains 40 to 80 servers and all the racks of each PoD have the same server composition. Next, we present the parameters of the system used in the generation of numerical results.

• i) Servers and Server Types

  Considering Amazon instances and Google clusters, we assume T = 12 server types with two types of resources, CPU cores and memory. Table 3 presents the amount of resources and power consumption of each server type. Note that Q_t values are for maximum utilization cases. Table 4 presents number of servers per server type per rack at each PoD. Table 5 shows number of servers per server type per PoD, which is obtained by multiplication of each entry of Table 4 by 12.

• ii) Communication Network Parameters

  Table 6 presents the performance characteristics of the chosen switches for the communications network. Power consumption parameter values of the switches, PD_{ℓ, e} and PS_{ℓ, e}, are the same as given in [36,37,38]. We also assume that dynamic power consumption of a NIC is given by PW_NIC = 0.6 microW. ToR switches offer a combination of internal (int) and external (ext) interfaces. The internal interfaces connect to NIC of the blade-servers while the external interfaces connect to Core switches. It is assumed that internal and external interfaces support up to 10 Gbps and 40 Gbps transmission rates respectively.

• iii) Parameters of VM Types

  We presume that number of VM types is R = 18 with their resource requirements given in Table 7. Resources of VMs consist of number of CPU cores and amount of memory. It is assumed that each physical core of a CPU is utilized as a virtual CPU (vCPU). In order to balance CPU, memory and network resources, Amazon t2 and m3 series are appropriate for many applications and servers, Microsoft SharePoint, and enterprise applications. c3 series with higher ratio of vCPU to memory represent compute-optimized Amazon instances which are appropriate for high-traffic web sites, on-demand batch processing, distributed analytics, web servers, and high performance science and engineering applications. r3 series represent memory optimized amazon instances and are recommended for memory bound applications such as high performance databases and distributed cache, in-memory analytics, genome assembly, and larger deployments of SAP. cg1 and g2 are also considered for game streaming, video encoding, 3D application streaming and other server-side graphic workloads.

• iv) Parameters of job types

  We assume that the number of job types equals to H = 7 with h = 1..H. Table 8 presents requirements and appropriate applications for each job type. The type of each job is determined probabilistically through the values given in the column for parameter α_h. The number of VMs required by a type h job is determined by the constant C_h. From Amazon recommendations in [39, 40], the table present the mixture of VM types required by a job of each type. In each job type, the VMs are chosen probabilistically from the allowed VM types according to the percentages given in the table. Thus, first the type of a job and the number of VMs it requires are determined and then the types of each of its VM.

Table 3 Characteristics of Server types

Table 4 No. of servers per type per rack of POD ℓ

Table 5 No. of servers per type per PoD

Table 6 Characteristics of Typical Switches

Table 7 VM configurations

Table 8 Jobs Types and their VM requirements

We assume that the traffic rate between two VMs of a job is either a random variable or a constant. In the former case, the mean and standard deviation of the traffic rate for each job type is given in the last column of the Table 8. In the latter case, the traffic rate for each job type is a constant that equals to the mean value of the variable traffic rate (ω_h). We considered both individual and simultaneous release of VMs of jobs at the end of a slot according to Bernoulli trials. In either case, the success probability in a Bernoulli trial is assumed to be ρ_h for a type h job. For this example, we assumed homogeneous Bernoulli trials with ρ_h=0.3 ∀h ∈ {1, …, H}. Finally for the power constraint in the probabilistic model, we assume that power supply of a rack is PR_{ℓ, e} = 25kW [26] and maximum power overloading probability of the racks is set to p = 0.02. In the following results, unless otherwise stated, the arrival of new jobs will be according to a Poisson process with parameter λ = 200 jobs/slot with constant server power consumptions and VM traffic rates.

The Fig. 3 presents optimal power consumption of the system as a function of the number of time slots with VM migration cost G_r as a parameter and with individual VM release. It may be seen that optimal power consumption increases with the rising cost of VM migration cost. The zero cost migration (G_r = 0) and no migration (G_r = ∞) provide lower and upperbound for power consumption with about 8% difference between them.

Fig. 3

Optimal power consumption as a function of time with migration cost G_r as a parameter

Figure 4 presents optimal power consumption of the system as a function of the number of time slots for both individual and simultaneous release of the VMs assigned to a job and migration cost of G_r=0.3. As may be seen, simultaneous release results in lower power consumption compared to individual release.

Fig. 4

Optimal power consumption as a function of time for individual and simultaneous VM release with parameter ρ_h=0.3 and migration cost G_r=0.3

Figures 5 and 6 show power consumption as a function of the number of jobs in the system for optimal placement of VMs as well as according to the deterministic and random heuristics for constant and random server power consumptions respectively. As may be seen, in both cases, optimal placement of the jobs results in the lowest power consumption, and next to it is deterministic placement. It is also seen that the random server power consumption results in lower system power consumption compared to constant server power consumption.

Fig. 5

Optimal, deterministic and random heuristic power consumptions as a function of the number of jobs in the system for constant server power consumption

Fig. 6

Optimal, deterministic and random heuristic power consumptions as a function of the number of jobs in the system for random server power consumption

Figure 7 presents optimal power consumption with migration cost G_r=0.3 as a function of the number of time slots. Also shown in the figure are the power consumptions of deterministic and random heuristics. It may be seen that optimal placement results in about 15% lower power consumption than deterministic heuristic and lower by a bigger amount than random heuristic. For the system of Figs. 7 and 8 shows communication component of the power consumption. As may be seen again, optimal placement results in lower communication power consumption than the two heuristics even by larger margins than the total power consumption.

Fig. 7

Optimal power consumption with migration (Gr = 0.3) together with deterministic and random heuristics as a function of time

Fig. 8

Communication power consumption for the optimal placement with migration (Gr = 0.3) together with deterministic and random heuristics as a function of time

Figures 9 and 10 show histogram of the number of TORS to CS links as a function of the link transmission rates for optimal, deterministic and random placement of jobs for constant and variable VM traffic rates respectively. As may be seen in both cases optimal placement results in lower communication traffic compared to the two heuristics.

Fig. 9

Histogram of the number of TORS to CS links for optimal, deterministic and random placement of the jobs as a function of the link transmission rate for constant VM traffic rate

Fig. 10

Histogram of the number of TORS to CS links for optimal, deterministic and random placement of the jobs as a function of the link transmission rate with variable VM traffic rate

Figure 11 shows the number of active racks for optimal with migration cost of G_r=0.3, deterministic and random placement of jobs at certain time slots. We note that the total number of the racks in the system is 48. As may be seen, optimal placement results in lower number of active racks compared to deterministic and random heuristics in any time slot (Time slot duration is considered 5 min).

Fig. 11

The number of active racks in the system at slots 5, 10, 15 and 20 for optimal with migration cost (Gr = 0.3), deterministic and random placement of jobs

We were able to solve IQP exactly for small size systems, which enabled us to examine quality of the solutions obtained through the CG technique. In Fig. 12, we plotted power consumption of CG/proposed rounding and IQP as a function of the number of jobs. The figure also plots results of random rounding of the relaxed CG solution, which provides an upper-bound for the performance of our optimization model. It can be seen that the optimality gap between the exact IQP results and upper-bound is up to 6%, while between the CG/proposed rounding and exact results is less than 1% for N < 50. Thus, the CG technique with the proposed rounding algorithm results in quite accurate solutions.

Fig. 12

Comparison of values of the objective functions among IQP, CG/Proposed Rounding and CG/Random Rounding

Next, we look at the run time of the optimization models in Fig. 13. It can be noticed that as the workload (number of jobs) in the datacenter increases, the runtime of both IQP and CG increase. However, the runtime of the IQP grows exponentially while that of CG almost linearly due to the fact that CG is able to determine the solution by scanning far fewer number of configurations. Please note that the runtime of CG was on the Intel Core i3-2467 M @ 1.60GHz and by application of the parallelism on 12 icores (Since T = 12), the run time can be reduced to few seconds. Moreover, the application CG allows scalability of the proposed platform for very large scales.

Fig. 13

Comparison of the run time between IQP and CG/proposed rounding

Discussion on assumptions

In this section, assumptions made in this work are listed and evaluated. Proposed assumptions along with alternative possibilities are shown in in Tables 9, 10, 11 and 12 (assumptions of this paper are in italics).

Table 9 List of Assumptions and alternative possibilities: Possible Topologies of Datacenters

Table 10 List of Assumptions and alternative possibilities: Possible Communication Models among job components

Table 11 List of Assumptions and alternative possibilities: Common assumption on distribution of data center traffic

Table 12 List of Assumptions and alternative possibilities: Power consumption Models of servers as a function of workload

The first Assumption made in this paper is related to Topology. We assumed Fat-Tree Topology. However, the analysis in this work, without loss of generality, can be extended to other tree cloud topologies types. The main reason behind this selection is that more than 70% of the cloud datacenters have tree-based architecture and to make the analysis realistic it is better to consider the most common scenarios. For more information, please refer to [41,42,43]. It is worth mentioning that this research focused on the infrastructure of large-scale hosted datacenters which is responsible for the management and maintenance of the data and processing jobs of many different companies. Thus, this research is better suited for public cloud scenarios.

Different communication demand models are considered in the literature. Communication demand models of cloud jobs are investigated in different layers as represented in Fig. 14. For instance, communication model can be defined for a cloud-based web application or a map-reduce processing job according to a graph at the application layer. Communication demand among cloud components also can be defined at transportation layer according to the socket (port and IP). In this paper, the prevalent definition of communication model is defined according to the IP addresses at network layer. Please note that, if at least one application resides in a VM communicate with another application on another VM, there is a communication link between two VMs. Moreover, in many cloud management platform like OpenStack, there is always some communication overheads among the components (ComputeNodes). Thus, in this paper, we assumed that all the VMs of a job communicate with each other at least once and the graph model among the VMs can be approximated by a Full-mesh model. Consequently, the magnitude of demand between two servers will be assumed to be proportional to the product of the number of VMs assigned to that job on the two servers. For instance, for a job presented in Fig. 14, there is only two communication demand (2 × 1) exist among the VMs of a job. In Table 10 other works in the literature that assume the mesh model are listed.

Fig. 14

Communication demand models in different layers

A workload of the cloud computing datacenters can be approximated by different processes. Poisson and Gaussian processes are widespread in this approximation. However, for high scale scenarios, due to the central limit theorem, the Gaussian process is more realistic. Many works, as listed in Table 11, applied Gaussian process regression to approximate the Datacenter traffics. Thus, as the size of the public clouds increases, the analysis of this research is more reliable and trustworthy. Please also note that it is too complicated to schedule the unpredicted workload in real time. The power consumption models of cloud computing servers are also listed in Table 12. Many works have found a strong linear relationship between the workload and total power consumption by a server so that the power consumption by a server increases linearly with the growth of server workload from the value of the power consumption in the idle state up to the power consumed when the server is fully utilized. As it explained earlier, it is assumed that the incoming workload (traffic and process) follows a Gaussian distribution. The Linear combination (summation) of Gaussian processes also follows a Gaussian distribution. As the best of our knowledge, the linearity assumption between the power consumption and the workload (Traffic and Process) has been controversial. So, one of the limitations of this paper is that it is constrained to this linear relationship. However, to avoid inaccuracy, the Gaussian assumption is made at the Rack level.

Conclusion

In this paper, we have studied optimization of power consumption in cloud computing centers through VM placement. We have developed joint optimization of power consumption of servers, network communications, and cost of migration with workload and server heterogeneity subject to resource and bandwidth constraints for a cloud computing center with hierarchical network topology. Optimization results in an IQP that can only be solved for systems with small sizes, then we show application of the CG technique that enables solution of systems with larger sizes. CG technique has an approximation as it solves continuous relaxation of the problem, which requires rounding of the solution to integer values. Comparison of the results of CG with IQP shows the accuracy of CG resulting in optimality gap less than 2%. Then, the optimization has been extended to manage temporal variation of the workload, which allows arrival and departure of jobs at the discrete-time instants. The system performs re-optimization of the power consumption under the new workload that also includes cost of migration. The numerical results show that optimization achieves power savings compared to the heuristic VM placement algorithms. In general, the field is short of work that solves optimization of power consumption problem and we hope that our work will help to bridge this gap. As far as we know this is the first work that applies CG technique to solve this problem. Results of this work may also be used to test accuracy of future heuristics for VM placement in cloud computing centers. The presented optimization method could also be used for the systems based on containers instead of VMs. We believe that the proposed optimization will be helpful to cloud service providers in realization of energy saving.