Notes on Cloud computing principles

Multi-tenancy

A tenant in the Cloud context is a user of Cloud infrastructure, i.e. Infrastructure-as-a-Service (IaaS) services [7]. A VM owner is an example of a tenant and if multiple VM owners are allocated on the same physical machine it is an example of multi-tenancy [8]. The difference between a multi-(end)-user service and a multi-tenant service is that a multi-user offering may benefit from having users know about each other and explicitly share social content to promote the network effect. A multi-tenant solution could internally benefit from shared physical resources but must give the impression of an exclusive offering to each of the tenants. As an example, hosting the Facebook service on a Web server in the Cloud would be an example of a multi-user service, but hosting both a Twitter Web server and a Facebook Web server in the same Cloud data center would be an example of multi-tenancy. From this definition, it is clear that the IaaS provider needs to provide mechanisms to isolate the tenants from each other.

Multiple tenants need to be isolated in terms of privacy, performance and failure:

Privacy Isolation. Multiple tenants must not have access to each other’s data. This may seem like an easy requirement to meet but in a typical file system there may be traces left after a file even after removing it, which would violate this property.

Performance Isolation. Multiple tenants must not be effected by each other’s load. If one tenant starts running a CPU intensive task and other tenants see a drop in performance as a result, then this property is violated.

Failure Isolation. If a tenant either inadvertently or maliciously manages to crash its compute environment, it should not effect the compute environment of other users. Imagine a Java VM hosting multiple applications such as a Tomcat Servlet engine. Now, if one servlet Web app crashes the VM, then the other apps in the same VM would also crash. This failure would in that case be a violation of the failure isolation property. Virtual machines offer a popular technique to ensure isolation, but in some cases the overhead of virtualization, of e.g. IO and network, is too high so a trade-off has to be made between isolation level and performance.

Ensuring these levels of isolation is closely related to the strategy used to allocate resources to tenants, which we will discuss next.

Statistical multiplexing

One major benefit related to data center consolidation that we discussed in the introduction is statistical-multiplexing [9]. The idea behind statistical multiplexing is that bursty workloads that are consolidated on the same Cloud infrastructure may in aggregate display a less bursty pattern. Figure 1 shows an example of statistical multiplexing with two workloads exhibiting complementing demand over time.

1. 1.

   Without an elastic Cloud infrastructure, the most common way of provisioning resources to tenants is to allocate resources that meet the peak demand of each workload. Clearly, this leads to a major waste in resources for the majority of the time. Statistical multiplexing allows an allocation that is substantially lower than the sum of the peaks of the workloads.

    

This model of perfect statistical multiplexing is hard to achieve in practice. The main reason for this is that workloads tend to be correlated. The effect is known as self-similarity. Self-similar workloads have the property that aggregating bursty instances will produce an equally bursty aggregate, something that is often observed in practice. However, there are many techniques to recreate the effects of statistical multiplexing without having to hope for it to occur organically. For instance, you could measure the correlation between workloads and then schedule workloads that are complementing on the same resources. These techniques are sometimes referred to as optimal packing of workloads or interference minimization [10]. Poor statistical multiplexing tends to lead to low utilization, or unmet demand, as we will discuss further when we review the economic principles governing under and over-provisioning.

Horizontal scalability

Amdahl’s Law predicts the expected speed-up of a program or algorithm when run over multiple machines. T(n) is the time taken to run on n machines. B is the fraction of the program that needs to run serially, i.e. that cannot be parallelized. Note that several disjoint sections in the execution path may need to run serially to collect, distribute or synchronize parallel computations. It is clear that minimizing B maximizes the speedup. However, the most important consequence of Amdahl’s Law is that it sets a theoretical cap on how many machines a program will benefit from running on, beyond which point adding new machines will not make the program run faster. If B is close to negligible we can expect linear scalability. Adding x machines will make the program run x times faster. If the program speedup grows at a slower rate than the number of machines added, which is the common case due to various overheads of distribution, we refer to sublinear scalability. The program may also speedup at a faster rate than the machines being added, in which case the program is said to exhibit superlinear scalability (see Figure 2). This effect may happen if there is some common resource like a shared cache that benefits from more usage, e.g., more cache entries and fewer cache misses.

Many advances in the database community have emerged, and been popularized, to achieve horizontal scalability when processing massive amounts of data in Clouds. The most prominent concepts include: data partitioning and sharding [12]; consistent hashing and distributed hashtables (DHT) [13],[14]; and eventual and quorum consistency [15],[16].

Over and under provisioning

As we alluded to in the section on statistical multiplexing, over-provisioning is a common strategy for allocating resources across tenants. Here we discuss the economic dilemma of over (Figure 3) versus under-provisioning (Figure 4) resources.

We can see that over-provisioning leads to a large area of idle resources over time. In financial terms this means high-operational cost, and lost opportunities to increase profit. To increase profit the IaaS provider may be tempted to lower the allocation to reduce the operational cost as seen in Figure 4. However, this leads to an even more severe drawback, unmet demand. Unmet demand means revenue loss, and can have long-term negative effects as customers who are denied access to a resource despite being willing to pay for it may not return. For this reason over-provisioning is more popular than under-provisioning. However, neither the IaaS provider nor the tenant may be able to perfectly predict the peaks, after all that is why they are running in the Cloud in the first place. In this case under-provisioning may occur inadvertently.

Hence, over-provisioning versus under-provisioning involves making a trade-off between profit and revenue loss.

Variable pricing

Given all the issues of allocating resources to bursty demand, it is natural to ask whether this burstiness can be suppressed somehow as opposed to being accommodated. That is exactly the idea behind variable pricing or demand-driven pricing. The idea is to even out the peaks and valleys with incentives. If the demand is high we increase the price. This leads to tenants who cannot afford the higher price to back-off and thereby demand is reduced. On the other hand, if the demand is low, a price drop may encourage tenants who would otherwise not have used some resources to increase their usage and thereby demand. The end result is a stable aggregate demand as in the statistical multiplexing scenario. The key benefits to IaaS providers include the ability to cash in on peak demand by charging premiums, and a mechanism to increase profit during idle times. Now, how can we ensure that the price is a good representation of demand? Here, microeconomic theory of supply and demand [17] helps.

If we plot the quantity of goods a supplier can afford to produce given a price for the good we get the supply curve. If we plot the quantity of goods requested by consumers given a price for the good we get the demand curve. The price at the point where the supply and demand curves meet is called the efficient marker price as it is a stable price that a market converges towards (see Figure 5). To see why this is the case, consider the gray dot on the supply curve in Figure 5. In this case the supplier observes a demand that is higher than the current quantity of goods produced. Hence, there is an opportunity for the supplier to increase the price of the good to afford to produce more goods to meet this demand. Conversely, considering the black dot on the demand curve, we can see that the demand is higher than the volume of goods that the supplier can produce. In this case the demand will naturally go down and the consumers are likely to be willing to pay a higher price to get their goods.

In general, variable pricing allows a provider to allocate resources more efficiently.

Price setting

There are many ways to set prices for goods in a market. The most commonly known are various forms of auctions, spot prices and reservations. In auctions, bidders put in offers to signal how much they are willing to pay for a good. In double actions, there are also sellers who put in asks denoting how much they are willing to sell the good for. The stock market is an example of a double auction. In computational markets, second price sealed bid auctions are popular since they are efficient in determining the price, i.e. reflect the demand, without too much communication. All bidders put in secret bids and the highest bidder gets the good for the price equalling the second highest bid.

In the case where there is not a completely open market price, and there is just a single provider selling off compute resources, spot pricing is a common way of setting demand based prices. The spot price is computed on a running basis depending on the current level of demand. There could for instance be a base pay that is discounted or hiked based on demand fluctuations. A spot market differs from a futures market in that goods are bought and consumed immediately. Futures markets such as options are less common in practical computational markets today.

Purchasing resources on a spot market involves a high risk of either having to pay more for the same allocation or being forced to reduce the allocation to stay within budget (see the section on Predictability below). A common way to reduce the risk for consumers is to offer a reservation market. A reservation market computes the expected spot demand for some time in the future and adds a premium for uncertainty to arrive at a reservation price. Essentially you have to pay for the provider’s lost opportunity of selling the resources on the spot market. This way the risk is moved from the consumer of compute resources, the tenant, to the provider. I.e., the provider’s actual cost or revenue when providing the resource may vary, whereas the cost for the tenant is fixed. If there is an unexpected hike in the demand and all resources have already been promised away in reservations there is no way for the provider to cash in on this demand, which constitutes a risk for the provider.

The research field of computational economies have tackled these problems as far back as the 1960s and 70s [18]-[20]. More recent computational market designs include [21]-[23]. Reviews of some of these designs can be found in [24],[25].

In summary, reservation markets move the risk of uncertain prices from the tenant to the provider as uncertain demand.

The tragedy of the commons

The next principle we will discuss is a social dilemma referred to as the tragedy of the Commons [26]. The dilemma was introduced in a paper in 1968 by Garrett Hardin, where the following scenario was outlined.

Imagine a public, government-owned piece of land with grass, in the UK referred to as a Common. Now, a number of shepherds own sheep that they need to feed on this Common to keep alive. The shepherds will benefit economically from the sheep because they can, for instance, sell their wool. Each shepherd faces the financial decision whether it would be more profitable to purchase another sheep to feed on the Common and extract wool for, or provide more food to each sheep by sticking with the current herd. Given that it is free to feed the sheep on the Common and the reduction in available food is marginal, it turns out that it is always optimal for a selfish shepherd trying to optimize his profit to buy another sheep. This has the effect of driving the Common into a slump where eventually no more grass is available and all sheep die and all shepherds go bankrupt.

One could argue that less selfish shepherds who are wary of the benefits of the group of shepherds as a prosperous community will not let the situation end in tragedy. However, there are many examples of communities that have gone extinct this way. In general what these communities have in common is that there is a high degree of free-riders, i.e. community members who take more from the common resources of the community than they give back. Sometimes the effects are temporal and not as obvious since no one purposefully abuses the community. One example is the PlanetLab testbed [27] used by systems researchers in the US. The testbed is distributed across a large number of organizations to allow wide area and large-scale experiments. The weeks leading up to major systems conferences such as OSDI, NSDI, SOSP and SIGCOMM see extreme load across all machines in the testbed typically leading to all researchers failing to run their experiments.

The opposite of free-riding is referred to as altruism. Altruists care about the community and are the backbone of a sustainable and healthy community. A good example of this is the Wikipedia community with a small (compared to readers) but very dedicated group of editors maintaining the order and quality of the information provided. The opposite of the tragedy of the Commons is the network effect where more users lead to greater benefits to the community, e.g. by providing more content as in the Wikipedia case.

The balance between free-riders and altruists as well as the regulations and pricing of resource usage determines whether the tragedy of Commons or the network effect prevails.

This concept is closely related to what economists refer to as externality [28], individual actions impose an unforeseen positive or negative side-effect on the society. The archetypical example is factory pollution. Such side-effects are mainly addressed in the Cloud by various infrastructure isolation designs such as virtual machines, or virtual private networks (see discussion in the section on Multi-tenancy above).

Incentive compatibility

One of the most frequently overlooked aspects of distributed systems is incentive compatibility [29]. Yet it is a property that all successful large-scale systems adhere to, the Cloud being no exception, and it is very often the main reason why proposed systems fail to take off. It is a concept borrowed from game-theory. In essence, an incentive compatible system is a system where it is in the interest of all rational users to tell the truth and to participate. In a systems context, not telling the truth typically means inserting incorrect or low quality content into the system to benefit your own interests. Incentive to participate is closely related to the notion of free-riding. If there is no incentive to contribute anything to a common pool of resources, the pool will eventually shrink or be overused to the point where the system as a whole becomes unusable. That is, the system has converged to a tragedy of the Commons. Ensuring that the system cannot be gamed is thus equivalent to ensuring that there is no free-riding and that all users contribute back to the community the same amount of valuable resources that they take out. A new, untested, system with a small user base also has to struggle with a lack of trust, and in that case it is particularly important to come out favorable in the individual cost-benefit analysis, otherwise the potential users will just pick another system. Tit-For-Tat (TFT) is an example of an incentive compatible algorithm to ensure a healthy and sustainable resource sharing system.

If Cloud resources are sold at market prices it ensures incentive compatibility,.i.e. ensuring that the price is following the demand (in the case of a spot market) or the expected demand (in the case of a reservation market) closely has the effect of providing an incentive for both suppliers and consumers to participate in the market. Earlier systems such as the Grid and P2P systems that did not have an economic mechanism to ensure incentive compatibility has historically had a much harder time of sustaining a high level of service over a long period of time due to frequent intentional and non-intentional free-riding abuses. Hence, demand-based pricing helps ensure incentive-compatibility.

Computational markets that have demand-driven pricing may however still not be incentive compatible. If it for instance is very cheap to reserve a block of resources ahead of time and then cancel it before use, it could lead to an artificial spike in demand that could dissuade potential customers from using the resource. This in turn would lead to the spot market price being lower, which could benefit the user who put in the original reservation maliciously. In economic terms, it is a classic example of someone not telling the truth (revealing their true demand in this case) in order to benefit (getting cheaper spot market prices). Another classic example is an auction where the bidders may overpay or underpay for the resource, just to make sure competitors are dissuaded to participate or to falsely signal personal demand.

Efficiency

Shared resource clusters such as the Grid are commonly monitored and evaluated based on systems metrics such as utilization. A highly utilized system meant the resources typically funded by central organizations such as governments were being efficiently used. This type of efficiency is referred to as computational efficiency. It is a valuable metric to see whether there are opportunities to pack workloads better or to re-allocate resources to users who are able to stress the system more, i.e. a potential profit opportunity (see the section above on Over and under provisioning). In a commercial system such as the Cloud it is also important to consider the value that the system brings to the users, because the more value the system brings to users the more they are willing to pay and the higher profit the Cloud provider is able to extract from a resource investment. This trade-off becomes apparent when considering a decision to allocate a resource to a user who is willing to pay $0.1 an hour for some resource and utilize at close to 100% versus another user who is willing to use the same resource over the same period of time but at 90% utilization and paying $0.5 an hour. There is likely more idle time and unused resources if the second user is accommodated but the overall profit will be higher (0.5-0.1=$0.4/hour).

To evaluate the economic efficiency [30] one therefore often goes beyond pure system metrics. In economics, utility functions are used to capture the preferences or the willingness of a user to pay for a resource. Maximizing the overall utility across competing users is then a common principle to ensure an overall healthy and sustainable ecosystem. This sum of utilities across all users is referred to as the social welfare of the system. To compare two systems or two resource allocation mechanisms for the same system one typically normalizes the social welfare metric by comparing the value to an optimal social welfare value. The optimal social welfare value is the value obtained if all users (in the case of no contention) or the highest paying user receive all the resources that they desire. Economic efficiency is defined as the optimal social welfare over the social welfare obtained using an actual allocation strategy. A system with an economic efficiency of 90%, for instance have some opportunity, to allocate resource to higher paying users and thereby extract a higher profit.

In essence, ensuring economic efficiency involves optimizing social welfare.

There is however an argument to be made that always allocating to the highest paying user does not create a healthy sustainable ecosystem, which we will discuss next.

Fairness

Consider the case where some user constantly outbids a user by $.0001 every hour in a competitive auction for resources. An economically efficient strategy would be to continuously allocate the resource to the highest bidder. The bidder who keeps getting outbid will however at some point give up and stop bidding. This brings demand down and the resource provider may lose out on long term revenue. It is hence also common practice to consider the fairness of a system. In economics, a fair system is a defined in terms of envy between users competing for the same resource [31]. Envy is defined as the difference in utility that a user received for the actual allocation obtained compared to the maximum utility that could have been obtained across all allocations for the same resource to other users. The metric is referred to as envy-freeness and a fair system tries to maximize envy freeness (minimize envy). Having high fairness is important to maintain loyal customer, and it may in some cases be traded off against efficiency as seen in the example above. Fairness may not be efficient to obtain in every single allocation instance, but is commonly evaluated over a long period of time. For example a system could keep track of the fairness deficit of each user and try to balance it over time to allocate resources to a user that has the highest fairness deficit when resources become available.

In addition to fairness considerations, there could be other reasons why a resource seller may want to diverge from a pure efficiency-optimizing strategy. If information is imperfect and the seller needs to price goods based on the expected willingness to pay by consumers, it may be a better long-term strategy to set the price slightly lower to avoid the dire effects of losing trades by setting the price to high. Another reason may be that some consumers have less purchasing power than others, and giving them benefits, so they can stay in the market, improves the overall competitiveness (and liquidity, see below) of the market, which in turn forces the richer consumers to bid higher.

Liquidity

The central assumption in variable pricing models (see the section above on Variable pricing) is that the price is a proxy or a signal for demand. If this signal is very accurate, allocations can be efficient and incentives to use versus back off of resources are well aligned. If there are too few users competing for resources the prices may plummet and the few users left may get the resource virtually for free. It is therefore critical for a provider to have enough competing users and to have enough purchases of resources for all the market assumption to come into play. In particular, this means ensuring that the second part of incentive compatibility is met, i.e. users have an incentive to participate. Most providers fall back on fixed pricing if there is too little competition, but that may lead to all the inefficiency that variable pricing is designed to address. In economics, this volume of usage and competition on a market is referred to as liquidity [32]. Lack of liquidity is a very common reason for market failure, which is why many financial and economic markets have automated traders to ensure that there is a trade as long as there is a single bidder who sets a reasonable price. A provider may, for instance, put in a daemon bidder to ensure that resources are always sold at a profit.

Predictability

The biggest downside of variable pricing models is unpredictability. If the price spikes at some time in the future, the allocation may have to drop even though the demand is the same to avoid breaking the budget. Exactly how much budget to allocate to resources depends on the predictability of the prices, i.e. the demand. If the demand is flat over time, very little excess budget has to be put aside to cope with situations where resources are critically needed and demand and prices are high. On the other hand, if some application is not elastic enough to handle resource variation, e.g. nodes being de-allocated because the price is too high, a higher budget may need to be allocated to make sure the application runs at some minimal level of allocation.

Essentially users as well as applications have different sensitivity to risk of losing resource allocations or resources being more expensive. In economics the attitude towards risk is described in the risk-averseness or risk attitude property of a user. There are three types of users that differ in how much they are willing to spend to get rid of risk (variation) [33]. Risk-averse users will spend more money than the expected uncertain price (i.e. hedge for future spikes c.f. the discussion on over-provisioning and under- provisioning) [34]. Risk-neutral users will spend exactly the expected price. Finally, risk-seekers will put in a lower budget than the expected price to meet their allocation needs (see Figure 6). An application that is perfectly elastic and that may scale down or up over time as long as the long term performance is guaranteed may choose a risk neutral strategy. Risk seekers are less common in computational markets, but they may be bettering on demand going down in the future. Risk-averse users are the most common group, and the premium they pay above the expected price is a good indicator for how much a resource provider can charge for reservations, which essentially eliminates this uncertainty.

In summary, the elasticity of a Cloud application is highly related to the risk-aversion of the resource purchase, i.e. how much to pay to hedge uncertainty.