Increasing virtual machine security in cloud environments

Update checker

Since the primary goal of the Update Checker is detecting obsolete software in (dormant) virtual machines, the term virtual machine is used throughout this article. Nevertheless, the solution is applicable to physical machines as well.

The concept of the Update Checker is to build a central database that contains all the information required for the task of checking for updates. This includes the list of installed packages, including the exact version of the installed package as well as the list of repositories that are used for each virtual machine. This information has to be imported into the central database when the virtual machine is first registered, and updated after each change of the virtual machine, i.e., after the installation of new software or the update of already installed software.

Since the Update Checker is not targeted at a single Linux distribution (compared to, e.g., Landscape for Ubuntu [8]), at least the two prevalent software management solutions are supported: apt/dpkg, used for example in Debian and Ubuntu, as well as yum/rpm, used for example in Red Hat and Fedora as well as SuSE. Both solutions use a specific package database format as well as a specific repository format. While apt/dpkg uses the same plaintext file format both as package database and as repository database, yum/rpm uses a Berkeley database as package database and an XML file as repository database. Nevertheless, this has no influence on the structure of the database used to store the required information, since both systems have the concept of distinct package names and a consistent versioning scheme in common.

The design of the solution is shown in Figure 1. There are specific importers for the package databases and for the repository databases of the different software management solutions. This makes the Update Checker easily adaptable to other software management solutions. Information about the installed packages of a virtual machine is stored in the Package DB. Metadata about the VM, i.e., the time stamp of the import, the repositories used, etc., is stored in the Metadata DB. Information about the available packages on the different repositories is stored in the Repository Cache. When invoked, the Update Checker takes the information from these databases and the Repository Cache and matches installed and available packages to detect obsolete software and stores the results in the Result Cache.

When a query for the state of one or more virtual machines is issued, the Update Checker first checks to see if the result of that query is already available in the Result Cache and returns the cached result if it is not obsolete. Cached results are considered obsolete after a configurable amount of time, depending on factors such as the frequency of updates or the need for security. Otherwise, it checks if the package lists of all repositories assigned to the virtual machine are available in the Repository Cache and not obsolete, i.e., the configured validity period has not yet expired. If this is not the case, the package lists are downloaded from the software vendor’s repository, parsed and stored in the Repository Cache for future use. When using the Repository Cache instead of the real repositories, there is the chance that the Update Checker fails to identify an outdated package. Nevertheless, the Repository Cache is very useful for checking many virtual machines and by using a small validity period, the risk can be minimized. Finally, the actual check of the virtual machine is started, comparing the version of each installed package with the version available at the repository. Information about outdated packages is then stored in the Result Cache, so that subsequent queries regarding the same virtual machine can be answered faster.

To help the user to judge whether the identified outdated software poses a risk to the virtual machine, the Update Checker infers information about the priority of an update. Unfortunately, there is no common way to do this for multiple distributions. As a first approach, the source repository of the updated packages is evaluated, since distributions like Debian or Ubuntu use special repositories for security updates. The source of an update can therefore be used as a hint of its significance.

The Update Checker allows to query for the number of available updates for a single or multiple virtual machines as well as for details about the outdated packages and available updates for a single virtual machine. The former query allows a good estimation of the state of the virtual machine, where zero means the virtual machine is up to date, while a number greater than zero means that there are updates available. If significance information is available, individual numbers for each level of significance as well as the sum of the numbers are returned. This can either be used in situations where an overview over a number of virtual machines is required, e.g. , a list of virtual machines in a management tool like the ICS, or as a status check for a specific virtual machine, e.g., before it is started by the XGE.

Since the availability of updates itself allows no judgment about the threat resulting from the outdated packages, even when significance information is available, the latter query allows a detailed examination of the status of a virtual machine, by giving a list of outdated packages. This allows the user of the virtual machine to do a threat analysis based on the outdated packages and decide whether immediate action is required or not. The described functionality is used as an example of the integration of the Update Checker with other components. The complete solution is shown in Figure 2.

Two different interfaces are provided by the Update Checker: a command line interface (CLI) and an API for use by other software. The former can be used, when an administrator manually wants to execute an update check or register a virtual machine. The latter is provided for other tools like the ICS or XGE, allowing them to easily access the status information. This interface is provided using the language-independent protocol XML-RPC [9], to be available to tools written in any language.

The Update Checker can also be configured to run the checks at regular intervals, e.g., daily or weekly. This speeds up queries by other tools, because the information is already available. Users can be informed about obsolete software in their virtual machines via email. Additionally, administrators can also be informed about all virtual machines using obsolete software, to get an overview of the security of all virtual machines running on their infrastructure.

To ease the registration of virtual machines, the remote importer is provided (see Figure 1). It uses software management solution specific Data Collectors to gather the information required for the Update Checker, sends it to the machine the Update Checker is running on and triggers the registration process.

It might seem cumbersome to manually re-register virtual machines after every change, but with the remote importer it is merely a single command. Furthermore, it can be easily automated when software for management and maintenance of virtual machines is used.

Online penetration suite

This section presents the Online Penetration Suite (OPS) to scan an arbitrary number of virtual machines for security vulnerabilities utilizing multiple security scanners. The OPS combines and interprets the different results and generates a machine-readable and a human-readable report. Furthermore, the OPS is able to manage (start, stop, migrate, etc.) virtual machines if necessary. This allows automatic testing of virtual machines in a virtualized infrastructure to detect known security vulnerabilities. Once the vulnerabilities are known, the administrators and users can fix them to protect their systems with respect to unwanted attacks.

Architecture

The OPS is divided into two parts: the logic part, containing the flow control and the report generator, and the backend part, operating the registered vulnerability scanners and the virtual machines. The architecture of the OPS is shown in Figure 3, containing two adapters for OpenVAS [10] and Nessus [11].

The OPS Logic module controls the processes of the OPS. It configures the security scanners, boots the virtual machines to test (if required) and starts the actual scans. Since the vulnerability scanners are basically third-party products with individual characteristics and modes of operation, they are abstracted by Adapters that hide the differences and provide an unified interface to start and monitor the vulnerability scanners. They allow the OPS not only to start the actual scans, but also to watch the scanners during the execution to detect any failures and react accordingly.

For a scan, the OPS needs two input parameters: the names of the target virtual machines and the name(s) of one or more vulnerability scanners. If no scanners are provided, the OPS chooses all scanners by default. A name uniquely identifies a virtual machine and allows the OPS to obtain further information like the IP and MAC address, path to the disk image(s), etc.

The Report Generator module collects the reports from the different scanners and generates the final result: a summary, containing the number of detected vulnerabilities categorized by a risk factor, and a combined report, containing the results from the security scanners in a unified format. To enable the Report Generator to analyze and understand the reports, the adapters have to convert the reports from the native format of the scanner to the unified OPS format.

The backend part of OPS consists of adapters to the required tools and libraries. It provides a module to control virtual machines using the libvirt [12] library as well as the vulnerability scanner adapters. Currently, the OPS supports two different scanners: OpenVAS [10] and Nessus [11], both well-known and established security-products.

Update checker

This section describes important parts of the implementation of the Update Checker, working from the top to the bottom of Figure 1. First, the machine and repository importers and their sources of information are described using the Debian Package Manager (dpkg) and the Advanced Packaging Tool (apt) of Debian and its derivates as an example. Afterwards, the internal databases and caches, the Scan Engine and the different interfaces are described. This section is concluded with details about the remote importer and the integration with other components. Further implementation details can be found in a previously published paper [13] of the authors. The implementation of the Update Checker has been done using the Ruby programming language.

Online penetration suite

The Online Penetration Suite is implemented in the Java programming language. Virtual machines are controlled using the Java binding of the libvirt library, the Nessus scanner is invoked using the Apache XML-RPC library and the reports of the vulnerability scanners are processed and converted using the Java API for XML Processing (JAXP).

Depending on the test configuration specified via the command line, the OPS frontend selects the required vulnerability scanners, starts their server components (if required), boots the virtual machines to scan (if they are not running already) and finally initiates and monitors the actual scan processes. All of these operations are hidden behind an interface that is implemented by the adapters, making the OPS easily extensible with new scanners. Since the report generation process is based entirely on reports in the unified OPS format, no vulnerability scanner dependent code is required for this step in the frontend.

The adapters use different techniques to control and monitor the actual vulnerability scanners. OpenVAS provides a command line interface, so its adapter needs to create a test configuration in the form of an XML file and pass it as an argument to the omp binary. Monitoring of OpenVAS requires analyzing the output of its client. For Nessus, the provided XMLRPC API is used. It contains methods to start and monitor the actual scan process. Both adapters contain code to convert the proprietary report formats into the unified OPS format.

Update checker

Measurements have been conducted to evaluate the Update Checker on an Intel Xeon E5220 machine with 1 GB memory. The first measurement is a local measurement testing all components of the Update Checker, i.e., machine import, repository import and update checking. Three Debian and three Fedora virtual machines have been used in this test, with varying numbers of installed packages and enabled repositories. Each test has been executed 20 times and average values have been calculated. The results are shown in Table 1.

In the first part of this evaluation, the different machine importers were tested. All required files were copied to the machine the test was executed on prior to the evaluation, thus no network communication is involved. Furthermore, before the measurement rpm -qa was executed on the source machine to generate a list of installed packages including their version. This is required to work around incompatibilities (i.e., the rpm binary on Debian squeeze could not read the rpm database of a Fedora 15 installation).

The growing import times can be explained with the growing number of installed packages that must be parsed.

The second part of the test measured the time required to download and parse all repository databases for the virtual machines (each machine had between 2 and 4 repositories configured) without using the repository cache. The times measured are thus artificial and are only of little relevance for actual usage, but allow evaluating the repository import and update checking. While the times for the Debian machines are quite stable, the increase of the time for Fedora is caused by the number of repositories used (2, 3 and 4, respectively). The very bad performance of the Fedora repository import is caused by the use of XML in the repository database.

The last part of the test evaluates the algorithm that actually checks for updates. Again, the increase in the times is caused by the growing number of packages. The reason for the worse results for Fedora are probably the longer and more complex version numbers used in Fedora, making the comparison harder and more time-consuming.

The measured values are promising. Checking for updates is a very fast process with the Update Checker. Because of the individual files used for the Package DB and Repository Cache, we do not expect performance degradation when the number of virtual machines increases. The relatively long time required for importing yum repositories is compensated by the repository cache, that results in every repository being downloaded and parsed only once during the configurable validity period of the cache.

To evaluate the influence of the repository cache, another measurement has been conducted that represents a more realistic scenario: checking all imported virtual machines for updates. The six machines from the last measurement were checked at once, taking advantage of the repository cache. The experiment was repeated 20 times and the average times are shown in Figure 4. The results indicate that the repository cache is very effective in cutting down the time required to check multiple virtual machines for updates.

To evaluate the scalability (and applicability for physical machines) of the Update Checker, 115 physical nodes from our compute cluster were imported. All machines were checked at once using the repository cache. The experiment was repeated 20 times and the time required to check all virtual machines was calculated. The results shown in Figure 5 provide evidence for the scalability of the Update Checker. The average check time was 34.53 seconds for all 115 machines, that is 0.30 seconds per machine.

Another measurement was conducted to evaluate the import time of the virtual machines, when the remote importer is used. This involves gathering all required files, executing rpm -qa in the case of rpm based distributions, sending everything to the Update Checker and starting the import process. For each virtual machine, 10 imports were executed. The results are shown in Figure 6.

As expected, the amount of time the import process requires grows with the number of packages in the database. Generally, the import process is faster for apt/dpgk based virtual machines than for yum/rpm based virtual machines. The source of this problem seems to be the use of the rpm binary to extract the information from the database.

Online penetration suite

The following section presents measurements related to the OPS. All tested systems are Xen domainU virtual machines running Debian Squeeze and located on Pentium IV systems with 1 GB memory. The OPS node is an Intel Xeon E5220 machine and 1 GB memory. All systems are interconnected with switched fast Ethernet.

The first experiment measures the total runtime of the OPS depending on the number of virtual machines. Figure 7 shows the results. The OPS used both vulnerability scanners in parallel while the number of target virtual machines was increased with every run. To get a robust mean, 100 trials were performed. Testing one virtual machine took 684 seconds on average, testing two machines took 859 seconds, testing three machines 1056 seconds, and it took 1279 seconds to test all four machines. Obviously, the measurement reveals that the runtime increases linearly with the number of tested systems. Furthermore, it reveals that it is more efficient to test multiple targets in parallel instead of scanning one after another.

In order to test the efficiency of the OPS, multiple tests against virtual machines running different versions of the Debian operating systems were conducted. The unpatched release version of Debian Etch (released April 2007), Lenny (released February 2009), Squeeze (released February 2011) and Wheezy (current unstable version) were used. The results of the tests are shown in Table 2.

The OPS successfully revealed a number of security vulnerabilities in all tested versions, including two high-risk flaws in each version. Debian Etch is the oldest release and contains the lowest number of vulnerabilities because it contains less features (in terms of installed services) than all other versions. Other flaws are related to the installed kernel version. The flaws appeared with newer kernel versions and thus, only in newer Debian versions.