Advances, Systems and Applications
Table 1 Existing benchmarks studies for performance evaluation
From: Towards a unified customer aware figure of merit for CSP selection
Study | Benchmarks | Metric/unit |
|---|---|---|
[26] | WSTest | Overall performance/transactions per second (jobs/s) |
[6] | GridNPB/ED + NAS/NGB | Overall performance/transactions per second (jobs/s) |
[27] | Scientific workflow | Overall performance/transactions per |
application | second (jobs/s) | |
[28] | Modified Grenchmark | Overall performance/queue waiting and response time per second |
Developed by the authors | Elasticity/VM acquisition and release time (s) | |
[10] | TPC-E | Overall performance/average transaction time (s) |
Unknown | Elasticity/VM acquisition and release time and time per web role action (s) | |
[11] | Developed by the authors | Elasticity/VM acquisition and release time (s) |
[29] | CPU-intensive web | CPU/duration of operation (s) |
Database read/write intensive | Disk storage/duration of operation (s) | |
[30] | Phoronix/crafty, dcraw | CPU/test duration, MFLOPS |
[31] | TORCH/Dhrystone, spectral, particle | CPU/total execution time (s) |
[32] | EnKF-based matching | CPU/total execution time (s) |
[8] | Bonnie++ | Disk storage/speed (KB/s) |
[8] | Compilation Linux Kernel | CPU/duration of operation (s) |
iperf | Network/transfer rate (KB/s) | |
ping | Network/RTT (ms) | |
[5] | CSFV, NPB, HPCC/HPL | CPU/total execution time (s), GFLOPS |
iperf | Network/message latency (s), TCP throughput (bps) | |
HPCC/HPL, DGEMM | CPU/GFLOPS | |
HPCC/Random Access | Network/speed (MB/s) | |
HPCC/Stream, Cachebench, HPCC/ b eff | Memory/GB/s |