Khan M, Becker T, Kuppuudaiyar P, Elster AC (2018) Container-Based Virtualization for Heterogeneous HPC Clouds: Insights from the EU H2020 CloudLightning Project In: 2018 IEEE International Conference on Cloud Engineering (IC2E), 392–397.. IEEE, Piscataway.
Chapter
Google Scholar
Rodriguez MA, Buyya R (2019) Container-based cluster orchestration systems: A taxonomy and future directions. Softw Pract Experience 49(5):698–719. https://doi.org/10.1002/spe.2660.
Article
Google Scholar
Abdollahi Vayghan L, Saied MA, Toeroe M, Khendek F (2018) Deploying Microservice Based Applications with Kubernetes: Experiments and Lessons Learned In: 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), 970–973.. IEEE, Piscataway.
Chapter
Google Scholar
Casalicchio E (2017) Autonomic Orchestration of Containers: Problem Definition and Research Challenges In: Proceedings of the 10th EAI International Conference on Performance Evaluation Methodologies and Tools on 10th EAI International Conference on Performance Evaluation Methodologies and Tools. VALUETOOLS16, 287–290.. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), Brussels, BEL. https://doi.org/10.4108/eai.25-10-2016.2266649.
Google Scholar
Hovestadt M, Kao O, Keller A, Streit A (2003) Scheduling in HPC Resource Management Systems: Queuing vs. Planning. In: Feitelson D, Rudolph L, Schwiegelshohn U (eds)Job Scheduling Strategies for Parallel Processing, 1–20.. Springer Berlin Heidelberg, Berlin.
Google Scholar
Klusáček D, Chlumský V, Rudová H (2015) Planning and Optimization in TORQUE Resource Manager In: Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing.. Association for Computing Machinery, New York. https://doi.org/10.1145/2749246.2749266.
Google Scholar
Jette MA, Yoo AB, Grondona M (2002) SLURM: Simple Linux Utility for Resource Management In: In Lecture Notes in Computer Science: Proceedings of Job Scheduling Strategies for Parallel Processing (JSSPP) 2003, 44–60.. Springer-Verlag, Berlin.
Google Scholar
Staples G (2006) TORQUE Resource Manager In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing.. Association for Computing Machinery, New York. https://doi.org/10.1145/1188455.1188464.
Google Scholar
Moab HPC Suite. https://support.adaptivecomputing.com/wp-content/uploads/2019/06/Moab-HPC-Suite_datasheet_20190611.pdf. Accessed 08 July 2020.
Mateescu G, Gentzsch W, Ribbens CJ (2011) Hybrid Computing-Where HPC Meets Grid and Cloud Computing. Future Gener Comput Syst 27(5):440–453. https://doi.org/10.1016/j.future.2010.11.003.
Article
Google Scholar
Mayer R, Jacobsen HA (2020) Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques, and Tools. ACM Comput Surv 53(1). https://doi.org/10.1145/3363554.
Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, et al. (2016) TensorFlow: A System for Large-scale Machine Learning In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. OSDI’16, 265–283.. USENIX Association, Berkeley. http://dl.acm.org/citation.cfm?id=3026877.3026899.
Google Scholar
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, et al. (2019) PyTorch: An Imperative Style, High-Performance Deep Learning Library. In: Wallach HM, Larochelle H, Beygelzimer A, d’Alché-Buc F, Fox EB, Garnett R (eds)Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, 8024–8035, Vancouver. https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html.
Brayford D, Vallecorsa S, Atanasov A, Baruffa F, Riviera W (2019) Deploying AI Frameworks on Secure HPC Systems with Containers In: 2019 IEEE High Performance Extreme Computing Conference (HPEC), 1–6.. IEEE, Piscataway.
Google Scholar
Hale JS, Li L, Richardson CN, Wells GN (2017) Containers for Portable, Productive, and Performant Scientific Computing. Comput Sci Eng 19(6):40–50.
Article
Google Scholar
Felter W, Ferreira A, Rajamony R, Rubio J (2015) An updated performance comparison of virtual machines and Linux containers In: 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 171–172.. IEEE, Piscataway.
Chapter
Google Scholar
Bernstein D (2014) Containers and Cloud: From LXC to Docker to Kubernetes. IEEE Cloud Comput 1(3):81–84.
Article
Google Scholar
Martin JP, Kandasamy A, Chandrasekaran K (2018) Exploring the Support for High Performance Applications in the Container Runtime Environment. Hum-Centric Comput Inf Sci 8(1). https://doi.org/10.1186/s13673-017-0124-3.
Plauth M, Feinbube L, Polze A (2017) A Performance Survey of Lightweight Virtualization Techniques. In: De Paoli F, Schulte S, Broch Johnsen E (eds)Service-Oriented and Cloud Computing, 34–48.. Springer International Publishing, Cham.
Chapter
Google Scholar
Zhang J, Lu X, Panda DK (2017) Is Singularity-Based Container Technology Ready for Running MPI Applications on HPC Clouds? In: Proceedings of The10th International Conference on Utility and Cloud Computing. UCC 17.. Association for Computing Machinery, New York. https://doi.org/10.1145/3147213.3147231.
Google Scholar
Hu G, Zhang Y, Chen W (2019) Exploring the Performance of Singularity for High Performance Computing Scenarios In: 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), 2587–2593.. IEEE, Piscataway.
Chapter
Google Scholar
Younge AJ, Pedretti K, Grant RE, Brightwell R (2017) A Tale of Two Systems: Using Containers to Deploy HPC Applications on Supercomputers and Clouds In: 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), 74–81.. IEEE, Piscataway.
Chapter
Google Scholar
VMWare (2018) Containers on Virtual Machines or Bare Metals?. VMware, Inc., Palo Alto. https://assets.contentstack.io/v3/assets/blt58b49a8a0e43b5ff/blta366cfae83d85681/5c742ba62617ffd7604a143c/vmwwp-containers-on-vms.pdf.
Google Scholar
Merkel D (2014) Docker: Lightweight Linux Containers for Consistent Development and Deployment. Linux J 2014(239):76–90.
Google Scholar
Kurtzer GM, Sochat VV, Bauer M (2017) Singularity: Scientific containers for mobility of compute In: PloS one.. PLOS, San Francisco.
Google Scholar
Gerhardt L, Bhimji W, Canon S, Fasel M, Jacobsen D, Mustafa M, et al. (2017) Shifter: Containers for HPC. J Phys Conf Ser 898:082021. https://doi.org/10.1088%2F1742-6596%2F898%2F8%2F082021.
Article
Google Scholar
Priedhorsky R, Randles T (2017) Charliecloud: Unprivileged Containers for User-Defined Software Stacks in HPC In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC 17.. Association for Computing Machinery, New York. https://doi.org/10.1145/3126908.3126925.
Google Scholar
S SK (2017) Practical LXC and LXD: Linux Containers for Virtualization and Orchestration. 1st ed. Apress, USA.
Google Scholar
Gropp W, Lusk E, Skjellum A (1994) Using MPI: Portable Parallel Programming with the Message-passing Interface.. MIT Press, Cambridge.
Xavier MG, Neves MV, Rossi FD, Ferreto TC, Lange T, De Rose CAF (2013) Performance Evaluation of Container-Based Virtualization for High Performance Computing Environments In: 2013 21st Euromicro International Conference on Parallel, Distributed, and Network-Based Processing, 233–240.. IEEE, Piscataway.
Chapter
Google Scholar
Casalicchio E (2019) Container Orchestration: A Survey. In: Puliafito A (ed)Systems Modeling: Methodologies and Tools, 221–235.. Springer International Publishing, Cham. https://doi.org/10.1007/978-3-319-92378-9_14.
Chapter
Google Scholar
Hightower K, Burns B, Beda J (2017) Kubernetes: Up and Running Dive into the Future of Infrastructure. 1st ed.. OReilly Media, Inc., Sebastopol.
Casalicchio E, Iannucci S (2019) The state-of-the-art in container technologies: Application, orchestration and security. Concurrency Comput Pract Experience 32(17):e5668. https://onlinelibrary.wiley.com/doi/abs/10.1002/cpe.5668.
Google Scholar
Pandey S, Tokekar V (2014) Prominence of MapReduce in Big Data Processing In: 2014 Fourth International Conference on Communication Systems and Network Technologies, 555–560.. IEEE, Piscataway.
Chapter
Google Scholar
Zaharia M, Xin RS, Wendell P, Das T, Armbrust M, Dave A, et al. (2016) Apache Spark: A Unified Engine for Big Data Processing. Commun ACM 59(11):56–65. http://doi.acm.org/10.1145/2934664.
Article
Google Scholar
Narkhede N, Shapira G, Palino T (2017) Kafka: The Definitive Guide Real-Time Data and Stream Processing at Scale. 1st ed. O’Reilly Media, Inc., Sebastopol.
Google Scholar
Sammons G (2016) Exploring Ansible 2: Fast and Easy Guide. CreateSpace Independent Publishing Platform, North Charleston.
Google Scholar
Gao PX, Narayan A, Karandikar S, Carreira J, Han S, Agarwal R, et al. (2016) Network Requirements for Resource Disaggregation In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. OSDI|16, 249–264.. USENIX Association, USA.
Google Scholar
Zhou N, Georgiou Y, Zhong L, Zhou H, Pospieszny M (2020) Container Orchestration on HPC Systems In: 2020 IEEE International Conference on Cloud Computing (CLOUD).. IEEE, Piscataway.
Google Scholar
Julian S, Shuey M, Cook S (2016) Containers in Research: Initial Experiences with Lightweight Infrastructure In: Proceedings of the XSEDE16 Conference on Diversity, Big Data, and Science at Scale. XSEDE16.. Association for Computing Machinery, New York. https://doi.org/10.1145/2949550.2949562.
Google Scholar
Higgins J, Holmes V, Venters C (2015) Orchestrating Docker Containers in the HPC Environment. In: Kunkel JM Ludwig T (eds)High Performance Computing, 506–513.. Springer International Publishing, Cham.
Chapter
Google Scholar
Liu F, Keahey K, Riteau P, Weissman J (2018) Dynamically Negotiating Capacity between On-Demand and Batch Clusters In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis. SC 18.. IEEE Press, Piscataway.
Google Scholar
Piras ME, Pireddu L, Moro M, Zanetti G (2019) Container Orchestration on HPC Clusters. In: Weiland M, Juckeland G, Alam S, Jagode H (eds)High Performance Computing, 25–35.. Springer International Publishing, Cham.
Chapter
Google Scholar
Fernandez GP, Brito A (2019) Secure Container Orchestration in the Cloud: Policies and Implementation In: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing. SAC 19, 138–145.. Association for Computing Machinery, New York. https://doi.org/10.1145/3297280.3297296.
Chapter
Google Scholar
Maenhaut PJ, Volckaert B, Ongenae V, De Turck F (2019) Resource Management in a Containerized Cloud: Status and Challenges. J Netw Syst Manag 28:197–246.
Article
Google Scholar
Buyya R, Srirama SN (2019) A Lightweight Container Middleware for Edge Cloud Architectures. Wiley Telecom. https://ieeexplore.ieee.org/document/8654087.
Hindman B, Konwinski A, Zaharia M, Ghodsi A, Joseph AD, Katz R, et al. (2011) Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center In: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation. NSDI 11, 295–308.. USENIX Association, USA.
Google Scholar
Wrede F, von Hof V (2017) Enabling Efficient Use of Algorithmic Skeletons in Cloud Environments: Container-Based Virtualization for Hybrid CPU-GPU Execution of Data-Parallel Skeletons In: Proceedings of the Symposium on Applied Computing. SAC 17, 1593–1596.. Association for Computing Machinery, New York. https://doi.org/10.1145/3019612.3019894.
Chapter
Google Scholar
Ciechanowicz P, Poldner M, Kuchen H (2009) The Münster Skeleton Library Muesli: A comprehensive overview. University of Münster, European Research Center for Information Systems (ERCIS). Available from: https://www.ercis.org/sites/www.ercis.org/files/pages/research/ercis-working-papers/ercis_wp_07.pdf.
Pisaruk V, Yakovtseva SWLM-operator. Gitlab. https://github.com/sylabs/wlm-operator. Accessed 13 Feb 2020.
Georgiou Y, Zhou N, Zhong L, Hoppe D, Pospieszny M, Papadopoulou N, et al. (2020) Converging HPC, Big Data and Cloud technologies for precision agriculture data analytics on supercomputers In: 15th Workshop on Virtualization in High-Performance Cloud Computing (VHPC’20).. Springer International Publishing, Cham.
Google Scholar
Howard J, Gugger S (2020) Fastai: A Layered API for Deep Learning. Information 11(2):108. https://doi.org/10.3390/info11020108.
Article
Google Scholar
Breiman L (2001) Random forests. Mach Learn 45(1):5–32.
Article
MATH
Google Scholar
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. (2011) Scikit-Learn: Machine Learning in Python. J Mach Learn Res 12:2825–2830.
MathSciNet
MATH
Google Scholar
Drusch M, Bello UD, Carlier S, Colin O, Fernandez V, Gascon F, et al. (2012) Sentinel-2: ESA’s Optical High-Resolution Mission for GMES Operational Services. Remote Sens Environ 120:25–36. The Sentinel Missions - New Opportunities for Science. http://www.sciencedirect.com/science/article/pii/S0034425712000636.
Salakhutdinov R, Mnih A (2008) Bayesian probabilistic matrix factorization using Markov chain Monte Carlo In: Proceedings of the International Conference on Machine Learning. vol. 25.. Association for Computing Machinery, New York.
Google Scholar
Aa TV, Chakroun I, Haber T (2016) Distributed Bayesian probabilistic matrix factorization In: CLUSTER, 346–349.. IEEE Computer Society, Piscataway.
Google Scholar
MPI: A Message-Passing Interface Standard. https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report.pdf. Accessed 26 Jan 2021.
Graham RL, Woodall TS, Squyres JM (2005) Open MPI: A Flexible High Performance MPI In: Proceedings of the 6th International Conference on Parallel Processing and Applied Mathematics. PPAM 05, 228–239.. Springer-Verlag, Berlin. https://doi.org/10.1007/11752578_29.
Google Scholar
SylabsSingularity-CRI. https://sylabs.io/guides/cri/1.0/user-guide/k8s.html. Accessed 03 Mar 2020.
Romana. https://romana.io/. Accessed 21 May 2020.
Sergeev A, Balso MD (2018) Horovod: fast and easy distributed deep learning in TensorFlow. CoRR. abs/1802.05799. Available from: https://arxiv.org/abs/1802.05799.
Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R, et al. (2013) Apache Hadoop YARN: Yet Another Resource Negotiator In: Proceedings of the 4th Annual Symposium on Cloud Computing. SOCC ’13.. Association for Computing Machinery, New York. https://doi.org/10.1145/2523616.2523633.
Google Scholar