Decentralized and scalable hybrid scheduling-clustering method for real-time applications in volatile and dynamic Fog-Cloud Environments

Abstract

A major challenge in Cloud-Fog settings is the scheduling of workflow applications with time constraints as the environment is highly volatile and dynamic. Furthermore, adding the complexities of handling IoT nodes, as the major owners of the workflow requests, renders the problem space even harder to address. This paper presents a hybrid scheduling-clustering method for addressing this challenge. The proposed lightweight, decentralized, and dynamic clustering algorithm is based on fuzzy inference with intrinsic support for mobility to form stable and well-sized clusters of IoT nodes while avoiding global clustering and recurrent re-clustering. The proposed distributed method uses Cloud resources along with clusters of mobile and inert Fog nodes to schedule time-constrained workflow applications with considering a proper balance between contradicting criteria and promoting scalability and adaptability. The Velociraptor simulator (version 0.6.7) has been used to throughtly examine and compare the proposed method in real workloads with two contemporary and noteworthy methods. The evaluation results show the superiority of the proposed method as the resource utilization is about 20% better and the schedule success rate is almost 21% better compared with the two other methods. Also, other parameters such as throughput and energy consumption have been studied and reported.

Introduction

Cloud computing is a promising model of the Internet of Things (IoT) in data processing and business applications [1, 2]. Using Cloud computing for IoT is not an easy task due to challenges such as synchronization, standardization, balancing between services and IoT needs [2,3,4,5,6].

Fog computing can be used as a viable choice for IoT providers with features such as less delay, more scalability, support for user and resource mobility, better performance for real-time interactive services, and data processing [7,8,9]. However, Fog computing needs to solve other issues such as the reliability and mobility of analytical data on edge devices. Another important challenge in Fog computing is resource allocation [10]. In Fog computing, nodes are differently based on input data and processing speed, which leads to problems on load balancing, when some Fog nodes are overloaded and some remain idle. These issues reduce the performance of the Fog environment [11,12,13]. Therefore, workflow scheduling is an important issue in a Fog environment.

Workflow is commonly used in a distributed computing environment [7, 8, 13,14,15]. A workflow is graphically performed using a Directed Acyclic Graph (DAG) to show the interdependence between workflow tasks, where nodes represent computational tasks on available resources and guided edges represent data flow dependency [8, 13,14,15]. In general, the runtime of a task in a particular resource is inversely proportional to the speed of that resource; therefore, optimal mapping of workflow tasks to computing resources available in Cloud/Fog is essential [16, 17].

In this paper, a distributed scheduling architecture for Cloud/Fog-based systems is presented. The proposed Cloud/Fog based scheduling management architecture has three hierarchical layers. The first layer is the IoT device layer. The second layer, has two sub-layers of Fog nodes and smart gateways, which are responsible for managing requests. This layer has a scheduling management system that receives all user requests, manages resources, processing and communication costs, and ultimately creates the most appropriate scheduling available. The third layer, the Cloud layer, hosts some computing nodes and provides external resources for the middle layer. In the proposed architecture, resources are mobile in the system, meaning that they can be in different places at different times. Also, resources have limited energy consumption and bandwidth. Therefore, computational resources should be clustered [18, 19].

In this paper, a new multi-criteria clustering algorithm is presented to manage resources in the Fog layer. The use of clustering algorithms in Cloud/Fog has been considered due to advantages such as request management, increased data transfer speed and energy efficiency. According to studies [18,19,20,21,22] in the field of clustering, the mobility of fog nodes and their stability in creating optimal clusters is very important. So, a proposed clustering algorithm is designed based on the rate of velocity, stability and mobility score that the fog layer nodes obtain using the fuzzy inference system. This makes the dynamic clustering algorithm more stable and efficient, and the requests are placed on closer and higher efficiency clusters.

This paper proposed a new scheduling algorithm. This algorithm includes three phases. The first phase focuses on extracting the Critical Path (CP) from the DAG. The CP algorithm extracts important paths based on the priority value of each task (PRT). The second phase is the new workflow scheduling algorithm (SDMS algorithm) that selects the best schedule according to the workflow deadline and definition of a new utility function. The determined deadline is crucial to improve Quality of Service (QoS) performance in the system. The third phase is task reservation.

Workflows are usually faced with a large number of tasks that must be processed anywhere. This requires that the provided scheduling methods are scalable and able to schedule tasks within a certain deadline. In this regard, many studies have been conducted in recent years on these methods due to the many advantages that cloud/fog environments have. However, there are shortcomings in this field, such as:

• Most of them are focused on unit workflow scheduling in Cloud or Fog environment [7, 8]. If the Cloud/Fog environment is considered at the same time, the benefits of these layers can be achieved.

• Since Fog nodes usually have mobility and the movement characteristics of nodes may lead to continuous changes in the network topology, it is very important to consider mobility. Many of the previous works have not considered it [8,9,10,11,12,13].

• Also, mobile sources in the Fog layer have limited energy and bandwidth. Therefore, clustering is one of the most effective methods to optimize energy consumption and resource management, a case that has not been addressed in many papers [23,24,25,26].

• The critical path is the longest path from the current task to an output task or an input task. Tasks on a critical path have priority over tasks on non-critical paths. Critical path helps to establish and control the schedule and minimize delays. This has been considered in few papers [25,26,27,28,29].

The above-mentioned challenges formed the main motivations of this research and consequently, the contributions of this paper are as follows:

• Providing a three-tier hierarchical architecture for Cloud/Fog-based systems. The proposed architecture introduces a sub-layer in the Fog to provide a scheduling management system with resource mobility.

• Extracting a Critical Path extraction method according to the priority of workflow tasks, which helps to create more accurate schedules.

• Considering the mobility behavior of the Fog layer nodes and calculating the mobility score of the Fog nodes using a multi-criteria fuzzy method and introducing a new dynamic clustering algorithm.

• Proposing a new workflow scheduling algorithm, which achieves the best scheduling using a new utility function.

The continuation of the paper is as follows: In the second section, related work is presented. In the third section, the scheduling management system architecture is presented and in the fourth section, the scheduling model and related algorithms are presented. The fifth section deals with the evaluation and the sixth section with conclusions.

Related work

Workflow scheduling, which aims to meet different needs, has been extensively studied in recent decades. Many algorithms are proposed to find multi-purpose workflow scheduling scenarios [23]. In general, workflow scheduling algorithms are divided into three types: (1) meta-heuristic-based techniques, (2) heuristic, and (3) meta-heuristic hybrid techniques. Many studies of workflow scheduling are performed with multiple competing goals for integrated optimization or by combining multiple goals into a single goal or limiting various other criteria [24]. The main disadvantage of this type of approach is that only one computational solution may not be able to perform the user settings correctly. Recently, some practical multi-purpose solutions for computational scheduling have been developed and received a great deal of attention [23, 24].

A new problem is how to schedule workflows with data privacy restrictions while minimizing runtime and financial costs for large data applications in the Cloud [24]. In [7], such problems are modeled as a multi-objective optimization issue, and a Privacy-Aware multi-purpose scheduling workflow algorithm called MOPA is proposed. The results are compared with two other algorithms.

Arabnejad V et. al [8] explains the issue of scheduling scientific workflow in commercial environments. In this study, workflow scheduling is used to achieve low cost with appropriate response time in Cloud environments. Workflow scheduling in Cloud environments is different from network and cluster computing environments, which are mainly used in providing flexible resources and payment charging models for each user. Therefore, scheduling workflows in the Clouds requires a different approach to mapping tasks to resources while reducing costs.

In [12], a new algorithm for planning called the efficient planning workflow is introduced by establishing a trade-off between time and cost, which includes four main steps: task selection, evaluation of the range of all types of implicitly requested samples (IRITR), evaluation of the additional budget and selection of the virtual machine. IRITR evaluation is a new concept for scheduling, which aims to determine the different types of VM samples that are more suitable for workflow execution to avoid excessive bidding and negative bidding, which leads to budget violations and deadlines respectively. Compared to previous works, the simulation results prove the effectiveness of their approach, especially when there are many different sample types.

ChoonLee Y et. al [23] deals with the issue of resource-efficient workflow scheduling. For this purpose, the Significant Maximum Reduction (MER) algorithm provides a resource productivity solution that optimizes the resource utilization of a work schedule performed by each specific scheduling algorithm. The main innovation of MER depends on identifying its "near-optimal" point between increasing and decreasing resource use. Finding such a point is very important and can lead to improvement in resource utilization, reduction of resource supply, and energy saving. Another significant contribution is the widespread use of MER.

The focus in [25] is on schedule flows in Cloud computing. They present a new algorithm which comprises the Intelligent Water Drop (IWD) algorithm to optimize Cloud workflow scheduling. Their algorithm represents a significant improvement over classic timetable scheduling algorithms.

In [26] a multi-objective optimization problem is presented with a focus on reducing the power consumption of the whole system and delaying the execution of tasks. Overcoming permutation convergence in the initial population and genetic operators are effective factors in this algorithm. The performance of the proposed algorithm has been studied by extensive experiments and the obtained results show that this algorithm performs better than its counterparts in terms of various criteria. In this regard ch-PICEA-g algorithm is presented in [27] which is an innovative multi-objective developed algorithm. In their proposed method, first logistic maps and tents are used and then the improved bodily function is used to achieve the optimal solution.

Han P et. al [28] proposes an efficient exploratory method called CMSWC to solve the workflow scheduling problem by minimizing the cost and length of the workflow. CMSWC has two stages: ranking and mapping. In addition, CMSWC combines specifically for multi-objective challenges: (1) The scheduling phase to prevent the exploration of useless resources for tasks, which significantly limits the search space. (2) A new method for selecting non-dominant solutions, combining the non-dominant fast sorting approach and change-based density estimation. Extensive experiments on real workflows show that their approach can generate better costs than several advanced approaches.

Wang B et. al [29] provides an optimization-based scheduling framework in the Cloud environment. They use the virtual machine multiple queue model and the relationships between them along with the automatic encoder to improve noise cancelation to extract service quality characteristics. In the evaluation results, their method effectively reduces energy consumption compared to others while guaranteeing the quality of service. In [30], due to service quality limitations, the problem of workflow scheduling in multiple grid environments has been considered. To achieve this goal, they have developed a two-tier scheduling strategy that includes general-level scheduling and local-level scheduling. Heterogeneity and all types of resources available in each grid cluster is an issue that significantly affects the complexity of scheduled workflows. Their proposed system consists of a cluster monitoring unit. The program management unit, and the cluster coordination unit. the cluster coordination unit uses the method of determining the quality of resources to prioritize the candidate's resources based on the multi-criteria decision-making problem space. Software simulation is used to evaluate the performance of the proposed system with other works.

Zhou X et. al [31] has proposed the Priority and Relative Distance (EPRD) efficiency algorithm to minimize work scheduling length for priority-limited workflow programs without violating the end-to-end deadline. The algorithm consists of two processes, first creating a task priority queue and then drawing a VM corresponding to its relative distance for a task. The proposed method can effectively improve VM performance and scheduling. Extensive detailed experiments based on workflow schedules are randomly generated and in the real world show that the amount of resource reduction and programming length of the EPRD algorithm significantly exceeds the existing algorithms. In terms of scheduling, scientific flows are compact data during which a large volume of medium data is generated. In the Cloud, some valuable intermediate data is stored for reuse. This storage is done manually according to the capacity of the system. Today, more mediocre data can be stored in the cloud due to the popularity of doing scientific workflow in the workplace.

In [32], they create a data dependency graph of the data resource in scientific flows. Meanwhile, this strategy considers user access due to delay. The results show that this strategy can significantly reduce the general scientific costs of academic operations. In [33], a four-layer architecture in a Fog environment is provided for delay and load balancing in scheduling. Minimum execution time, maximum completion time, arrival time and work size are considered as input criteria. In this study, the K-means +  + clustering algorithm is used to cluster nodes in the Fog layer, and an artificial neural network is used to determine the extent to which a Fog node is used. Finally, they used iFogSim for simulation and the parameters of energy consumption, load balance, response time, scheduling time and latency were used to calculate the performance of the scheduling algorithm.

Pham X-Q et. al [34] considers workflow scheduling based on a Cloud/Fog computing system. Fog layer nodes are used to run large applications in collaboration with Cloud layer nodes. This allows for a good deal of agreement between implementation time and cost. In [35], task scheduling and data placement in systems embedded in the Cloud/Fog environment are investigated. Resource management is always an important issue for system performance. To deal with the complexity of the above calculations, an effective computational solution based on equation has been proposed and validated by extensive simulation studies.

Maio M et. al [36] focuses on scheduling extreme data. Extreme data is a redefinition of big data that is distinguished by the vast amount of information that needs to be analyzed. They use Fog calculations to schedule heavy data workflows that require accurate response times. They propose a beam-based method for loading tasks in the Fog called multi-objective workflow loading. Their algorithm considers three goals: cost, response time, and reliability. The simulation results show examples of the usage of severe data with precise delay requirements. Energy saving is one of the main goals in network, Cloud, and Fog computing.

In [37], an energy-aware method based on the DVFS technique for energy saving in Fog computing is proposed. Power savings are achieved using DVFS-enabled processors in the Fog, and compared to them, they can operate at lower voltages and frequencies at idle times. In addition, the IWO-CA hybrid evolution algorithm is used to arrange tasks without violating priority constraints. According to the results in the evaluation section, significant energy savings are achieved in the calculations of programs that have a predetermined deadline. Allocating and providing resources in the Fog-Cloud environment is a challenging task, given the dynamic changes in user needs and the limited resources available in Fog computing.

In [38], they propose a deadline-based algorithm for resource allocation and provision using resource ranking and resource provisioning in a combined and hierarchical manner. In this study, dynamic changes in user behaviors are investigated. The performance of the proposed algorithms in terms of total data processing time, sample cost, and network delay, with the increasing number of program submissions, are compared with available and better algorithms. The simulation results of their algorithm in CloudSim show the better behavior of this algorithm.

In [39] a workflow scheduling algorithm based on load balancing is proposed. Based on the cloud position, the execution time of the tasks is checked and according to a queue model in the Cloud, the response time of the whole system is reduced. Next, they present their scheduling algorithm using the shortest path of the algorithm. The simulation results show the reduction of energy consumption and the increase of resource utilization. Konjaang KJ et. al [40] have proposed a virtual machine mapping algorithm. Their method is to efficiently manage resources in the Cloud to reduce runtime and cost in order to improve the quality of services and user requirement. The evaluation results show a reduction in execution time, cost and energy consumption.

Pham X-Q et. al [41] in a new distributed computing platform, task scheduling based on Cloud/Fog systems is presented. The main purpose of their algorithm is to balance the cost and performance of runtime, and the results show that the quality of system service is improved. Azizi S et. al [42] presents the scheduling method in a Fog environment in order to reduce total energy consumption and achieve high service quality, while minimizing deadline violations. Through simulation experiments, the superiority of the proposed algorithm is shown in the number of tasks, reducing the deadline violations and optimizing energy consumption.

Rodriguez M-A et. al [43] analyzed various features of Cloud workflow scheduling algorithms. In particular, the planning model, application, and resources have been studied. Since extensive researches have been done on the field of planning in general, there are accepted and accurate classifications for these features. Some of these are planning decisions, planning goals, and optimization strategies. The software model is viewed from the point of view of workflow multiplicity, i.e. the number and type of workflows that algorithms can process. Zaman Khalid, et al. [44] presents a technique for efficient distribution of resources in all nodes in a centralized manner. Their model is established in terms of cost-effectiveness using threshold values. The proposed system reduces cloud copies and costs. Azure's $6 static solution has three nodes. The adaptive data replication management system reduces the number of copies from three to two, saves memory, and lowers the price from $6 to $4. MATLAB and other programming languages are used for simulation. Azure is compared to Date Replication and Reproducer.

In cloud computing, services play a key role. Services are well-defined and independent components. Today, the demand for using fuzzy inference as a service in the field of complex and critical systems is increasing. In such systems, along with software development, the cost of diagnosing and fixing software defects increases. Therefore, the use of formal methods, which provide a clear, concise and mathematical interpretation of the system, is very important for the design of these fuzzy systems. To achieve this goal, [45] introduces a fuzzy inference cloud service (FICS) and proposes a new discipline for its formal modeling. FICS provides fuzzy inference services to consumers. Four new formal verification tests are also provided, which allow detailed analysis of certain behavior discipline patterns in FICS. In [46], a new approach to combine partitioning, sequencing and scheduling algorithms and multi-objective optimization is presented. Their approach integrates a distributed resolution with a multi-agent system and a fuzzy inference system to deal with the uncertainty problem. They compare their approach with the main related approaches and the analysis of the obtained results validates the performance of the approach as well as its effectiveness in scheduling workflows on fog-cloud computing considering resource utilization.

In IoT-based environments, workflows are usually faced with a number of tasks that need to be processed anywhere. This requires that the provided scheduling methods are scalable and able to schedule tasks within a certain time limit. In this regard, many studies have been investigated in recent years on these methods considering the advantages of Cloud/Fog environments. However, there are gaps in this regard. Most of the studies have focused on workflow scheduling in Cloud or Fog environment [8,9,10,11,12,13]. This paper has examined both layers in order to achieve their benefits.

Also, the mobility of nodes in the Fog layer to optimally allocate tasks to resources has become an important challenge in scheduling methods. On the other hand, they have limited energy and bandwidth resources. Therefore, clustering is one of the most effective methods that simultaneously considers the mobility of nodes and their management. In this paper, a new clustering algorithm that takes these challenges into account is presented. The next gap is to consider the critical path for tasks in a workflow [30]. Scheduling tasks on the critical path is very important because the critical path helps to establish and control the schedule and minimize delays [47]. This paper calculates the priority of tasks by presenting a critical path algorithm, which helps in better and faster scheduling.

Table 1 shows the comparison between the proposed work and related works.

Table 1 Comparison of the proposed framework with related work

Scheduling management system architecture

In this paper, a workflow scheduling management system in a Cloud/Fog computing system is proposed. In these systems, due to the large volume and increasing demand for services, as well as the production of big data and their storage, the workload of Cloud servers increases [43]. Thus, it is desirable to consider the distribution management of services in the Fog layer. On the other hand, the QoS (e.g. delay) of end-users is important [48]. This is because the delayed response can have catastrophic consequences. For this purpose, it is studied how to distribute the workload among the Fog layer nodes at different time intervals to provide service and achieve an optimal schedule in the system. The type of received tasks and the priority of their execution are also considered. As a result, higher priority tasks need to be scheduled sooner [47, 49].

The proposed Cloud/Fog computing system has three hierarchical layers, as shown in Fig. 1. The first layer is the device layer, which includes IoT devices. These devices act as an intermediary and send user requests to higher layers. The second layer, called the Fog layer, has a set of nodes that receives and processes part of the workload of user requests. The third layer is also responsible for providing external resources to execute the workload sent from the middle layer and hosts computing machines. Because the system computing resources are distributed to Cloud and Fog nodes, there is a smart gateway in the middle layer that receives all user requests, manages resources in Cloud and Fog nodes and processing and communication costs, along with data request results from nodes. It also creates the most appropriate schedule for an input workflow [50, 51].

Fig. 1

General architecture of the proposed Cloud/Fog-based scheduling management system

Execution of workflows in the middle layer is done through a workflow management system [29,30,31]. When implementing workflow scheduling in a Cloud/Fog environment, two issues need to be considered. The first one, known as resource provision, involves the selection and provision of computational resources that are used to perform tasks [36]. The second one is scheduling or task allocation, in which each task is scheduled on the appropriate resources. A runtime estimation component is performed using historical data, protocol data, or time series forecasting models, among other methods, to evaluate the performance of Cloud resources and the amount of time it takes to perform tasks in different resources [52,53,54]. This data is used by the resource supply component and the scheduling component to make accurate and efficient decisions about task allocation. The data management component in the workflow management system manages the motion, placement, and storage of data as needed to perform task operations. Finally, the task distributor is responsible for interacting with Cloud APIs to send tasks ready to run on existing resources. Figure 1 shows the general architecture of the proposed Cloud/Fog-based scheduling management system.

Proposed components for the Smart Gateway are explained as follows:

• Cloud/Fog information services

This component provides a variety of information for storage devices and resources in the Cloud/Fog, including their features and characteristics, cost of use, location, and other information needed to select resources and make decisions about optimal scheduling.

• Scheduling management system

  The core of the system is the scheduling management system that is responsible for managing the actual execution of the workflow. It includes the following components:

  • Resource provisioning

    The resource provisioning component is responsible for selecting and supplying Cloud/Fog resources and is done according to the requirements of the scheduling system and service quality requirements.

  • Runtime estimation

    When a task is entered into the system, preprocessing is performed and a deadline is assigned to each task, which is provided in the system by the runtime estimation component.

  • Task dispatcher

    This component is responsible for interacting with Cloud APIs to send tasks ready to run.

  • Data management

    The data management component in the scheduling management system manages the collection, motion, placement, storage, and dependency between data when needed to perform task operations.

  • Scheduler

    This component performs the task of scheduling according to the information and factors obtained from other components to make accurate and efficient decisions, using scheduling algorithms.

  • Monitoring

    The monitoring includes components that enable dynamic and continuous monitoring of workflow tasks and resource performance, as well as resource management.

This paper aims to extend the scheduling component in this architecture, and for this purpose, the following parameters in Table 2 which are described in detail in Evaluation and Discussion section, are considered to evaluate the proposed method.

Table 2 Performance evaluation parameters for proposed method

Problem model

Task graph model

A task graph is represented by a Directed Acyclic Graph (DAG), G = (V, E, w, c), where V represents a set of tasks and E represents a direct relationship between tasks. Each graph has a weight that specifies the workload of the task (number of instructions), and each edge indicated by e_ij is the cost of communication between the two tasks i, j (t_i, t_j), which represents the amount of communication data transmitted from the t_i task and is used as input data for the t_j task. pred (t_i) means tasks that are placed directly before t_i, and succ (t_i) refers to direct successors to t_i. If t_i has no precedent, pred (t_i) = 0, and if a task without any substitutes, succ (t_i) = 0, in which case the output task is called t_exit. It is assumed that a task cannot start until all its inputs are sufficiently collected [30].

Resource clustering algorithm

Clustering is one of the most effective algorithms that can be used to optimize energy consumption and management [18,19,20,21]. The basis of dynamic clustering is to create stable clusters. Considering the inherent mobility behavior of the nodes of the Fog layer and the speed and matching of the movement of these nodes, the purpose of clustering is to create more concentrated clusters with nodes close to each other in order to provide faster execution of the algorithm on the nodes of the Fog layer [22]. Creating centralized clusters with a proportional number of members makes the nodes have better local communication [23]. It also increases the throughput rate, the lifetime of the nodes, reduces the overhead of network computing, reduces the transmission delay and, as a result, delivers more optimal packets.

The purpose of this section clustering, in addition to maximizing the nodes’ lifetime, is to put together nodes that are in the same condition in terms of mobility while minimizing the number of clusters. To achieve this goal, a score-based clustering algorithm is proposed. In this algorithm, each node calculates its score based on four parameters and distributes the score. The cluster head node selects the cluster member nodes (Fog nodes) based on the score received from neighboring nodes. Therefore, the cluster head determines its members in a distributed manner with the minimum computational overhead required and based on the latest information on the current condition of neighboring nodes.

Here it is assumed that the cluster heads are fixed and are assigned to a cluster according to the points they gain. The other nodes then join the clusters according to their mobility score. Since the system is dynamic and architectural elements can be entered to or exited during the execution of the system, and also due to the mobility of the nodes, the clusters are constantly changing, and as a result, the proposed clustering algorithm is defined dynamic too. Three important parameters are considered for selecting cluster heads. These parameters are fully described below. Each cluster node as well as each cluster member node calculates a value for itself using the proposed fuzzy-based method to indicate its suitability for a cluster. Table 3 shows the term descriptions and Table 4 shows messages used in the clustering algorithm.

Table 3 Term description

Table 4 Message format

The main structure of the proposed algorithm (Algorithm 1) for selecting the cluster head and its members has the following steps:

• The first step (calculating the cluster head score).

Initially, the score of each cluster head is calculated using a multi-criteria method based on the fuzzy inference method (described in Input and output variables of the fuzzy SoCH system section).

• Step 2 (calculating the mobility score for each node).

In this step, a multifunctional method based on fuzzy inference is proposed to calculate the mobility score of Fog layer nodes. In Input and output variables of the fuzzy SoMob system section this method is fully explained.

• Step 3 (creating a list of available cluster heads for each node).

At this point, each node creates a list of cluster heads in its location by sending a "CHmsg" message to its neighbor cluster heads.

• Step 4 (distributing the global message).

Each node sends the message "JOINmsg" announcing its request to connect to the clusters in its range (available list).

• Step 5 (selecting the cluster numbers).

At this stage, the cluster head checks the number of its members and if the number of members in that cluster has not reached the maximum, by sending the message "ACCPETmsg", it selects the node as a member of the cluster and connects the node to the cluster.

Fuzzy inference system for calculating scores of the cluster head

A fuzzy decision-making controller consists of the following [30]:

• 1) Input and output variables. These variables are usually determined based on the knowledge of experts.

• 2) Fuzzification interface. This interface converts input variables into fuzzy sets.

• 3) Fuzz rules. The fuzzy inference process uses these rules.

• 4) De-fuzzification interface. This interface translates fuzzy linguistic values into a clear real number that is the output of the fuzzy inference process.

Input and output variables of the fuzzy SoCH system

In this section, a fuzzy-based method for calculating the score of the cluster head and members of each cluster is presented.

This paper uses three parameters BW_CH, RT_CH, and PP_CH as input parameters in the fuzzy inference system (Eqs. 1, 2 and 3). The output of the method is used as a qualitative criterion for each node of the cluster head. The output variable will also be called (SoCH), which shows the score of each node of the cluster and the score of each cluster in general. The input variables are as follows:

$${BW}_{CH}=\frac{\frac1n\sum_{i=1}^n\mathrm{Bandwidth}}{\mathrm{Number}\;\mathrm{of}\;\mathrm{members}}$$

(1)

$${RT}_{CH}=\mathrm{Delay}\;\mathrm{of}\;\mathrm{network}(\mathrm{Response}\;\mathrm{Time})$$

(2)

$${PP}_{CH}=\frac{\sum_{i=1}^n\mathrm{Processing}\;\mathrm{power}}{\mathrm{Number}\;\mathrm{of}\;\mathrm{members}}$$

(3)

where BW_CH is the average bandwidth of cluster members, RT_CH is the response time, and PP_CH is the average processing power of cluster members.

Algorithm 1. Resource Clustering in Fog/Cloud RCFC Function()

Fuzzy inference, de-fuzzy inference, and membership functions for the fuzzy SoCH system

The concept of fuzzy inference is the use of fuzzy logic to adapt a given set of inputs to a given output. For this, fuzzy logic, fuzzy rules, linguistic variables and fuzzy sets are the main members. Fuzzy variables can take values, such as low, medium or high, etc., the degree of membership that the numerical between 0 to 1 can obtain with the membership function.

To express the membership function can be used a curve or linear shape. The membership functions designed for three inputs as well as one output are summarized and shown in Table 5. A graphical representation of the fuzzy system membership functions is shown in Fig. 2 and the output surfaces of a fuzzy system with two inputs and one output are shown in Fig. 3 (a, b, c).

Table 5 Membership functions and linguistic values of the fuzzy system

Fig. 2

Graphical presentation of the membership functions of the fuzzy SoCH System. a Membership function for the input variable: BW. b Membership function for the input variable: RT_CH. c Membership function for the input variable: PP_CH. d Membership function for the output variable: SoCH

Fig. 3

Output surfaces of the fuzzy SoCH system with two inputs and one output. a inputs: PP_CH and BW_CH. b inputs: RT_CH and BW_CH. c inputs: PP_CH and RT_CH

Input variables of the fuzzy method for selecting the cluster head:

• 1. BW_CH. Low(L), Medium(M) and High(H) are defined as a fuzzy set.

• 2. RT_CH. Weak(W), Moderate(M), Strong(S) are defined as a fuzzy set.

• 3. PP_CH. Low(L), Medium(M), High(H) are defined as a fuzzy set.

The fuzzy method output variable for selecting the cluster head:

• 1. SoCH. This output variable has four fuzzy sets which are defined as Bad (B), Moderate (M), Good (G), and Excellent (E).

Fuzzy rule base for the fuzzy SoCH system

Existing knowledge about the problem is stored in the "if" language rules as shown in Table 6. These rules guide system behavior. An "if–then" rule can make the usual human decisions using language tags and membership functions. Each "if–then" rule, on the other hand, defines the behavioral dynamics of the target system.

• Rule 10: if (BW_CH is Moderate) and (RT_CH is Weak) and (PP_CH is Low) then (SoCH is Bad)

• Rule 18: if (BW_CH is NOT Medium) and (RT_CH is Strong) and (PP_CH is High) then (SoCH is Good)

Table 6 Fuzzy rules for the output variable to select the cluster head

Input and output variables of the fuzzy SoMob system

The proposed clustering algorithm for calculating the mobility score of cluster members considers Stability (ST_M) and Speed ​​(V_M), and scores obtained from the nodes of the cluster head (calculated in Input and output variables of the fuzzy SoCH system section) where each node is given a score based on these parameters, which is the main basis for clustering by cluster heads. The input variables are as follows:

• 1. Speed of each node (V_M). The average velocity for each node up to the current time T is calculated by Eq. 4:

  $${V}_{M} =\frac{1}{T}\sum_{t=1}^{T}\sqrt{{({X}_{t}-{X}_{t-1})}^{2}+{({Y}_{t}- {Y}_{t-1})}^{2}}$$

  (4)

  where (X_t, Y_t) and (X_t-1, Y_t-1) are the coordinates of node v at time (t) and (t—1) respectively.

• 2. Stability (ST_M). Stability refers to the stability of the cluster member node relative to the cluster head nodes in its range. A higher stability number indicates that a node has been in the same range for a longer time, which means that the intended node is in a more stable state. Stability is calculated by Eq. 5.

  $${ST}_{M}= \sum_{i=1}^{n}{T}_{RF}- {T}_{RL}$$

  (5)

  where T_RF is the time of the first packet received and T_RL is the time of the last packet received, and n represents the number of neighbors of a node.

• 3. Cluster head node score (SoCH). It is calculated in Input and output variables of the fuzzy SoCH system section.

Fuzzy, de-fuzzy, and membership functions for the fuzzy SoMob system

In a fuzzy model, it is important to select the appropriate language values to provide input and output variables and to select membership functions to determine the scope of the program. As explained above, the next step is to convert the explicit values of the input variables to fuzzy sets. Table 7 shows the membership functions and linguistic values of the desired fuzzy system. A graphical representation of fuzzy system membership functions is shown in Fig. 4 and surface output plots of the fuzzy SoMob system with two inputs and one output are shown in Fig. 5 (a, b, c).

Table 7 Fuzzy system membership functions and linguistic values

Fig. 4

Graphical presentation of the membership functions of the Fuzzy SoMob System. a Membership function for the input variable: V_M. b Membership function for the input variable: SoCH_M. c Membership function for the input variable: ST_M. d Membership function for the output variable: SoMob_M

Fig. 5

Surface output plots of the Fuzzy SoMob System. (a) inputs: SoCH_M and ST_M. b inputs: SoCH_M and V_M. c inputs: ST_M and V_M

Input variables of the fuzzy method for selecting a member of a cluster:

• 1. V_M. Three fuzzy sets are defined for this input variable: Weak(W), Moderate(M), Strong(S)

• 2. ST_M. Three fuzzy sets are defined for this input variable: Low(L), Medium(M), High(H)

• 3. SoCH. Four fuzzy sets are defined for this input variable: Bad(B), Moderate(M), Good(G), Excellent(E)

The output variable of the fuzzy method for selecting a cluster member:

• 1. SoMob. This output variable has five fuzzy sets which are defined as Very Low (VL), Low (L), Moderate (M), High (H), Very High (VH).

Fuzzy rule base for the fuzzy SoMob system

In the next step, determine the behavior of the system using fuzzy rules is determined. The rules for calculating the output are given in Table 8.

Table 8 Fuzzy rules for the output variable to select the cluster member

Figure 6 shows the sequence diagram of resource clustering in the Cloud/Fog architecture.

Fig. 6

Sequence diagram of the proposed clustering algorithm

Scheduling algorithm

The purpose of this paper is to present a distributed scheduling algorithm for scheduling the tasks of a workflow to a set of Cloud/Fog resources. Since the workflow consists of a set of dependent tasks to ensure that priority is limited between tasks, each task must begin when all previous tasks have been completed. Also, the runtime of a schedule should not exceed the set time for it, so the purpose of the scheduling algorithm is to select the appropriate resource for assigning tasks by observing the workflow deadline, the data dependency between tasks and also resource mobility. Figure 7 shows an overview of the workflow scheduling steps that will be described below. The goal in this paper is to provide an algorithm for allocating workflow to a set of resources in the Cloud/Fog environment. This algorithm tries to select the best resource with the priority of Fog resources while guaranteeing the deadline of the workflow.

Fig. 7

An overview of the workflow scheduling steps

The proposed workflow scheduling algorithm consists of three phases: The first phase is to obtain a critical path to find higher priority tasks. In the next phase, according to the clustering done on the resources (described in Fuzzy inference system for calculating scores of the cluster head section), m clusters are selected from the available clusters for the workflow. Then, by using the exhaustive search method, all possible schedules are generated on the clusters and the best schedules are selected according to the utility function. The third phase is to reserve resources for tasks on each of the selected schedules. In this phase, if a task fails to make a reservation on a resource, the entire scheduling process fails.

• Phase 1; Critical path extraction algorithm

In the DAG scheduling process, one of the most important keys is identifying important tasks. This paper uses the Critical Path extraction algorithm (CP) to generate the critical path. This algorithm obtains the task priority (PRT) factor using the Multi-Criteria Decision-Making method (MCDM) according to the following parameters:

• Communication Cost (${CC}_{j}^{i}$)

The cost of communication between two tasks, t_i and t_j, is the amount of data that must be transferred between them. This value can be measured in bytes, kilobytes, etc. The higher the cost of communication between related tasks, the more important the tasks become, and therefore these highly dependent tasks can be set to the same cluster or near clusters (according to the mobility score) which are available with the fast connection link.

• Execution Cost of Task (${\mathrm{ECT}}_{{\mathrm{t}}_{\mathrm{i}}}^{\mathrm{RS}}$)

This factor is the cost of performing the t_i task on all available clusters for that task, which has the type of resource required for the task and based on the estimated processor cycles required to perform the t_i task, and the average resource computing power with the type required in all clusters are calculated. Calculating the execution cost according to the processor cycles instead of the runtime makes it more accurate in estimating the runtime of each task on each resource based on the resource computing power. ${ECT}_{{t}_{i}}^{RS}$ is calculated by Eq. 6.

$${ECT}_{{t}_{i}}^{RS}= \frac{{t}_{i}\cdot Length}{{CCP}_{RS}}$$

(6)

where ${CCP}_{RS}$ is the average cluster computing power for resources of the type required for each task and is calculated by Eq. 7.

$${CCP}_{RS}= \frac{1}{M}\sum_{{R}_{j}\in {Cluster}_{RS}}{R}_{j}\cdot ComputationPower$$

(7)

where Cluster_RS is the set of all the resources of the type required by each task in the cluster and M is the number of resources in the Cluster_RS set. A task becomes more critical with increasing the execution cost.

• Time Pressure of Scheduler (${TPS}_{{t}_{i}}$)

This factor indicates how much pressure the scheduler has to perform a task, and its value is between zero and one. Let $EST({t}_{i} . {RS}_{n})$ be defined as the task start time estimation t_i, which is estimated before the actual execution of the task on the resource. When the value of ${TPS}_{{t}_{i}}$ is closer to one, the program management unit is under more pressure to schedule the task t_i. This factor is determined by Eq. 8.

$${TPS}_{{t}_{i}}=1- \frac{(EST\left({t}_{i} . {RS}_{n}\right)- \alpha )}{Max({\cup }_{{t}_{i \in DAG\cdot Tasks}}(EST\left({t}_{i}.{RS}_{n}\right)- \alpha ))}$$

(8)

where α is the current time of the system.

Since the Cloud and Fog environment are heterogeneous and the calculation time of tasks varies from one resource to another resource, accurate calculation $EST({t}_{i} . {RS}_{n})$ is not possible.

In addition, factors such as bandwidth between clusters and the amount of data transfer between tasks affect the data transfer time. Therefore, for any unscheduled task, data transfer must be performed and the execution time of each task must be calculated approximately. If t_entry is the input node in the DAG of the application, the $EST({t}_{entry} . {RS}_{n})$ is calculated by Eq. 9.

$$EST\left(t_{entry}\;.{\;RS}_n\right)=\alpha-\varepsilon$$

(9)

where ε is the time when the scheduler needs t_i to schedule the task.

$EST({t}_{i} . {RS}_{n})$ is calculated by Eq. 10.

$$EST\left(t_{i\;}.{RS}_n\right)={Max}_{t_j\in pred\left(t_i\right)}\left\{\begin{array}{c}Min({DL}_{t_j}.\;EFT\left(t_j\;.{\;RS}_m\right)+\frac{{CC}_j^i}{BW}\quad\quad if\;t_{j\;}is\;not\;scheduled\\AFT\left(t_j\;.\;{RS}_n\right)+\frac{{CC}_j^i}{BW}\quad\quad otherwise\end{array}\right.$$

(10)

Let $EFT\left({t}_{j} . {RS}_{n}\right)$ be the estimated end time of t_j without considering the actual resource of the task processing, and be calculated by Eq. 11.

$$EFT\left(t_{j\;}.\;{RS}_n\right)=EST\left(t_j\;.\;{RS}_n\right)+AEC\left(t_j\;.\;{RS}_n\right)$$

(11)

after scheduling $EST\left({t}_{i} . {RS}_{n}\right)$ and$EST\left({t}_{i} . {RS}_{n}\right)$, with the actual start time of $AST\left({t}_{j} . {RS}_{n}\right)$ and the actual end time of $AFT\left({t}_{j} . {RS}_{n}\right)$ being replaced respectively.

• Number of Task Successors (${Num\cdot Suc}_{{t}_{i}}$)

If a task has more successors, it indicates that the immediate successors of this task are waiting for their main tasks to be performed. As a result, this task usually depends on more resources for communication, and this makes it even more important.

• Source Request Rate ($S{RR}_{{t}_{i}}^{RS}$)

This factor is defined according to the type of resources requested for each task. ${SRR}_{{t}_{i}}^{RS}$ specifies the average availability of the user's requested resource type to the number of requests given to that resource. the more amount that is available compared to the number of requests in this type of source then the more critical path containing the tasks that requested this resource. This amount is calculated for all clusters containing this resource in the form of Eq. 12:

$${SRR}_{{\mathrm t}_{\mathrm i}}^{{RS}_{RT}}=\frac1P\sum_{RT\;\in{\;Cluster}_{RS}}{SAR}_{t_i}^{{RS}_{RT}}$$

(12)

where P is the number of clusters that have the type of intended resource and ${SAR}_{{t}_{i}}^{{RS}_{RT}}$ is the resource rate for the type of required resource t_i that is calculated in the form of Eq. 13.

(13)

Now the priority of each task can be calculated according to Eq. 14.

$${PRT}^{t_i}=w_1\times\left(\frac{\sum_{t_k\in DAG\cdot{Succ}_{t_i}}\left({((CC)}^N+{PRT}_{DAG}^{t_k}\right)}{Max\left(2\times{Num\cdot Suc}_{t_i}.\;1\right)}\right)+w_2\times{TPS}_{t_i}+w_3\times{Num\cdot Suc}_{t_i}+w_4\times\mathrm{ECT}\left(t_j\;.\;{RS}_n\right)+w_5\times{SRR}_{{\mathrm t}_{\mathrm i}}^{{RS}_{RT}}$$

(14)

where w_i (i = 1,2,3,4,5) is the weighting factor that determines the effect of the value i-th on the priority of the task.

Algorithm 2 shows how to extract the critical path.

This algorithm first uses Depth First Search (DFS) to find all possible paths in the DAG (line 2). The priority of each task is then calculated if it is not scheduled. The PRT value is calculated according to Eq. 14 (lines 4 to 7). The algorithm then obtains the critical path from the sum of the PRT values of the tasks. Thus the sum of the PRT values of each path is calculated and CP selects the path with the highest PRT sum as the critical path. The scheduling algorithm is then called (line 18) and the result determines whether the scheduling is successful or not.

Algorithm 2. CP Function()

•  The second phase; the Soft Deadline Mobility Scheduling (SDMS) algorithm

The proposed scheduling algorithm is implemented hierarchically. In the first step, a list of available clusters is selected for the workflow, and then the best clusters are selected as candidate clusters (${CL\cdot Candidates}_{WF}$) to perform the workflow. These clusters are examined according to the factors of distance of the workflow to the desired cluster and the velocity vector matching of each cluster. Then, using the Multi-Criteria Decision-Making method (MCDM), the factor ${CL\cdot Candidates}_{WF}$ is calculated according to the following parameters.

• Workflow Distance to Cluster (${Distance}_{WF}^{{CL}_{i}}$)

The shorter the distance from the workflow to a cluster, the better the choice. To obtain this distance, Eq. 15 can be used, which considers the relative velocity of two nodes:

$$\overrightarrow{{V}_{WF}}- \overrightarrow{{V}_{{CL}_{i}}}$$

(15)

so that the velocity vector of a cluster is equal to the vector sum of all resources within a cluster, which is calculated by Eq. 16.

$$\overrightarrow{V_{{CL}_i}}={\sum_{k=1}^n\overrightarrow{V_{RS}}}_{RS\;\in C{luster}_i}$$

(16)

where K is the number of resources within a cluster. The distance parameter is calculated according to Eq. 17.

$${Distance}_{WF}^{{CL}_{i}}=\sqrt{{\left({X}_{T}^{{CL}_{i}}-{X}_{T}^{WF} \right)}^{2}+{\left({Y}_{T}^{{CL}_{i}}- {Y}_{T}^{WF} \right)}^{2}}$$

(17)

where (${X}_{T}^{{CL}_{i}} . {Y}_{T}^{{CL}_{i}}$( and ${(X}_{T}^{WF} . {Y}_{T}^{WF})$ are the coordinates of the cluster ${CL}_{i}$ and $WF$ at time T respectively.

• Cluster Velocity Vector Matching with the Workflow (${SVM}_{WF}^{{CL}_{i}}$)

This parameter shows the matching of the velocity vector of a cluster with the workflow. Since the layer nodes of the device are fixed, their velocity is also considered zero.therefore, if the result of Eq. 18 is a positive number, it indicates that the cluster will be in the direction of the workflow, otherwise, there will be a mismatch. In this regard, the matching of the velocity vector of a cluster with the workflow will be equal to the size of the velocity vector of the cluster.

$${SVM}_{WF}^{{CL}_i}=\begin{Bmatrix}\left|V_{{CL}_i}\right|\;if\;cluster\;matches\;to\;the\;workflow\\\left(-1\right)\times\left|V_{{CL}_i}\right|\;if\;cluster\;does\;not\;match\;to\;the\;workflow\end{Bmatrix}$$

(18)

According to the defined parameters, the equation for selecting the best cluster (${CL\cdot Candidates}_{{t}_{i}}$) is then defined as Eq. 19.

$${CL\cdot Candidates}_{WF}= {w}_{1}\times {Distance}_{WF}^{{CL}_{i}}+ {w}_{2} \times {SVM}_{WF}^{{CL}_{i}}$$

(19)

that w_i (i = 1,2) is the weighting factor that determines the effect of the value i-th on the candidate clusters.

Then the cluster with the highest score is selected as the BestCluster for the workflow. In the second phase of the scheduling algorithm, using the exhaustive search method, all possible scheduling for the workflow is created. At this stage, the best schedules are selected using the utility function (ScheduleScore). The desired workflow deadline is also selected and it will be sent to the next stage for reservation. To obtain the utility function, this paper uses the following factors to score the execution time of a workflow in parallel and serial mode.

• 1. The execution time of a workflow in parallel mode depends on the extent to which the tasks of a workflow can be performed in parallel mode which is defined as follows:

Job Execution Time on Most Powerful Resource with Max Parallel(${ETMP}^{{CL}_{x}}$)

• 2. If No.1 fails to be scheduled, all tasks must be performed consecutively which is defined as follows:

Job Execution Time on Most Powerful Resource with Serial(${ETMS}^{{CL}_{x}}$)

• 3. This parameter considers the state in which the workflow is executed on the best cluster that can do its work in parallel, which is defined as follows:

Job Execution Time on with Max Parallel (${ETMP}^{{CL}_{mp}}$)

• 4. The last parameter is related to when the workflow is executed on the best cluster that can do its work in serial, which is defined as follows:

Job Execution Time on with Serial (${ETMS}^{{CL}_{ms}}$)

The utility function is calculated by Eq. 20.

$$Utility\;Function=\frac{{ETMP}^{{CL}_x}}{{ETMP}^{{CL}_{mp}}}+\frac{{ETMS}^{{CL}_x}}{{ETMS}^{{CL}_{ms}}}$$

(20)

The SDMS algorithm is presented in Algorithm 3.

In this algorithm, the scheduler first selects the m cluster of available resource clusters for each task of all the workflow tasks (lines 1 to 7) and lists them. Next, the best clusters are selected for the workflow (lines 11 to 14), and the EST and EFT factors are calculated for each task (lines 15 to 18). In the following step, all possible schedules are created for the workflow, which also takes into account the considered deadline for the workflow. Next, using a utility function (ScheduleScore), the schedules with the highest score (minimum utility function) are included in the list of the best scheduling (lines 20 to 26). Then on line 28, the Task Reservation function is called and the reservation result is returned. If a task can successfully reserve a resource, the EST and EFT values will be replaced by the AST and AFT values respectively (line 35).

Algorithm 3. SDMS Function()

• • Phase 3; Task reservation

Once the scheduling is done and the best ones have been selected, the time has come to reserve resources for the tasks. First, a schedule is selected from the list of best schedules, and a resource is reserved for a task. At any stage of the reserving process, if a task cannot be mapped to a resource, the entire schedule will fail and the algorithm consider the next schedule. Finally, if no scheduling is set, the RSA value that determines the scheduling result will change to False.

For this purpose, this paper uses the resource reservation algorithm in [30]. This algorithm receives the workflow as a set of tasks and a list of the best schedules and examines the reservation of resources on the tasks. For each best schedule, tasks are sorted in descending order, and for each task, a reservation is made. In each repetition, an unplanned task with the highest PRT is selected. If at any stage the resource reservation fails, the whole process for that schedule is canceled and the function will move on to the next schedule in the list of best schedules. If all tasks in the STL array are successfully scheduled, this function returns the True value and indicates that the entire workflow has been successfully scheduled.

Evaluation and discussion

In this section, the scheduling algorithm presented in the previous section is evaluated. Velociraptor simulator [55], which is a domain-specific simulator software for Cloud/Fog environments is used to evaluate the performance of the proposed method. The performance of the proposed method in comparison with GRP-HEFT [56] and MOODS [57] is shown and the MATLAB tool is also used to implement the proposed fuzzy inference system. The duration of each simulation was 300 time steps and the simulation process was repeated 10 times for each method. Finally, this paper reported the average of all experiments. To correctly compare the proposed method with MOODS and GRP-HEFT, the following modifications should be considered:

• In the proposed method, budget constraint is not considered. Since these two methods MOODS and GRP-HEFT have this constraint, it is not considered.

• GRP-HEFT and MOODS do not support Fog resources for scheduling, but it has been added to both.

• GRP-HEFT uses a greedy scheme.

• In the proposed algorithm and MOODS, the first fit scheme is used.

• The implementation of GRP-HEFT and MOODS in the performed experiments has some minor differences compared with the original work as it was necessary to make the implementation feasible.

• Configuration of environment

This paper used the information published by Facebook in the OpenCompute project https://www.opencompute.org/ to configure the cloud environment. The number of hosts in the clusters in this project is divided into 20 and the parameter “Cloud provider count” has been added to mimic a multi-cloud environment. Table 9 shows the Fog environment configuration and Table 10 the Cloud environment configuration. Processor capacity is estimated by Cisco https://www.cisco.com/c/dam/global/da_dk/assets/docs/presentations/vBoo.amp_Performance_Benchmark.pdf in a million instructions per second (MIPS) and a trillion instructions per second (TIPS).

• • Data-sets

Table 9 Configuration of Fog layer for simulation experiments

Table 10 Configuration of Cloud environment for simulation experiments

The DS Lab. Workflow Jobs Dataset [58] has been used, which contains 100,000 DAGs created in a hybrid form based on Epigenomics and Montage scientific workflows structures with different job sizes and communication costs in the experimental studies.

An overview of the data-set configuration is given in Table 11. This paper used this data-set for input workloads to compare the performance of the proposed method and the two works GRP-HEFT and MOODS. Details of Montage and Epigenomics workflows are discussed in [59]. Figure 8 shows the Montage and Epigenomics workflows.

Table 11 Job dataset configuration

Fig. 8

a an Epigenomics workflow, b a Montage workflow [59]

Notice: In each simulation round, the simulator software randomly selected 70 DAGs from the dataset and used these DAGs as the input workload.

• Consumption of energy

In this section, consumption of energy is used based on a simplified model in [60] for Cloud and Fog hosts. For this purpose, Eq. 21 and assumptions in [60, 61] have been used.

$$P=I+ \sum_{i=0}^{N-1}{\alpha }_{N}{\rho }_{N}\left(i\right)+ \sum_{J=0}^{C-1}{\alpha }_{C}{\rho }_{C}\left(j\right)+ \sum_{K=0}^{D-1}{\alpha }_{D}{\rho }_{D}\left(k\right)+ {\psi }_{m}\left(\sum_{j=0}^{C-1}{\rho }_{C}\left(j\right)\right)+{\psi }_{M}\left(\sum_{j=0}^{C-1}{\rho }_{C}\left(j\right)\right)$$

(21)

where I indicate idle power consumption. According to [60], each subsystem produces linear power consumption according to its individual utilization. In Eq. 21, the power consumption of a core is obtained by multiplying the factors (ρC), (ρD), (ρN) by a constant coefficient (αC, αD, αN) which are the core usage, disk usage and network usage, respectively. These factors do not depend on the workload of the program. In this equation, there is no separate subsystem for memory and the power consumed by access to memory is included in the calculations of power consumed by other subsystems. CPU instruction execution tends to highly correlate to memory accesses in most applications.

• Fault model

To run the simulation, a simplified fault model is considered during its execution to the environment. Information about the main probability values of the error activation model is given in Table 12, which is based on [62] and some intuitions.

Table 12 Failure model settings

Some random deletion errors are indicated by P (NodeHasOmissionFault) which are activated during runtime. If an error is triggered that is irreparable, the node will no longer be available and all tasks in its advanced backup list will be removed without notice. This makes it highly likely that all tasks will be scheduled or rescheduled. P (taskFailedAndDeadlinePassed | fullFailurePermanent) fail on other resources within their timeframe because the cluster head loses effective window control and scheduling of failed node tasks.

• Location and movement model of nodes

The dimensions of the environment and the motion model of IoT nodes are given in Table 12. Random Way-Point (RWP) [63] is used as the motion model. In this section, it is assumed that if a node starts a path, according to Table 13, it immediately reaches a random speed between 8 m/s to 18 m/s and moves at this constant speed to reach the target. After a node has reach its target, it selects a new destination and a new speed.

Table 13 Physical environment specifications

In the first step of the simulation with a uniform distribution of a fixed position in the environment for each data centre of Cloud and Fog-point is selected, which is shown in Fig. 9 the movements of a single node in the environment. The dimensions of the environment in all experiments are considered to be 20 km × 20 km.

Fig. 9

An example of a node path in a $20km\times 20km$ environment that has no predetermined path or obstacles and is implemented with the RWP algorithm

Figure 10 illustrates the location of datacenters in one of the simulation settings in the environment. According to Table 9, there are three cloud providers Alabamacom, TravoSystems and Comtroniz. For instance, if the ID of a data center in Fig. 10 is Datacenter_0_Alabamacom, that data center belongs to the Alabamacom Cloud provider (sequence number = 0). These names do not belong to any real commercial or non-commercial organization. Figure 11 shown the location of Fog-points in the environment. According to Table 8, there are four Fog providers Ney, Tabib, Aseman, and Peymaneh. For instance, if the ID of a Fog-point in Fig. 11 is F_9_Ney, that Fog-point belongs to the Ney Fog provider company (sequence number = 7). These names do not belong to any real commercial or non-commercial organization.

Fig. 10

Datacenters location in the environment

Fig. 11

Fog-points location in the environment

Experimental results

In this subsection, the results of comparing the proposed method, with the GRP-HEFT [56] and MOODS [57] are reported.

Resource utilization rate and scheduling success rate are important parameters in our evaluation that affect other parameters. In order to have a better resource utilization rate, resources should be given more work, and this will increase the workload of the resources and increase the rejection rate and waiting time. On the other hand, the higher the resource utilization rate and the higher the success rate of scheduling, the more energy will be spent, but the waiting time will be shorter.

Likewise, if the number of alive nodes are many during execution and we can have clusters with closer and more stable resources, as a result, we will have better and faster data transfer, and this will cause higher throughput and a lower failure rate. But having more alive nodes will not necessarily increase the throughput, and by not having an advantage in the number of alive nodes, it is possible to have a significant throughput. It can be concluded that usually the parameters are in conflict and the better of one does not necessarily make the other better. In general, in this paper, it has been tried to put the contradicting parameters in a trade-off and as the results show, our proposed method has performed better compared to the other two methods.

• Fog/Cloud resource utilization for Scheduling performance

Resource utilization in the Fog and Cloud can be defined as the amount of resource time used to perform scheduled tasks on it, which is defined as Eq. 22. Table 14 shows the resource utilization in the proposed method compared to GRP-HEFT and MOODS. The proposed method has a much higher resource utilization than the other two methods.

Table 14 Resource utilization

$$Resource\;Utilization=\frac{busyTime}{totalUpTime}\times100$$

(22)

• Success rate for scheduling performance

The success rate is defined in terms of the number of successful requests in relation to the total number of tasks, and represents the fraction of tasks that have been successfully scheduled and completed before the deadline. In the proposed scheduling algorithm, all scheduling is done according to the deadline and the success rate of the algorithm is due to the successful completion of the workflow before the deadline. Table 15 shows the success rate of resources in the proposed method compared to GRP-HEFT and MOODS. This rate is significantly higher in the proposed method than in the other two works, indicating that the proposed algorithm performed better.

Table 15 Job scheduling success rate

Getting a better result in resource utilization and scheduling success rate shows that schedules are selected using the utility function that are better in terms of completion time and as a result, the selected cluster is a better cluster for workflow tasks. In fact, the better rate of resource utilization shows that the clustering algorithm has worked very successfully.

• Time related parameters for scheduling performance

This paper has examined the parameters degree of parallelism, makespan and waiting time as time related parameters in relation to scheduling performance.

• ◾ Parallelism degree

In this part, the degree of parallelism (number of tasks running in parallel) for GRP-HEFT and MOODS is compared to the proposed algorithm, which is shown in Figure 12. As can be seen in Figure 12.a, the proposed algorithm has performed better in performing tasks in parallel, and in a period of time this number has reached a maximum of 30 tasks, while in GRP-HEFT and MOODS this value eventually reaches 23 tasks.

Fig. 12

Parallelism degree. a Proposed Method. b GRP-HEFT. c MOODS

Due to dynamic clustering and creating clusters with closer and more stable resources, it can be concluded that more tasks are executed at the same time, and this causes us to consume more energy.

• ◾ MakeSpan

Makespan is the amount of time from providing a workflow to completing it. Figure 13 shows a comparison between the Makespan diagram of the proposed algorithm with GRP-HEFT and MOODS. As has been seen in Figure 13(a, b, c), the proposed method and two other methods performed similarly in the initial time frame, but later the proposed scheduling algorithm performed better (Figure 13.a) and more tasks were completed. It should also be noted that all completed work has been completed before the deadline. There is no guarantee that the work will be done, but if it is done, it will be done before the deadline.

Fig. 13

MakeSpan parameter. a Proposed Method. b GRP-HEFT. c MOODS

• ◾ Waiting time

Waiting time is the average time it takes for a task to complete the scheduling process. Figure 14 shows the workflow waiting time. The waiting time for the proposed method (Figure 14.a) is longer than the other two works (Figure 14.b and c). Given that the success rate and resource utilization of the proposed method is higher, this causes more work to be done, so that naturally the waiting time will be longer.

• Energy consumption

Fig. 14

Waiting Time parameter. a Proposed Method. b GRP-HEFT. c MOODS

The energy consumption for the proposed scheduling algorithm, MOODS and GRP-HEFT is examined, and the results are shown in Figs. 15, 16 and 17. Figure 15.a shows the average energy consumption per step by the physical Cloud /Fog host, and the diagram in Fig. 15.b shows this for the total energy consumption at each step by all physical hosts. In the proposed algorithm, the average energy consumption in some time stages reaches 200, while this number will be in GRP-HEFT and MOODS 140 and 250 respectively. The total energy consumption at each stage in some cases is higher than 28,000, while in the GRP-HEFT algorithm, this value is less than 18,000 in the usual steps. Also, in MOODS, the total energy reaches over 34,000. It must be kept in mind that the more resources used, the more energy will be consumed. As in the proposed scheduling algorithm, more tasks are successfully scheduled, so higher energy consumption values are expected to be seen.

Fig. 15

Energy consumption report for the proposed Method. a Average energy consumption in each step by a Fog/Cloud physical host. b Total energy consumption in each step by all physical host

Fig. 16

Energy consumption report for GRP-HEFT. a Average energy consumption in each step by a Fog/Cloud physical host. b Total energy consumption in each step by all physical hosts

Fig. 17

Energy consumption report for MOODS. a Average energy consumption in each step by a Fog/Cloud physical host. b Total energy consumption in each step by all physical hosts

Figure 18 shows the percentage of clusters and their members compared to the proposed scheduling algorithm, GRP-HEFT and MOODS. As can be seen in Fig. 18.a, 88% of the clusters formed in the proposed algorithm have less than 11 members, and only 1% have more than 20 members, while in GRP-HEFT (Fig. 18.b) 26% of the clusters have between 15 and 20 members and 18% have more than 20 members. In MOODS, 71% of the clusters have less than 6 members, 19% have between 6 and 10 members, 9% have 11 to 20 members and 1% have more than 20 members (Fig. 18c). Figure 19 also shows the package delivery report for the proposed method and the other two methods. As can be seen in the figure, in total a larger amount of information was transmitted in the proposed method than in the other two methods, of which 72.4% were related to receiving via Wi-Fi and only 9.1% of the data were lost. These numbers in GRP-HEFT are 27.8% and 16.6% respectively. MOODS also accounted for 61.7% of its data transmission via Wi-Fi and 10.3% of its data were lost through network failure.

Fig. 18

Cluster formation. a Proposed Method. b GRP-HEFT. c MOODS

Fig. 19

Packet delivery report. a Proposed Method. b GRP-HEFT. c MOODS

Figure 20 shows the throughput and Fig. 21 shows the number of alive nodes in the simulation of the proposed algorithm, GRP-HEFT and MOODS. The throughput of the proposed algorithm (Fig. 20.a) in some time stages reaches a number close to 4000 MBps, while this number is in GRP-HEFT 1400 and in MOODS 1800 MBps. This demonstrates the superiority of the proposed algorithm in throughput. Since the number of alive nodes in this method is not much different from the other two methods, the reason for the high throughput rate can be considered the success of the clustering algorithm in creating more stable and optimal clusters. In the number of alive nodes, the proposed algorithm works well and the number of nodes stays alive over time and their number decreases with a lower slope.

Fig. 20

Throughput. a Proposed method. b GRP-HEFT. c MOODS

Fig. 21

Alive nodes. a Proposed method. b GRP-HEFT. c MOODS

In fact, when there are more nodes in the network, as a result, more data is sent in the network. Because in the proposed method, the scheduling is done on a cluster and also more data is sent via Wi-Fi, this makes the data transfer speed higher and at the same time the failure rate decreases. This issue also has a very positive effect on throughput.

Conclusion

In this paper, a hybrid scheduling-clustering method for Cloud/Fog-based systems was proposed. The interaction of Fog computing and Cloud computing significantly increases the performance of real-time interactive applications. This Cloud/Fog computing system has three hierarchical layers. The second layer (Fog layer), which is responsible for managing requests, has a scheduling management system that manages all user requests under the heading of workflow and creates the most appropriate available scheduling for the workflow.

A decentralized and dynamic fuzzy inference based clustering algorithm was proposed that operates with the inherent mobility of nodes to form stable clusters with an appropriate number of members while avoiding global clustering. In this algorithm, first, the node score of the cluster heads were calculated using fuzzy inference system. Then, each node member of the cluster obtained its mobility score in the same way and used four important parameters, and according to the message it sent to connect to the cluster head nodes in its range, it is connected to a cluster. This allows the nodes with the closer mobility score to connect to their corresponding clusters.

The scheduling management component in the proposed architecture in the Fog layer performs scheduling workflow in three phases. In the first phase, using a critical path algorithm, important paths were extracted based on the priority value of each task. In the next step, all possible schedules were created using a scheduling algorithm and the best one was selected using a new utility function. In the last step, tasks were assigned to resources.

In this regard, the contributions of this paper can be summarized as follows:

• Devising a three-layer architecture that simultaneously takes into account the advantages of the cloud and fog layers to manage the scheduling of workflow tasks and also manage the mobility of fog nodes.

• Presenting a dynamic, light-weight, multi-criteria, and decentralized clustering algorithm to create clusters considering the mobility of nodes.

• Extending the scheduling component and introducing a new scheduling algorithm that considers: priority of tasks obtained by using a critical path algorithm, a novel utility function, and considering workflow deadlines.

This paper used extensive software simulation to compare the performance of proposed method with two new methods in real workload. The evaluation results showed that proposed method has performed significantly better in success rate, completion time of jobs and utilization of the resources.

For future research we will focus on adding inherent support for workload prediction mechanisms in IoT layer to allow pro-active distributing of workloads and to enable future-aware scheduling of resource-intensive workloads.