An overview of QoS-aware load balancing techniques in SDN-based IoT networks

Abstract

Increasing and heterogeneous service demands have led to traffic increase, and load imbalance challenges among network entities in the Internet of Things (IoT) environments. It can affect Quality of Service (QoS) parameters. By separating the network control layer from the data layer, Software-Defined Networking (SDN) has drawn the interest of many researchers. Efficient data flow management and better network performance can be reachable through load-balancing techniques in SDN and improve the quality of services in the IoT network. So, the combination of IoT and SDN, with conscious real-time traffic management and load control, plays an influential role in improving the QoS. To give a complete assessment of load-balancing strategies to enhance QoS parameters in SDN-based IoT networks (SD-IoT), a systematic review of recent research is presented here. In addition, the paper provides a comparative analysis of the relevant publications, trends, and future areas of study that are particularly useful for the community of researchers in the field.

Introduction

Today, applying the Internet of Things (IoT) has become one of the most important and attractive topics in the network realm and has attracted the attention of many researchers [1]. Applying IoT in making smart homes, cities, and industries has an impact on health, productivity, and energy. It has made many changes in our lifestyle and provided desirable solutions to address the tasks of users [2,3,4].

IoT is an interconnected network of things that can interact via a network infrastructure to provide or receive services [5]. Service means doing a task on the network to fulfill a set of objectives, such as maximizing reliability and minimizing execution time, and resource cost. The service requirements are separated into functional and non-functional categories. Functional characteristics are those that the service is required to perform. Non-functional features relate to service quality, such as availability, scalability, cost, response time, energy consumption, and security. Non-functional features are sometimes contrasting [6]. So, it is important to create a suitable framework to maintain Quality of Service (QoS) in the IoT network [7]. On the other hand, with the continuous growth of IoT devices, data traffic exponentially increases. This increase in simultaneous data production requires improving QoS [7,8,9]. Consequently, the significance of enhancing QoS parameters to optimize the overall performance of the network and provide novel technologies and communication schemes has multiplied [7, 10].

The IoT domain starts with smart objects with constrained resources in terms of accuracy and data rate, computing power, energy, memory, and storage [11]. These limitations can lead to heavy traffic in some applications, that require enormous computing, storage, and communication resources such as industry, healthcare, and smart cities [12]. The lack of these resources in devices and IoT infrastructure has led to service response time as one of the significant challenges of these applications. Meanwhile, cloud computing in the IoT has found an effective role in solving many of these problems [13].

Cloud-based IoT includes heterogeneous and intelligent devices that constantly exchange heavy traffic from the network to servers. Servers strive to improve the QoS to end users by providing real-time and reliable services [14]. Nevertheless, any delay in response time would negatively influence QoS, particularly in real-time applications [15]. Inadequate traffic distribution across cloud servers also leads some of them to become overloaded, resulting in a QoS decrease. To achieve this objective, it is required to control and balance the traffic load in both the network infrastructure and IoT servers [14]. To solve this problem, fog computing is proposed to meet the real-time requirements of the IoT [16]. Fog servers may be overloaded and unable to handle all requests in terms of resource constraints and an increased number of object requests. It needs dynamic workflow management techniques with significant processing resources that are not available on fog servers [9].

In this regard, the Software-Defined Networking (SDN) model introduces a favourable solution to improve QoS and increase the flexibility of the IoT network. It can manage the infrastructure as well as heterogeneous IoT devices by separating the logical control layer (traffic management decision-making) and the hardware data layer (traffic transfer mechanism to the intended destination) [9, 17]. SDN may forward requests (tasks) of IoT applications such as industry, health, and smart city to the fog or cloud server as near as feasible to address the issue of simultaneous delay and load-balancing while managing the dynamic traffic flow [16, 18]. This feature is obtained by SDN characteristics, such as global network visibility, programmability [19], openness, and virtualization [20], monitoring global network resources. SDN can find optimal load assessment [12] and the optimal location for task processing [21], manage dynamic incoming traffic to network nodes, balance traffic load [22], and thus increase network efficiency and QoS [16]. It is used to develop the IoT network as the SDN-based IoT network (SD-IoT) [16]. In general, the IoT and SDN have a set of complementary requirements and capacities. IoT can strengthen SDN by solving the issues related to scalability, mobility, and real-time requirements. On the other hand, users have a series of QoS requirements that can be answered by combining the IoT and SDN.

The IoT alone cannot solve the challenges related to the QoS. Numerous studies show that SDN is an effective approach to the IoT. By separating the control layer from the infrastructure layer, simplifying the hardware used in the network as much as possible, and having real-time information about all the network conditions (links status and load on the hardware), it is possible to define the QoS from the direction of the user, service provider and infrastructure in SDN. Therefore, new load-balancing algorithms based on QoS can be proposed. The integration of IoT and SDN is helpful to load-balancing, QoS improvement, and traffic engineering techniques applications [20, 23]. SDN can serve a distinct role in the IoT environment, enabling the network to be programmable and dynamically adjustable while ensuring interoperability across heterogeneous IoT networks. By integrating both technologies, the IoT network will have a full view of network resources and task requirements. Network resources can then be efficiently assigned based on the task requirements during a specific time based on load balancing. This study is to find out the increasing demand to handle the workload on resources present at the SD-IoT and to enquire about load-balancing approaches. A comprehensive review has been conducted to evaluate load-balancing approaches of SD-IoT, and these approaches were compared based on different metrics. The following Motivations particularly inspired this article:

• The increased need to understand load distribution techniques in SD-IoT for proper utilization of resources.

• Since the concept of load balancing addresses the improvement of QoS from the directions of users, service providers, and infrastructure providers, it motivates us to focus on load balancing techniques.

• Based on the increasing workload in IoT, the need for load-balancing among resources has been recognized. Despite available research, the existing work has been identified and summarized in a systematic way that depicts issues and challenges for future research work.

Therefore, to express the importance of load-balancing in providing and ensuring the QoS of IoT networks, significant contributions have been performed in this study:

• Highlighting key and influential elements of SDN and IoT in QoS.

• Describing the SD-IoT network architecture and load-balancing problem in SD-IoT.

• Studying and comparing available load-balancing techniques in SD-IoT aiming at improving QoS.

• Defining the QoS parameters used in studied load-balancing approaches.

• Introduction of simulators/tools in designing and testing load-balancing algorithms in SD-IoT networks.

• Future research opportunities in QoS-based load-balancing in SD-IoT.

The background of the SD-IoT network, an introduction to the SD-IoT architecture, and the function of load-balancing in the SD-IoT network to enhance QoS parameters are all covered in Background Sect. 2. The research approach is described in Research method Sect. 3. Research questions and highlights section 4 includes responses to research concerns about analyzing and comparing various load-balancing approaches, as well as QoS factors and their simulators. Discussion section 5 discusses research questions and load-balancing techniques. Future research trends and opportunities section 6 presents future research opportunities. In Conclusion Sect. 7, the conclusion and finally, the references are provided.

Background

We first provide an overview of the concept of SD-IoT. Then, the SD-IoT architecture is explained, and afterwards, the load-balancing technique and some QoS parameters are discussed.

Description of SD-IoT network

SDN network is one of the cost-effective and compatible architectures, with the ability to network reconfiguration (e.g., implying a controller restart based on network traffic), scheduling flexibility, scalability, flow-based, ease of access, and optimal management. SDN has provided opportunities for IoT networks to develop agile networking, load-balancing processes, and QoS improvement to separate the control layer from the data layer [22, 24, 25].

SDN controllers store data regarding fog/cloud servers as well as incoming and outgoing traffic from IoT devices. Based on the load on the servers and certain specified criteria, the SDN controller determines the sort of collaboration between the servers and ensures load-balancing and efficient use of computing, storage, and communication resources [26].

SD-IoT architecture

IoT architecture should be scalable and efficient. It should be able to manage enormous tasks with high QoS [14]. Due to the increasing number of wireless devices, the IoT architecture covers a variety of communication technologies from Long Range Networks (LoRa) and cellular networks to wireless sensor networks [4]. Each potential IoT and SDN design is explained individually in Fig. 1, followed by the final hybrid architecture.

Fig. 1

Types of IoT, SDN, and SDN-Based IoT architectures

The most popular IoT network architecture includes three layers of cloud computing, fog computing, and end devices [27]. The cloud layer provides flexible and efficient computing resources for IoT applications [8, 28]. The cloud is responsible for providing services that require more computing or are not supported by the fog layer [17].

The cloud has QoS restrictions due to the great distance to network devices, including higher costs, delay, energy consumption, carbon emissions, and inefficient resource utilization [11, 17]. The fog layer is a computational fog model that improves communication and processing performance in a variety of applications by putting computing, storage, and communication closer to IoT devices [11, 17]. This leads to QoS improvements, such as reducing service delivery delays, and increasing data rates and bandwidths at the edge of the network [8, 26, 29]. Fog computing seeks to reduce communication and processing overload between edge devices and cloud data centers, thereby preventing network performance (QoS) degradation [17, 30]. However, due to resource constraints and the locality of fog servers, only a limited number of things can be serviced [31]. The end-devices layer is made up of heterogeneous devices with unique IDs and diverse functionalities, as well as users who may connect to the network at any time and place, exchange data, and need high-quality services [8, 22].

At cloud/fog layers, some resources may become overloaded with an increasing number of requests from end devices. Utilizing SDN architecture and the ability of the controllers the manage incoming traffic and allocation to network resources, requests are assigned to the best and closest server in the fog or cloud to establish load-balancing in the network [32, 33]. The SDN network architecture is divided into three layers: data, control, and application [34].

The data layer consists of a collection of packet components (tasks). The tasks are directed to the intended destination. Data transfer between end users occurs according to the rules set by the control layer [26, 32, 35]. The control layer sets network transfer rules and manages workflows [26, 35, 36]. With a global view of the network and workflow awareness, the control layer can monitor network conditions and act as a decision-maker in offloading input tasks to fog/cloud servers to improve QoS [37]. The control layer consists of controllers and is responsible for routing, security, load-balancing, and monitoring [19, 38]. The application layer implements network control logic and strategies and designs services such as analysis, monitoring, transferring plans, manageability, traffic engineering, load-balancing, and security [36, 39].

The ability to design SDN networks may give incentives for optimizing traffic management in IoT networks to deliver services to end devices [12, 40]. On the SD-IoT network, QoS and network performance can be enhanced at the same time [18, 23]. The SD-IoT architecture comprises three levels for IoT QoS management [38]. The layer of the infrastructure consists of network elements, including devices, gateways, and switches that are distributed in different geographical locations [41]. At this layer, transmission elements, such as switches are used to send the input data flow to the upper layers for further processing.

The control/fog layer includes controllers and fog servers as the main components of this layer that are geographically distributed [41, 42]. The controller creates flow rules and policies for the flow tables to manage the workload in the infrastructure layer [41]. It's worth noting that the control layer maintains QoS requirements in terms of network status monitoring and topology discovery, as well as making load-balancing choices based on the application layer's specifications [43, 44]. The application/cloud layer, the highest layer is allocated to IoT services and applications [42]. This layer interacts with the controllers to apply load-balancing, flexibility, and performance optimization [6, 28, 43].

IoT queries/requests are routed over a gateway to SDN switches. The flow route is determined by SDN switches. The switch asks the associated SDN controller for routing information. The flow tables of switches are updated with new rules. Then, the request is routed to a fog/cloud server which has the desired service and tolerable load. Finally, the server provides services to users [36, 45, 46]. Figure 2 shows the service delivery process on the SD-IoT network.

Fig. 2

Service delivery process on the SD-IoT network

Load-balancing in SD-IoT

In this section, the importance of the load-balancing technique to improve the QoS of IoT is explained. Then the role of the SDN controller in load-balancing in the IoT network is discussed. Finally, the types of SDN controller architecture and the introduction of appropriate controller architecture to improve load-balancing and QoS are described.

The workload balance between network resources is the most essential problem that the service provider must address [47]. It is vital to stress QoS criteria while assigning resources to activities. As a result, it is critical to building effective load-balancing strategies for adjusting network traffic flows to minimize network congestion and fulfill the QoS requirements of IoT applications [48]. In load-balancing studies, the distribution of received tasks as well as resource utilization rates is commonly used to make decisions about load distribution [49]. One way to deal with network resource overload is to transfer the load from the overloaded resources to the underloaded ones [44].

The controller may evaluate real-time traffic rules for switches using a global view of the network (for example, current load status, the residual capacity of cloud/fog resources, and congestion level). Tasks are routed to cloud/fog servers in this instance, and the network meets its load-balancing aim [11, 42, 44]. The SDN controller is responsible for managing the load distribution between network nodes and improving the QoS parameters [45]. The architecture of existing controllers can be categorized into two general categories, centralized and decentralized, as shown in Fig. 3.

Fig. 3

Types of SDN controller architecture

The centralized architecture consists of a single controller to manage the entire network and in terms of the problems, such as scalability, accessibility, and reliability, it is not able to meet the requirements of IoT QoS [50, 51]. Of the increase in IoT network traffic, this strategy will be unable to fulfill users' expanding demands and may become a single point of failure [6, 12, 45, 52].

Decentralized architecture, is based on the hierarchy of controllers as primary and secondary controllers. The primary controller can be referred to as the root controller and the secondary controllers as local controllers where the primary controller assigns control to the secondary controller [53, 54]. decentralized controllers can be divided into multi-controller and distributed controller architectures.

A multi-controller architecture is required to manage the traffic of SD-IoT Networks, which can provide scalability and reliability and yet preserve the simplicity of the control function [22]. Through east–west interfaces, there is communication between each controller and the network management [55]. In a large-scale network, the network is divided into multiple domains. Each domain is managed by a controller. Multi-controllers have been implemented to support mobility management, flow processing and flow forwarding in distributed environments [9]. However multiple controller placements are a problem with the issue of the required number of controllers and controller placement to balance the traffic load at minimum delay [9, 56].

Service providers use a distributed controller architecture as a special type of decentralized controller, to manage traffic load and distribute tasks to appropriate resources to control network parameters. Through the interaction of controllers, network utilization and service delivery performance are improved [9, 46, 47]. The architecture of distributed controllers is important for large-scale SD-IoT load-balancing, which improves QoS parameters, such as reliability, scalability, and accessibility [37].

Each controller is linked to several switches. Uneven load distribution across controllers is caused by the nature of static mapping between switches and controllers, unanticipated network traffic, and dynamic topology change [37]. To prevent the overload of controllers, load distribution is known as a load-balancing technique [25]. Each controller is responsible for a specific geographic area that will have a local view of the network status. Information about service requests, workflow transmission, and resource allocation is managed via the coordination of distributed controllers [46].

In a distributed architecture, distributed controllers can be divided into flat and hierarchical architectures. In flat architecture, controllers in the same layer have the same tasks and communicate directly with each other. Controllers are present at numerous layers of a hierarchical design and have various tasks in each layer. Communication and coordination among the lower controllers are the responsibility of the top controllers [20, 57]. To increase the scalability of the network, it is recommended to distribute the traffic load of IoT devices among the controllers and reduce the computational delay in SDN networks under QoS requirements; So, the design of a hierarchically distributed SDN controller system is proposed [10, 45].

The controllers in a distributed and decentralized architecture periodically exchange network control information with each other, but due to the controller's static connection to the switch, as well as changes in the traffic load of the switches, unfavourable load distribution between the controllers can occur and negatively affecting QoS parameters like response time and network throughput [58]. In general, load balancing may aim at preventing overload, removing the overload, or a combination of both. Neural networks [59], prediction [38], and virtualization of network functions [10] were introduced as solutions to prevent overloading. Moreover, informing the network support [60] and periodic tracking [25] [46] have been devised as solutions to eliminating overload in the network.

QoS parameters

The main purpose of load-balancing is to improve the optimal QoS parameters in the network. To assess load-balancing solutions, researchers looked at many factors. To find a better load-balancing algorithm and identify the advantages and disadvantages, several QoS metrics are utilized. Table 1 introduces the most used QoS parameters in various studies to investigate the effect of load-balancing on QoS in the SD-IoT network.

Table 1 Some QoS parameters in related studies

Some other QoS parameters include; Jitter (Deviation from the average data reception delay) [14], stability (distribution of network traffic among resources to maintain service continuity) [16], cost (payment of service cost by the user) [6], processing time (duration of service operation on CPU resources) [22], waiting /transfer time (time required to transfer the task to the server to receive the service) [22], security (protection against attacks to maintain the accuracy of information exchanged in the network) [2], reliability (correct and timely performance of the task) [28], and network lifetime (energy consumed by the network) [67]. These parameters have received less attention in almost all reviewed articles.

Research method

The Systematic Literature Review (SLR) strategy is used to collect and categorize load-balancing techniques in SD-IoT. SLR is a method to find, evaluate, interpret, and combine existing studies related to specific areas and report findings [68, 69]. This section describes the SLR method. This study was performed to increase understanding of load-balancing techniques in SD-IoT.

Data resources

Searching was conducted in September 2023 without any specific time limit and based on the article's title. As a result, 62 articles were found between 2015 and 2023. Research articles in journals and conferences were considered by the IEEE,¹ Springer,² Science Direct,³ Wiley,⁴ ACM,⁵ MDPI,⁶ and Google Scholar⁷ to extract related articles.

Searching strategy

Based on a routine literature review, this paper evaluates current efforts and trends and lays the groundwork for future research on load-balancing in the SD-IoT network to enhance QoS. To begin the search, Google Scholar is chosen as the primary search engine. The search terms are identified based on the planned study subject and queries as a first step in shaping the search field. The search keywords of SD-IoT, IoT, SDN, load-balancing, and QoS were used, and the "AND" and "OR" logical operators were used to link keywords and find related articles. The related studies were thoroughly analyzed and summarized based on the main feature of the study, main tasks of the proposed algorithm, research objective environment, major participation, evaluation tool, data set, and criteria used for evaluation.

Figure 4 shows the selection process for related articles. Search in individual publications is the first step. In the second step, the initial search leads to selecting 310 articles. It does not make sense to read all of these articles because some of them are not directly related to the topic or are of low quality. Therefore, in the third step, those articles are carefully studied and the most appropriate ones are selected for deeper analysis.

Fig. 4

Article identification process overview

Pre-2015 journals rarely addressed load balance-related issues in SD-IoT. So in this step, other types of studies such as reviews, reports, working articles, and non-English articles are ignored. As a result, 129 articles were chosen based on the following criteria: published between 2015 and 2023, English language, and subject relevance. The quality evaluation was the fourth phase. Following an examination of the abstracts and, in some instances, the whole papers, 62 relevant studies were identified. Those papers directly address load-balancing in SD-IoT and the improvement of specific QoS metrics. Related papers on load-balancing in SD-IoT are analyzed and extract significant concerns on optimization challenges.

Highlights and research questions

The highlights and research questions are part of the SLR. To analyze load-balancing in SD-IoT, related research highlights and questions are listed along with the motivations for such highlights and questions. They are as follows:

Highlight 1. Listing the challenges in IoT networks that led to the use of SDN. The growth of traffic from IoT devices leads to congestion and reduced QoS. Therefore, management and control of network resources, scalability, flexibility, and load-balancing seem to improve the QoS. This highlight is explored throughout the paper.

Highlight 2. In SD-IoT, load balancing is critical. The rising demand for network services puts more strain on the network, lowering its efficiency and using more energy. Load balancing helps with network traffic control and QoS, as discussed in Sect. 4.

Question 1. What are the present approaches for load-balancing in SD-IoT to improve QoS parameters? In Sect. 4, load-balancing techniques are categorized based on the policies used in the selected articles.

Question 2. Which QoS parameters are emphasized to evaluate load-balancing techniques in SD-IoT? The answer given to this question in Sect. 4 helps researchers evaluate and recognize their innovations.

Question 3. What are QoS requirements expected of the user, service provider, and infrastructure's directions? Simultaneous QoS optimization for user entities, service providers, and infrastructure may provide mutual advantages and improve network efficiency. This problem is discussed in Sect. 4.

Question 4. What is the frequency of SD-IoT network applications in using load-balancing techniques? Responses to this question are discussed in Sect. 4.

Question 5. Which common simulation tools are used for load-balancing in SD-IoT? To model and simulate load-balancing techniques, some basic aspects should be considered, including simulation scenarios, data set type and format, data storage, and communication protocol to control data traffic between nodes. This question will also be discussed in Sect. 4.

Question 6. What are the future research opportunities and open issues in load-balancing in SD-IoT? Suggestions help researchers identify future research trends and opportunities in improving QoS in the SD-IoT network. A description of research opportunities is provided in Sect. 5.

Qualitative evaluation

The frequency of related publications in each journal over time is shown in Fig. 5. Figure 6 shows the proportion of articles published by each publisher.

Fig. 5

Publication of selected articles in the journal by year of publication

Fig. 6

Circular chart of articles published

Research questions and highlights

The highlights seek to clarify the role of load-balancing in the SD-IoT network and identify challenges and techniques applied to improve QoS. Questions also help identify future research areas. In the following, the symbol “RH”s are used for Research Highlights, and "RQ"s are used for Research Questions to answer the above-mentioned highlights and research questions.

RH1. The IoT makes it possible to control and monitor numerous interconnected intelligent objects, such as physical devices and sensors, from remote locations [19]. IoT devices have limited resources in terms of processing power, memory capacity, bandwidth, and battery life (energy) and they would reduce QoS, especially delays in real-time services [7, 70]. In terms of storage and processing power, cloud computing infrastructure has been suggested. The workload offloading to strong cloud resources through the network is intended to overcome hardware limitations and conserve device energy [71]. Users and service providers alike may profit from cloud computing [47].

Offloading tasks in the cloud for real-time applications due to long distances between devices and cloud resources leads to increased workload processing delay, high power consumption, less mobility support, lack of location awareness, security, privacy, and bandwidth congestion [7, 8]. To address these issues, fog-based computing infrastructure is used to move network resources closer to devices that transmit data to delay-sensitive applications, allowing the network to achieve some sort of balance [2, 72]. Fog computing minimizes resource costs, filters raw data, increases service access, and reduces delay to support real-time applications with lower operating costs [8, 11, 73]. The delay-sensitive workload is locally performed at the edge of the network by fog servers and heavy computational load with lower delay sensitivity in remote cloud data centers [41].

To increase network performance and QoS, SDN technology efficiently distributes network resources to workloads. As the number of service requests increases, network nodes become overloaded, and load-balancing techniques are introduced by controllers to reduce traffic congestion and eliminate overload [36, 39, 74]. Offloading of tasks to resources is performed by SDN controllers that could fully program the network with the aim of providing real-time services, fast and reliable data transfer and, in short, improving the QoS [17, 26, 75]. In general, the SDN-based solution attempts to maximize network capacity while simplifying the management of the IoT [19].

RH2. Increasing demand for IoT applications requires improved resource management to protect QoS, which requires a centralized view of all available network resources. SDN provides a centralized view for controlling network resources and network flows [76]. SD-IoT networks may go beyond the processing capacity of nodes by increasing the demand for IoT applications, causing network congestion and network node overload, and reducing QoS [16, 17]. Traffic management involves dynamic load-balancing strategies to adapt to network circumstances, regulate network nodes, and enhance QoS to take advantage of SD-IoT global visibility and flexible control. Various aspects of load-balancing for IoT tasks can be optimized using the approaches provided by SDN [6]. In SD-IoT, the workload should be balanced between the resources by the SDN controller to provide the desired level of QoS. Load balancing is considered an important component in distributed computing technology that directly affects the availability of system applications and services [77, 78]. The classification of reviewed studies is shown in Fig. 7 based on the main purpose and strategy used.

Fig. 7

Classification of studies reviewed based on the main purpose and strategy used

RQ1. The concept of load-balancing in the SD-IoT network has been the subject of much research. Load balancing plays a significant role in increasing network QoS parameters. In much research, the relationship between the controller and the transmission nodes is considered to control traffic.

The SDN controller is an important component for load-balancing and distributing resources. So far, a variety of load-balancing approaches including migration, routing, demand response, scheduling, offloading, clustering, classification, allocation, admission control, aggregation, virtualization, placement, flow change, and architecture have been presented to improve QoS in the SD-IoT environment. Due to the importance of the load-balancing technique in Table 2, a column entitled load-balancing method has been added in each article to provide the roadmap to the reader. This is one of the novelties in this research paper. Figure 8 presents the percentage of load-balancing techniques in SD-IoT covered by various reviewed articles.

Table 2 Characteristics of routing-based load balancing techniques

Fig. 8

Percentage of techniques considered in reviewed papers

Some techniques based on artificial intelligence and meta-heuristic algorithms are proposed for routing, traffic engineering, resource allocation, management, security, traffic classification, and ultimately QoS optimization. In the event of network congestion, load-balancing algorithms split the traffic load across various flow channels. The load-balancing has been done at the server level in most of the research, and SDN controllers may be utilized to choose servers to transfer tasks. The performance objectives as well as the mechanism used in the studies were reviewed. In Tables 2, 3, 4, 5, 6, 7, 8, 9 and 10 load-balancing techniques based on the centralized or distributed architecture of SDN controllers were considered. The incoming traffic is balanced using the SDN controller and shares the load, which results in guaranteeing quality of service parameters. Controllers direct real-time traffic to resources based on QoS. In general, IoT uses SDN to maximize resource capacity utilization and thereby maintain QoS.

Table 3 Characteristics of offloading-based load balancing techniques

Table 4 Characteristics of architecture-based load balancing techniques

Table 5 Characteristics of classification-based load balancing techniques

Table 6 Characteristics of migration-based load balancing techniques

Table 7 Characteristics of allocation-based load balancing techniques

Table 8 Characteristics of scheduling-based load balancing techniques

Table 9 Characteristics of clustering-based load balancing techniques

Table 10 Characteristics of other reviewed load balancing techniques

The comparisons of selected load-balancing techniques in the SD-IoT network were thoroughly reviewed and analyzed, and the observations are summarized in Tables 2, 3, 4, 5, 6, 7, 8, 9 and 10. Load-balancing techniques have specific QoS parameters. Some researchers have developed a single criterion called the single-objective criterion, while others have found that several criteria, known as multi-objective and many-objective criteria, are more appropriate. Tables 2, 3, 4, 5, 6, 7, 8, 9 and 10 show the type of network and architecture, the load-balancing method, the desired QoS parameters, the balancing entity, and the application.

Table 11 lists the review studies on the issue of load-balancing in the IoT and SDN networks. Each research is represented by parameters such as review type, publication year, article identification process, taxonomy, network type, comparative analysis, future trends, and covered years. Only four articles have used the SLR method to study load-balancing methods, five articles on the IoT, and six articles on SDN. Therefore, the present study is the first study to review load-balancing methods in SD-IoT using the SLR method.

Table 11 Related surveys in the field of load-balancing in SD-IoT networks

Based on the studies related to the SD-IoT network, some load-balancing techniques to improve QoS parameters in the SD-IoT environment were thoroughly studied and analyzed, along with the most important advantages and disadvantages. These approaches are applied to fog, and cloud layers for load balancing and to achieve better resource utilization. Based on the description in Table 12, we can say that there is a need to work more in the area of load-balancing in the fog computing environment, which mainly considers the processing power and overload of the resources. Table 12 shows the journal or conference type with the name of the publication and reference number, as well as the main subject, key contribution, advantages, and disadvantages, which compares existing load balancing techniques in detail based upon the approaches used.

Table 12 An overview of the various load-balancing solutions for SD-IoT

RQ2. At each layer of the network architecture, QoS parameters affect the QoS of the entire network. All QoS parameters for analyzing the load-balancing efficiency are presented in this research so that network performance can be evaluated and recognized, as well as the benefits and drawbacks of load-balancing approaches. Some optimization parameters are in conflict with each other e.g., scalability, resource efficiency, reliability and bandwidth, power consumption, delay, and cost. To support load-balancing decisions, effective and efficient forecasting of QoS values, are important. Any change in network status is a reason to predict QoS before making load-balancing decisions. By predicting the QoS in the IoT, it is possible to increase the utilization of resources.

Numerous studies have been conducted on load-balancing techniques to reveal critical research issues. To improve QoS, various parameters were introduced by researchers in the research background in the realm of single-objective, two-objective, three-objective, or four-objective optimization problems. Having numerous QoS parameters, maintaining a trade-off between parameters from the direction of users, service providers, and infrastructure is another novelty in this paper. A set of Pareto non-dominated solutions is formed as a result of parameter trade-offs. The SDN controller should select which solutions are best for load-balancing. Finally, the user may choose the preferable solution from this set depending on the criteria supplied.

By reducing QoS constraints, a multi-objective/many-objective model can be created based on maximum workflow [109]. In our study, 62 articles were selected to study the QoS parameters used in load-balancing techniques in the SD-IoT network. During the review, 18 parameters have been identified. The parameters used by different researchers in each of the techniques used by load balancing are listed in Table 13. Studies also focused on providing a load-balancing approach with optimal delay.

Table 13 Load-balancing QoS metrics in the reviewed techniques

The QoS metrics considered in the load balancing approaches are grouped into two broad categories; Qualitative metrics and Quantitative metrics. Also, the metrics may be either dependent or independent. The taxonomy of the load balancing metrics is shown in Fig. 9.

Fig. 9

Taxonomy of load balancing metrics

Figure 10 graphically shows the percentage of load-balancing criteria considered in the articles under review. The most used criterion was the delay with 20.5%, followed by the response time with 15.9%. Other criteria such as throughput, resource efficiency, load-balancing, and packet loss rates included 15.1%, 11.4%, 6.8%, and 5.4%, respectively. Then, energy consumption and packet delivery rate accounted for 5.3%, and 3% respectively. Meanwhile, the least used criteria included reliability and network lifetime with approximately 0.7%. Figure 11 shows the percentage of considered parameters in centralized and distributed architectures. In centralized techniques, 58% and 39% of the researchers have attempted to improve delay and throughput, respectively. Also, response time, delay, and resource utilization have the highest attention by the researchers in the distributed techniques.

Fig. 10

Percentage of QoS criteria in the studied techniques

Fig. 11

Percentage of QoS criteria in centralized and distributed techniques

RQ3. QoS parameters help network infrastructure providers to improve network performance and infrastructures. Users can evaluate their needs by evaluating QoS parameters, and service providers can manage the performance and quality of their services with an emphasis on increasing satisfaction and attracting more users.

Many researchers are attempting to offer desired solutions for users, service providers, and infrastructure providers individually. But, the QoS may be analyzed from a combination of a variety of perspectives or directions, including users, service providers, and network infrastructure. Table 14 lists several QoS parameters in each direction, as well as a brief discussion of each.

Table 14 Directions of QoS parameters

Figure 12 shows some QoS parameters in each direction. For example, network architecture is important from the user, service provider, and infrastructure directions. According to the most common load-balancing techniques shown in Fig. 8, parameters such as delay, response time, resource efficiency, and load-balancing are improved from the user and service provider directions.

Fig. 12

QoS parameters from different directions

RQ4. As shown in Fig. 13, most studies were conducted in smart cities with high-traffic loads, such as industry and multimedia-related applications. It shows the importance of IoT applications in improving the quality of human life. Also, the load-balancing techniques used for each application are shown in Table 15.

Fig. 13

IoT applications studied

Table 15 Application-based load-balancing techniques

RQ5. Identifying different simulation tools helps network developers to test load-balancing algorithms to improve QoS. A simulator is a framework that provides a virtual environment to test performance, deployment models, resource allocation, and load balancing. Simulators play a major role in testing and validating, before deploying on real hardware. Every simulator has some unique features and is used to evaluate different parameter performance. Table 16 shows the various load-balancing algorithms/ frameworks, their simulation tools and testbeds, the type of controller and data set, the load-balancing layer, the number of architectural layers, and QoS space dimensions. The studied load-balancing algorithms mainly focus on the selection of fog and edge resources in the control plane to allocate input tasks. Mininet and Matlab tools are mainly used to evaluate the simulation results of load-balancing techniques in the reviewed studies.

Table 16 Common SD-IoT load-balancing tools/simulations

In the following, some of the studied controllers are discussed concerning the load-balancing problem and achieving the desired QoS in SD-IoT networks. Table 17 shows of the Characteristics SDN controllers proposed by the researchers. Moreover, the percentage of evaluation tools/testbeds used in the reviewed articles are shown in Figs. 14 and 15, along with the controllers.

Table 17 Characteristics of the discussed SDN controllers

Fig. 14

Evaluation tools/techniques used in the reviewed articles

Fig. 15

Controllers were used in the reviewed articles

As can be seen in Fig. 16, Load Balancing algorithms can be single objectives, meaning they focus on one performance parameter whereas multi-objectives can focus on two or three parameters and many objectives on four or more than four parameters. Most algorithms reviewed in this literature are multi-objective. In Fig. 17, the load balance layers based on studies are shown.

Fig. 16

Type of Reviewed Algorithms

Fig. 17

Load balancing layer in the studied techniques

Discussion

One of the most significant applications of SDN is load balancing in IoT, which is the process of distributing traffic evenly across multiple servers to optimize resource utilization and ensure high availability of services. SDN-based load-balancing techniques have gained popularity in recent years due to their flexibility, scalability, and cost-effectiveness. However, these techniques also face several challenges and issues that need to be addressed for their wider adoption and improved performance.

Aiming to identify the better mechanism for the load-balancing problem and to distinguish its pros and cons, the load-balancing mechanisms should be evaluated and compared to each other. The optimal migration policy depends on the number of migrations and produced traffic, current QoS parameter values, and SLA. Migrations reduction can minimize the number of active resources, which is effective in QoS parameters. Migration may have some problems such as security, delay, and energy consumption, as well as the complexities of deciding to choose a proper service provider. A good routing can satisfy the QoS requirement and minimize the energy consumption of the entire network. It also can balance distributing traffic loads over network links and increase the network lifetime. Rerouting is related to changing the selected route due to QoS requirements. It prevents network congestion and improves QoS, but may cause routing overhead. Network architecture is designed or evolved based on network parameters such as QoS and load-balancing, which is a time-consuming process. The policy is used during the network design process where there is a possibility of unbalanced resource loading. The offloading technique is done at the time of execution and according to the users' requirements.

Clustering is used in some issues such as routing, security, and quality of service. It can lead to load-balancing, increasing throughput, and network stability. The formation of stable clusters in an overloaded and dynamic network is necessary to reduce bandwidth consumption and prevent congestion. Classification is used for load balancing based on QoS and efficient use of resources. It reduces the waiting time for tasks and increases load-balancing. The resource allocation strategy guarantees the needs of the applications based on the provider's infrastructure resources. It aims for load-balancing and QoS and may have the possibility of resource overload.

An admission control mechanism can control the acceptance of users and avoid load imbalance of task flows. The aggregation of tasks is used to increase the lifetime of the network and reduce the number of tasks sent and received by the network nodes. However, in applications that require end-to-end confidentiality, task aggregation becomes a challenging practice. Virtualization technology through minimizing the operational costs satisfies the user needs in the shortest possible time. The position and location of the controller can increase the acceptance rate of tasks, lack bottlenecks, and improve QoS. At the same time, controller placement delay can be high in large-scale networks.

The purpose of flow changing is to split traffic into multiple paths with minimal congestion for optimal use of resources and reduced response time in high-scale networks. Demand response with developing a demand side management program to control and schedule the user’s tasks are used for traffic routing and load balancing in the entire network. It can lead to a lower end-to-end delay and higher delivery ratio. By utilizing demand response approaches, it is possible to reduce or shift energy consumption from peak hours to periods of less demand. Scheduling the process of mapping tasks to available resources is somewhat based on user requirements. Task Scheduling is important to increase resource utilization by considering the balance between performance and QoS.

To conclude this section, a summary of widely adopted SD-IoT load-balancing techniques along with a description of the approach, layer used, and advantages and disadvantages of each technique is covered in Table 18. Most of the load-balancing techniques have focused on the cloud, and fog/control layers in the SD-IoT network. Mechanisms that are decided before starting the network and are used continuously over time are called static load-balancing techniques and are shown in blue color and Mechanisms that are used and changed depending on the conditions and QoS parameters during the network execution are called dynamic load-balancing techniques are shown in green color.

Table 18 Comparison of load-balancing techniques in SD-IoT

Table 19 contains a list of research questions along with conclusions to plan the survey on load balancing and to determine the current issues on load balancing. These questions contain the basic idea of this article.

Table 19 Review research questions and conclusion

Future research trends and opportunities

In SD-IoT load balancing mechanisms, there are still many issues and challenges that need to be discussed and resolved in the future by extra development and optimization of research. In the following, open research topics in the SD-IoT field will be discovered to answer Question 6 of the study.

Future Direction 1. Considering new QoS parameters: Examining the existing articles in the field of load balancing, it can be concluded by collecting and analyzing data that some QoS parameters such as availability, security, fault tolerance, reliability, network survival, and traffic patterns have been ignored in almost all reviewed articles. Consequently, the adoption of these parameters in load balancing can be an efficient roadmap for future researchers and increase the efficiency of current methods.

Future Direction 2. Multi-directional many-objective QoS: To maintain the QoS from different directions, load balancing is required for the proper use of provisioned resources. It is important to improve QoS by considering multiple combinations of parameters from different directions. A well-designed resource allocation mechanism is a significant issue for users, service providers, and infrastructure providers to keep improving QoS as many objective issues.

Future Direction 3. Multi-constraint many-objective QoS: Examining constraints with improvements in QoS parameters will be attractive as many-objective problems related to the network. With the arrival of new tasks to receive the service, the optimal trade-off is achieved between the objectives, and solutions stated in which the constraints will determine the acceptance of the solutions to the problem. Designing efficient traffic engineering algorithms that can handle diverse traffic patterns is a challenging task. Adopting new optimization algorithms can be very motivating for future studies. Optimization algorithms such as greedy [92], particle swarm [6], Ant colony [101], Lion Swarm [84], and grey wolf [56] are effective for many-objective problems in load-balancing.

Future Direction 4. Application-based load-balancing: Although the smart city is a widely used application in research papers and real-world scenarios, some other applications, such as healthcare and industry have been less studied. Exploration of data and extraction of information in such applications can be considered an interesting open topic for future studies. The type of application can influence the choice of load-balancing technique. The use of load-balancing techniques in different applications is shown in Table 15.

Future Direction 5. Simulation tools observation: To answer question 5, such future research can be defined. Adopting appropriate simulation tools will increase the quality of the study. Almost all proposed load-balancing techniques are tested in the simulation environment. So there is a need to implement load balancing in the real environment. In addition to simulation tools, no single and comprehensive data set is observed in the reviewed articles. Some authors have accepted the randomly generated data as their data set. Appropriate reference datasets gathered in this field can be very useful for future studies.

Future Direction 6. Adaptive and load tolerant load-balancing (trade-off between load-threshold and usual QoS Parameters): As the number of tasks increases and resource saturation challenges require efficient load-balancing approaches to accepting maximum tasks. In this article, multi-directional QoS is discussed from the user, service provider, and infrastructure directions, which leads to many-objective QoS. By trade-off between workload and determining load threshold, two goals of QoS and load threshold are considered, which can be in conflict.

Future Direction 7. Controller-Architecture-based load-balancing technique: Centralized architecture involves a single controller that manages the entire network. While this approach may be effective for small-scale networks, it becomes more challenging to manage larger networks due to communication delays, bottlenecks, and reliability. Decentralized architecture, on the other hand, allows each controller to control its local area, making it a more feasible and economical option for larger networks. Distributed architecture combines controllers of both centralized and decentralized architecture, allowing local controllers to communicate with each other and with a central controller to achieve a global solution. This approach provides a more flexible and adaptable network, making it a promising area for future research. In multi-controller architectures, there is a need for a dynamic load-balancing mechanism that can handle burst traffic and adjust controller loads without compromising traffic balancing. Based on the studies, for centralized controllers, the routing technique is suitable, for multi-controllers, the architecture policy technique, and for distributed controllers, the migration technique is suitable.

Future Direction 8. Blockchain technology: To maintain, update, monitor, and exchange information between controllers to achieve load balance, blockchain architecture can be used. Blockchain technology can provide a secure and decentralized platform for load balancing, allowing for greater transparency and accountability in network management.

Future Direction 9. Prediction-based load-balancing: Depending on the behaviour of service requesters, the status of servers can be used to predict workload and classification of incoming traffic for efficient server allocation and QoS improvement based on several parameters, using Machine learning techniques. Machine learning techniques can help optimize SDN-based load balancing and traffic engineering algorithms by predicting traffic patterns and resource utilization.

Future Direction 10. Heterogeneous in QoS optimization: Most of the studied works have considered the available resources to be homogeneous for the simplicity of the problem, while resources such as CPU can have different capacities, costs, and energy consumption.

Conclusion

IoT applications and produced data volume and dynamic data flow requests have increased over time. It may lead to network overload and congestion, instability in network nodes (switches, controllers, and servers), and lowering QoS. Using SDN architecture, load-balancing methods can transfer the burden across resources and improve the QoS in IoT environments. SDN is a programmable and powerful solution for data flow control in the heterogeneous IoT network; it offers opportunities to design the network and improve QoS by separating the data layer from the control layer. So, in this paper, we described the properties of combined IoT and SDN networks, along with the architecture and effective role of SDN in IoT to meet QoS needs in different directions. A review of the load-balancing literature in SD-IoT networks and their typical features, the available solutions along with their advantages and disadvantages, related QoS parameters, and appropriate tools and testbeds are covered in this review.

To the best of the authors' knowledge, there is no comprehensive research that considered all QoS directions in the field of load-balancing. Also, some parameters such as availability, fault tolerance, and reliability are highlighted as influent QoS parameters. Furthermore, this paper discusses and compares load-balancing technics to show an overview of the latest approaches for upcoming works in this scope.