Smart and efficient EV charging navigation scheme in vehicular edge computing networks

Abstract

With the increasing number of electric fast charging stations (FCSs) deployed along roadsides of both urban roads and highways, the long-distance travel of electric vehicles (EVs) becomes possible. The EV charging navigation scheme is significant for the quality of user experience. However, the variable conditions of both power grid and traffic networks make it a serious challenge. In this paper, we propose an efficient EV charging navigation scheme while considering both the electric and computation resource sharing. With the support of vehicular edge computing networks in intelligent transportation systems (ITSs), EVs perform both the flexible power load and edge computing nodes. When the traffic network in the established route starts to become congested, EVs can select to enter the nearest FCS. In addition to being supplemented by electric resources, EVs also benefit by sharing their own computing resource with FCSs. We formulate the EV charging navigation as a mixed integer programming problem, the EV moving route planning, FCS selection, and staying time in FCSs are optimized, to balance the relationships among the traveling time, traveling cost and reward. To address the influence caused by the randomness of traffic conditions and charging prices, a two-stage charging navigation algorithm combined with $A^{*}$ algorithm and deep reinforcement learning (DRL) is proposed, with a novel designed reward function. Eventually, numerous experimental results show the effectiveness of the proposed schemes.

Introduction

Due to low driving cost and carbon emission, EVs become one of the fastest development in intelligence transportation systems (ITSs) [1, 2]. Combined with the 5G communication technology, EVs transform to the critical part of the vehicular edge computing networks (VECNs), the driving experience of EVs is improved significantly [3, 4]. The supporting environment of EVs is also gradually improved. Along with the growing deployment of FCSs, the long-distance travel of EVs becomes possible. Different from the EV charging in residential or workplace parking lots, an efficient EV charging navigation scheme is necessary to reduce the waiting time and charging cost in FCSs. Especially for the long distance travel scenario, the whole traveling cost and time are affected directly by the selection of FCSs and the moving route planning. However, for the time-varying traffic conditions, the charging prices, and the driving operation limitations, it is challenging to manage the EV charging navigation efficiently.

Recently, multiple studies have been conducted for the EV charging navigation to reduce the charging cost [5, 6], waiting time in FCSs [7, 8], and improve the power grid reliability [9]. To guarantee the electric requirement of the journey, EVs should be charged fully as early as possible. However, EVs can be considered as the flexible power loads, the disordered and frequent charging will make serious impact on the local power grid. Dynamic pricing of charging is an effective method to guide EVs to reduce the traveling cost. Under the real time pricing and other uncertain factors, reinforcement learning (RL) and deep reinforcement learning (DRL) methods are widely used to find the optimal EV charging navigation decisions [10,11,12,13,14]. In addition, compared with the fuel powered vehicle, the charging time of EVs is longer. The waiting time in FCSs should be considered, which will affect the FCS selection and route planning of EVs. For long-distance journey, the objective function of EV charging navigation problems include minimizing the traveling time, charging cost and time, waiting time in FCSs, and energy consumption on road.

With the rapid development of the current internet of thing technologies, EVs become smart and comfortable and the data amount of task need to be processed in real time increases significantly. VECNs, which integrate the computation resources of both roadside unit (RSU)/base station (BS) and moving vehicles on road, provide computing services nearby, become an acceptable solution for the various emerging application requirements in ITS [3, 15]. EVs and VECNs can interact with each other. VECNs empowered EVs. With the support of VECNs, the EVs can obtain real time useful information. For example, the electric charging prices in FCSs and the traffic situation prediction. Thus, EV selects an efficient and comfortable route. EVs constituted VECNs. EVs become the crucial component of VECNs. With the popularization of high level intelligent assisted driving and other applications [16], the computation resource capacity of EVs is improved greatly [17]. Specially, the idle computation resources of the moving vehicles [18], the vehicle platoon [19], and the vehicles at roadside parking lots [20] can be integrated to support the computing needs in VECNs.

Motivated by the above analysis, in this paper, we propose a smart and efficient EV charging navigation scheme in VECNs for a long-distance travel scenario. We extend the functional properties of EVs, both the electric and computation resource sharing between EVs and FCSs are mainly considered. The EVs perform both the flexible power load and edge computing nodes. When the traffic becomes congested in the incoming moving road, EVs can select to enter the roadside FCSs to obtain the electric supplements while avoiding traffic congestion. Moreover, when EVs stay in FCSs, they can share their own computation resources with the local edge computing network to obtain reward. After charging, the EVs may still select stay in the parking lot of FCSs, when the price of computation resource sharing is high, or the road congestion is still serious. We formulate the EV charging navigation as a joint optimization problem to minimize the whole traveling cost, including the traveling time, energy consumption cost, and the resource sharing reward. The EV moving route, FCS selection, and the staying time in FCSs are jointly optimized. For the time-varying traffic conditions and charging prices in FCSs, a two-stage EV charging navigation algorithm combined with the $A^{*}$ algorithm and DRL-based EV charging navigation is proposed. Different from the existing EV charging navigation strategies which focus on how to obtain the power supply quickly and efficiently, the route planning and power resource allocation are studied, and the charging stations ensure that sufficient charging piles and normal voltage can be selected firstly [6], in our work, we take full advantage of the VECNs. We arrange the travel route rationally to balance the traveling time and the charging time. And we use the idle computation resources of EVs during staying in FCSs to obtain reward, which can reduce the local computing pressure of VECNs and the total traveling cost of EVs. The main contributions are summarized as follows.

• We construct a hierarchical system architecture for the EV charging with VECNs. Besides the normal traveling time and cost, both the electric and computation resource sharing are mainly considered.

• We formulate a mixed integer programming optimization problem to minimize the total system cost, including the traveling time, traveling cost, and computation resource sharing reward. The EV moving route, FCS selection and staying time in FCSs are optimized jointly.

• We propose a near optimal two-stage EV charging navigation algorithm combined with the $A^{*}$ algorithm and DRL to solve the formulated optimization problem.

In addition, we give a set of simulation examples to show the efficiency of the proposed schemes. The rest of this paper is organized as follows. We review the related work in “Related work” section. In “System architecture” section, we describe the system architecture. In “System model and problem formulation” section, we introduce the system model and formulate a joint optimization problem. In “Two-stage solution algorithm design” section, a two-stage EV charging navigation algorithm is designed. Simulations are conducted in “Simulation and analysis” section. Finally, we draw conclusions in “Conclusion” section.

Related work

The EV charging strategy and route planning is an inevitable problem in recent years, the traveling time and cost [5, 6, 11, 13, 21], waiting time in FCSs [7, 8, 10], user satisfaction [4, 12, 14, 22] are considered. Mostly, the above factors are analyzed jointly. For the single EV charging scenario, the real time collaboration between smart grids and ITSs is considered in [11], a DRL-based method is used to extract features from massive traffic and power grid data, to make the decision model to learn the optimal charging scheme continuously, the charging cost also be considered. Ref. [12] proposed a flexible smart charging strategy to reduce the grid congestion, and in [13], a model free approach based on safe DRL is proposed to optimize the EV charging/discharging schedules. An adaptive EV charging and routing strategy is studied in [7]. For the multiple EVs’ charging navigation, the problem becomes complexity. Ref. [10] discussed the joint problem of charging navigation and route selection from multiple EVs to multiple FCSs. A deep learning-based low battery EV scheduling is proposed. The EV fleet charging problem is analyzed in [22], combined with dynamic power prices and traffic conditions, the user satisfaction has been considered. Different from the above works focus on the EV own cost minimization, ref. [14] considered the maximization profit of the charging station, and proposed a real time charging scheduling and charging price adjustment strategy based on RL. Ref. [21] proposed a charging path planning and early warning scheme for EVs in the case of insufficient energy.

In VECNs, the moving and parked vehicles are used to provide additional computation resources for the objective vehicles. Ref. [23] proposed a three-layer architecture of VECNs to schedule the offloading tasks, to minimize the task response latency. The multi-hop task offloading scheme in VECNs is studied in [24], the mobility of vehicles on road is considered. Moreover, ref. [25] studied the application of parked vehicles in VECNs, the workload distribution and the social welfare maximization problems are analyzed. In [26], both the parked and moving vehicles are scheduled for the multi-access edge computing networks, the task offloading and resource allocation are jointly optimized. In [27], the computation resources of vehicles are integrated when they pass through the coverage areas of RSUs. For the VECNs empowered EV, ref. [28] proposed a big data analysis system based on mobile edge computing (MEC), MEC server is performed as an intermediary to realize the interactions between EVs and charging stations.

Different from the above cited works, our work combines the EV charging navigation with the VECNs in ITS, EVs perform as both the power load and edge nodes. It considers the optimal EV charging navigation schemes in a long-distance travel scenario. Under the the real time state of charge (SoC) constraint, and the vehicle normal travel should not be affected, the EV charging is arranged reasonably as the relationships among the traveling time, traveling cost and reward are balanced. EVs share their own computation resources to obtain reward when they enter the FCSs for charging. In details, when a special FCS lack of computation resources gives a high reward price, the EVs can preferentially select to enter the FCS, even it may have enough power. When the traffic road becomes congestion, the EVs can select enter the nearest FCS. Moreover, when the incoming traffic road segments still be heavily congested after EV charging, the EV can select stay in the FCS, the whole traveling time, the energy consumption, and the reward are considered. During the whole path, EVs reduce the total travel cost through the flexible transfer of their own computing resources.

Fig. 1

System architecture diagram

System architecture

In this section, we propose the system architecture of VECNs empowered EV charging in ITS. The basic architecture consists of three layers: cloud computing center (CCC) layer, edge computing (EC) layer, and user layer, shown as Fig. 1. The main notations are summarized in Table 1.

CCC layer

CCC functions as a service center for VECNs. With the help of ITS, the CCC obtains the global historical and real time traffic network and power grid information. Moreover, the CCC can process, analyze and manage the received data to predict the dynamic changes of the system, including the traffic conditions, the charging prices and the available charging piles in FCSs.

EC layer

EC layer is composed of RSUs/BSs and other auxiliary facilities in FCSs. Each RSU/BS can cover a wider area, contains one or more FCSs. The EC layer acts as both the connect bridge for the entire framework, and provides the computing service for the ground devices. For the upper level, EC layer enables rapid integration and uploads the processed results. Normally, it includes the data cleaning, sorting and processing. For the lower level, when the RSUs/BSs in EC layer are suffering scarce resources, the received computation tasks can be offloaded back to the vehicles [29, 30].

User layer

User layer includes vehicles and smart devices. The EVs and devices can offload tasks to EC layer for calculation, and also receive tasks back. EVs can make use of their own idle computation resources to help the EC layer to relieve the computation pressure and obtain the corresponding benefits [31].

There are three interactions exist. Both the computation resource and information data sharing exist among the CCC, EC and user layers. The EVs can perform as crowd sensing nodes to collect the traffic information and upload to the CCC layer directly or with the relay of EC layer. The CCC can perform traffic flow forecasting via deep learning methods [32], and provide the forecasting results to EVs to make the optimal charging navigation. In addition, the EC layer integrates the decentralized and independent computation resources in user layer, and collaborates with CCC to provide the latency sensitive and computation intensive services for EVs and devices.

Table 1 Notations used throughout this paper

System model and problem formulation

System model

The system model of EV charging navigation in VECNs for long-distance travel scenario is proposed in Fig. 2. An EV travels from a start point to the destination, a set of FCSs are deployed at roadsides, including the normal service stations and the parking lots. For the traveling distance and the battery capacity, the EV should enter the FCS to obtain electric supplement at least once. With the VECNs, the RSUs/BSs deployed in FCSs can cover a set of traffic road segments. The CCC in ITS performs traffic forecasting with a slot-by-slot fashion, the time interval is fixed as $T_{re}$, and EV can receive the forecasting information of road traffic conditions at the beginning of each slot. As shown in Fig. 2, to avoid the congestion areas, there are two suitable paths for the EV, and the traveling distance, number of passing FCSs, waiting time and charging time in the FCSs are different. The EV can charge at one or several FCSs, the whole traveling time and energy consumption are balanced. It can perform both the electric and computation resource sharing with the FCSs. Providing the high resource sharing prices as incentives to influence the EV’s charging scheme, the total traveling cost of EV is reduced. The problem of regional computation resource imbalance is also alleviated effectively.

Fig. 2

System mode of EV charging navigation in VECNs

Regarding each FCS as a road segment point, denoted as $k_i$, the road segments are represented as $x_{ij}$, the entire route $\{x_{ij}\}$ of the EV $P_x$ is

$$\begin{aligned} P_x=\left\{ k_0,x_{01},k_1,...,k_i,x_{ij},k_j,...,k_N\right\} , \end{aligned}$$

(1)

where the total numbers of road segment points and FCSs are N and K, $N>K$. Three aspects of time consumption are considered respectively, as: the waiting time and actual charging time in FCSs, and the road driving time. The total waiting time in FCSs is $T_w^{total}$, denote $T_w^i$ as the EV’s waiting time in FCS i, we have

$$\begin{aligned} T_w^{total}=\sum \limits _{i=1}^K T_w^i. \end{aligned}$$

(2)

For EVs can provide their own computation resources with FCSs, the waiting time in FCSs consists of two parts, the waiting time before charging $T_{w,before}^i$, and after $T_{w,after}^i$, as

$$\begin{aligned} T_w^i=T_{w,before}^i+T_{w,after}^i. \end{aligned}$$

(3)

When the EV completes charging actions, it has two choices: keep going or continue to stay, $\sigma _i$ indicates a congestion factor in the incoming road, obtained via the CCC in ITS. $\sigma _i=T_r^{ij^{'}}/T_r^{ij}$. Denote $\sigma _{th}$ as the accepted congestion degree of EV.

$$\begin{aligned} T_{w,after}^i=\left\{ \begin{array}{rl} T_{w,after}^i,&{}\,\, if \,\, \sigma _i>\sigma _{th},\\ 0\quad ,&{} \,\, otherwise. \\ \end{array}\right. \end{aligned}$$

(4)

The total charging time of the whole journey $T_c^{total}$ is composed of the time consumption of EV at each FCS, as

$$\begin{aligned} T_c^{total}=\sum \limits _{i=1}^K T_c^i, \end{aligned}$$

(5)

where $T_c^i$ denotes EV’s charging time at FCS i. We set the charging rates of each FCS are fixed, denoted as ${\omega }$. Capacity of EV battery is denoted as $E_{max}$. $e_i^{init}$ and $e_i^{end}$ represent the EV’s start and end SoC in FCS i. The charging time of EV in FCS i is expressed as

$$\begin{aligned} T_c^i=\frac{\left( e_i^{end}-e_i^{init}\right) E_{max}}{\omega }. \end{aligned}$$

(6)

It is not recommended for the EV stay in the parking lot of FCSs in an extended long time, for the congestion in FCS. We have

$$\begin{aligned} T_w^i+T_c^i \le T_{s,th},\quad \forall i. \end{aligned}$$

(7)

where $T_{s,th}$ is the staying time threshold of EV in FCSs. The traveling time of EV on road is represented as

$$\begin{aligned} T_r^{total}=\sum \limits _{i=1}^N T_r^i=\sum \limits _{i,j=1}^N d_{ij}/v_{ij}, \end{aligned}$$

(8)

where $d_{ij}$ is distance and $v_{ij}$ is speed. We adopt the time-of-use(TOU) price mechanism, the power prices are divided into peak, flat and valley ones respectively. The charging price in FCS i, denoted as $\tau _i^p$, shown as

$$\begin{aligned} \tau _i^p=\left\{ \begin{array}{lll} \tau _i^{p_v}, &{} &{} if \,\, T_c^i \in valley\quad period, \\ \tau _i^{p_f}, &{} &{} if \,\, T_c^i \in flat \quad period, \\ \tau _i^{p_p}, &{} &{} if \,\, T_c^i \in peak \quad period. \end{array}\right. \end{aligned}$$

When the power price takes fluctuation during the EV charging, we set that the charging cost of EV in FCS i, denoted as $E_c^i$, is calculated based on the current time period [33], as

$$\begin{aligned} E_c^i=y_i\tau _i^pT_c^i, \end{aligned}$$

(9)

where $y_i\in \{0,1\}$, indicates whether the EV chooses FCS i.

Actually, the driver or passengers may still use the devices on EV, when they wait for charging. Thus, denote $\xi _t$ as the proportion of redundant computation resources at time t, the cost of EV caused by computation resource sharing in FCS i, $E_s^i$ is given as

$$\begin{aligned} E_{s}^{i}=y_iE_cC_s \int _{T_{start}}^{T_{end}}\xi _t dt, \end{aligned}$$

(10)

where $E_c$ denotes the unit price of energy cost, $T_{end}-T_{start}=T_w^i+T_c^i$. The total energy cost of EV in FCSs during the journey is

$$\begin{aligned} E_c^{total}=\sum \limits _{i=1}^K \left( E_{c}^{i}+E_{s}^{i}\right) . \end{aligned}$$

(11)

Normally, the energy consumption of EV moving on road is associated with the traffic conditions, the driving models and other factors [34, 35]. In this paper, we set it is associated with the traveling distance only, denote $\psi _e$ as the average electricity cost per kilometer and the whole traveling cost of EV on road is

$$\begin{aligned} E_{r}^{total}=\sum \limits _{i,j=1}^N r_{ij}\psi _e. \end{aligned}$$

(12)

For the EV can provide its own computation resources to obtain reward, the resource requirements and the charging benefits of each FCS are completely different [27]. The average computation resource price in FCS i is shown as

$$\begin{aligned} p_i=\frac{1}{T_w^i+T_c^i}p_{b}\int _{T_{start}}^{T_{end}}\delta _t exp\left( \frac{\ln C_{i}}{\ln C_{need}}\right) dt, \end{aligned}$$

(13)

where $p_b$ denotes the basic unit price of computation resource sharing, which is determined by ITS. $\delta _t$ indicates the level of resource demand FCS. Set a fixed time interval $T_{re}$, the value of $\delta _t$ is updated once every time slot. When the FCS does not require computation resources, $\delta _t=0$, otherwise, the value of $\delta _t$ increases according to the demand level for resources. $C_i$ denotes the shared computation resource, and $C_{need}$ denotes the total requirements in FCS.

Then, the computation resources sharing reward of EV, $R^{total}$, is shown as

$$\begin{aligned} R^{total}=\sum \limits _{i=1}^K y_ip_iC_i\left( T_c^i+T_w^i\right) . \end{aligned}$$

(14)

Problem formulation

According to the EV SoC, the charging and computation resource sharing prices in FCSs, and the traffic conditions, from the start point, the EV makes the optimal decisions during the whole journey, includes the moving route $\{x_{ij}\}$, the selection of FCS $\{y_i\}$ and the waiting time of EV after charging in FCSs $\left\{T_{w,after}^i\right\}$. The objective is to minimize the total time and energy costs of the journey, and maximize the whole resource sharing reward. The objective function F is shown as

$$\begin{aligned} F={} & {} \eta _1\left( T_w^{total}+T_r^{total}+T_c^{total}\right) \nonumber \\{} & {} +\eta _2\left( E_r^{total}+E_c^{total}\right) -R^{total}, \end{aligned}$$

(15)

where $\eta _1,\eta _2$ represent the conversion cost coefficients of time and energy respectively. Then, the EV charging navigation in VECNs is formulated as problem P1.

$$\begin{aligned} \min \limits _{\left\{x_{i,j},y_{i},T_{w,after}^i\right\}} F, \end{aligned}$$

(16)

$$\begin{aligned} \left\{ \begin{array}{l} C1:\left( e_i^{end}-e_i^{init}\right) E_{max}-E_r^{next}\ge \zeta E_{max}, \\ C2:\sum \limits _{i=1}^K y_i\le m, \\ C3:C_i\le C_{th},\quad \forall i, \\ C4: T_w^i+T_c^i \le T_{s,th}, \quad \forall i, \\ C5:\sum \limits _{i,j}x_{ij}-\sum \limits _{i,j}x_{ji}=\left\{ \begin{array}{lr} 1, &{} i=s_0\\ 0, &{} i\ne s_0, i\notin D\\ -y_i, &{} i\in D \end{array}. \right. \end{array}\right. \end{aligned}$$

where $x_{ij}$ $\in \{0,1\}$. When the EV arrives road segment point j from point i, $x_{ij}=1$, otherwise $x_{ij}=0$, it means that this path is not selected. C1 indicates that when the EV selects the FCS, it must satisfy it has enough energy to reach the selected target FCS. The minimize energy of EV is denoted as $\zeta E_{max}$, $E_r^{next}$ denotes the next reach road segment point of EV. Besides, C2 limits the number of EV charging in the whole process. m denotes the maximum number of charging, $m\le K$. C3 restricts the computation resources from EV must not exceed the resources carried itself $C_{th}$. C4 restricts the staying time in FCS should not exceed the time threshold $T_{s,th}$. C5 indicates that the EV can only travel on a fixed path, the vehicle starts from $k_0$ and passes through the FCSs in the set D, and the paths can be connected in sequence.

P1 is a mixed integer programming problem, it is hard to find the global optimal results directly. Next, we design a near optimal two-stage algorithm to solve it.

Two-stage solution algorithm design

For the EV charging navigation, the issues that need to be considered throughout the whole journey include: the dynamic traffic conditions, idle charging piles, charging prices and demand of computing resources in FCSs, the objective is to optimize the EV moving route, the FCS selection and the staying time in FCSs to obtain the best income of EV. Due to the support of VECNs in ITS, the EV can receive the forecasting traffic conditions. However, the situation of the above mentioned influence factors will be quite different from the start moment, and the charging time of EV in FCSs is relatively long. It is not suitable to plan the global optimal solution for the entire journey at the start point. In addition, if the EV obtains the optimal decision at the beginning of each time slot, it will cause the destination FCS on the vehicle side change constantly. Although the results obtained at the algorithm and data level may be the optimal solutions, it does not conform to the EV usage and charging habits. Therefore, we make use of incremental updates method at different decision points, as the road segment nodes and FCSs. In our work, we make decisions when the EV stays in these three types of road segment points. 1) The start point of EV. The EV should determine the moving route. 2) The road segment points with traffic congestion. When the unexpected congestion occurs, the EV can select enter the nearest FCS to avoid it. 3) FCSs. When the EV passes the FCSs, it can make a decision whether enter the FCS and optimize the staying time in it.

Therefore, we design a two-stage EV charging navigation algorithm to solve problem P1, combined with the $A^*$ algorithm and deep Q-network (DQN), as shown in Algorithm 1. With the help of VECNs in ITS, the EV makes decision at the given road segment points. At the first stage, in the start point, the EV obtains the moving route of the whole journal $\{x_{ij}\}$ in the offline manner, via the $A^*$ algorithm, with the traffic prediction from the CCC in ITS. Then, at the road segment points, the EV obtains the optimal decisions whether enter the FCS $\{y_i\}$ and the corresponding waiting time after charging in the FCS $\{T_{w,after}^i\}$. Moreover, when a sudden traffic jam occurs on the planned traffic route, the EV can select whether to continue to wait on road or go to the nearest FCS based on the forecasting traffic congestion degree. Although the algorithm running time is long, it has enough time to make the final optimal decisions. Next, we will give the details of each stage of the proposed algorithm.

Algorithm 1 Two-stage EV Charging Navigation Algorithm

EV route planning based on $A^{*}$ algorithm

In this section, we use $A^{*}$ algorithm to obtain EV moving route in the start point, based on two reasonable assumptions: the CCC in ITS can provide the traffic prediction for EV, and number of FCSs be deployed at roadsides. Set $\left\{{y_i, T_{w,after}^i}\right\}$ as fixed values, P1 reduces to a subproblem P1-1, as

$$\begin{aligned} \min \limits _{\{x_{i,j}\}} F, \end{aligned}$$

subject to constraints C1 and C5. The $A^{*}$ algorithm is used to find the optimal solution of P1-1. Except for the start and end points, the FCSs are regarded as road segment points. Considering the SoC constraint of EV, the cost of EV in each point is evaluated, and the best one among all the connection points is obtained until reach the destination. Finally the shortest suitable path connects the FCSs in traffic network is obtained. To improve the accuracy and calculation speed, in the $A^{*}$ algorithm, the priority of each point is calculated and be compared to obtain the next highest one. The priority function in point i is shown as

$$\begin{aligned} f(i)=g(i)+h(i), \end{aligned}$$

(17)

where f(i) represents the priority of the point, a smaller value means a higher priority. g(i) represents the cost value of the point i from the start point, h(i) represents the estimated value. We set the EV can move up, down, left and right only. g(i) and h(i) are calculated by the Manhattan distance. Under EV SoC constraint, the point with the minimal f(i) is selected as the next moving one. The details of the proposed $A^{*}$ algorithm are similar with the description in refs. [36, 37].

Fig. 3

DQN algorithm framework

FCS selection based on DQN algorithm

In the section, we perform the FCS selection of EV and the staying time optimization in FCS, when the EV passes the road segment points. After the moving route of EV $\{x_{i,j}\}$ had been optimized via the proposed $A^*$ algorithm, P1 is reduced to a subproblem P1-2, as

$$\begin{aligned} \min \limits _{\left\{y_{i},T_{w,after}^i\right\}} F, \end{aligned}$$

subject to constraints $C2-C5$.

DQN is used to find the optimal solutions. Different from the existing DRL-based EV charging schemes, we design a novel and efficient reward function, corresponding to the scenario that we consider both the electric and computation resource sharing. With the support of the CCC and EC layers in ITS, the EV can obtain the real time and the collected traffic network and power grid information. It makes optimal decisions through the proposed DQN-based algorithm. The framework is shown as Fig. 3. To express the proposed DQN-based algorithm, we introduce the formulation of Markov decision process (MDP) firstly, as:

1)State: To obtain the optimal decision, we consider a set of states, as the estimated traveling time costs $\left\{ T_{t,r}^{01^{'}},T_{t,r}^{12^{'}},\cdots ,T_{t,r}^{ij^{'}}\right\}$, the charging price $\tau _t^p$, the computation resource sharing prices $\left\{ p_0^t,p_1^t,\cdots ,p_i^t\right\}$ in FCS i, SoC of EV $e^t$, EV location information $l_t$, the free charging piles at the FCS i in the path $\left\{ h_0^{num},h_1^{num},\cdots ,h_i^{num}\right\}$,

$$\begin{aligned} S_t={} & {} \left[ \left( T_{t,r}^{01^{'}},T_{t,r}^{12^{'}},\cdots ,T_{t,r}^{ij^{'}}\right) ,\tau _t^p,\left( p_0^t,p_1^t,\cdots ,p_i^t\right) ,\right. \nonumber \\{} & {} \left. e^t,l_t,\left( h_0^{num},h_1^{num},\cdots ,h_i^{num}\right) \right] . \end{aligned}$$

(18)

2)Action: The action of EV $A_t$, corresponds to the actions in each FCS at each step $\{a_t^i\}$. $a_t^i\in \{0,1\}$ indicates whether to charge in FCS i, 0 means it passes and does not perform charging, 1 means entering the FCS for charging.

$$\begin{aligned} A_t\in \{i,0,1\}. \end{aligned}$$

(19)

After charging in FCS i, the EV should determine the waiting time $T_{i,after}^t$, based on the real time traffic conditions $\sigma _{i,t}$. When the traffic road becomes congested, $\sigma _{i,t}\ge \sigma _{th}$, EV can stay in the FCS under constraint C4, otherwise, the EV should leave the FCS after charging, and $T_{i,after}^t=0$.

3)Reward :  In this paper, the benefits of EV are obtained according to the action $A_t$. We define a reward value $r_f$ to influence the EV to make the correct choice, and ensure that $r_f$ is the same magnitude as the reward obtained by computation resources. When the EV does not select the FCS and the battery level falls below the minimum threshold, a negative reward is given. When the EV battery is fully charged but continues to stay in this FCS to provide computation resources, if the road segment ahead becomes congested, the reward for providing computation resources is given. Otherwise, the reward is 0. When the EV enters the target optimal FCS, it will receive a positive reward value of the selection and providing computation resources. When the EV does not choose the FCS with the largest reward to charge, the EV will receive a negative reward and the reward for providing computation resources will be multiplied by 0.8, to encourage the EV to choose the recommended FCS. Therefore, the reward function of EV is expressed as

$$\begin{aligned} {R_t=} \left\{ \begin{array}{l} -r_f, \quad \textrm{if} \quad e^t<\zeta E_{max}, \\ 0, \qquad \textrm{if} \quad e^t=E_{max}, \\ \left( T_w^i+T_c^i\right) p_i^t, \,\, \textrm{if}\,\, e^t=E_{max},\,\, \sigma _i\ge \sigma _{th}, \\ r_f+\left( T_w^i+T_c^i\right) p_i^t, \,\,\text {if EV selects FCS}\,\, i, \\ -r_f+0.8\left( T_w^i+T_c^i\right) p_i^t, \,\,\text {if EV selects other FCSs}. \end{array}\right. \end{aligned}$$

According to the above discussion, Algorithm 2 provides a detailed description of the DQN training process. First, the evaluation and the target network parameters are initialized with the same structure. Besides, the experience pool is initialized to store the samples. A randomly selected probability value $\rho$ and the $\varepsilon$ algorithm are proposed to explore new actions.

$$\begin{aligned} {a_t=} \left\{ \begin{array}{l} \text {randomly select action}, \quad if \quad \rho \le \varepsilon , \\ \textrm{argmax}_{a_t^i} Q\left( s_t^i,a_t^i\right) , \quad if \quad \rho >\varepsilon . \end{array}\right. \end{aligned}$$

When $\rho \le \varepsilon$, the action is chosen randomly, otherwise, the EV selects the action that makes the Q value largest. The optimal policy $\pi ^*$ that makes the action value function maximum is taken out, as

$$\begin{aligned} Q_{\pi ^{*}}=maxQ_\pi (S_t,A_t). \end{aligned}$$

(20)

The loss function for DQN, using the mean squared error (MSE) method to minimize the error between the predicted Q value and the target one. The DQN loss function is shown as

$$\begin{aligned} L(\theta ){} & {} =\left[ \left( y_t-Q\left( s_t^i,a_t^i;\theta \right) \right) ^2\right] , \end{aligned}$$

(21)

$$\begin{aligned} y_t{} & {} =r\left( s_t^i,a_t^i\right) +\gamma maxQ\left( s_{t+1}^i,a_{t+1}^i;\theta \right) , \end{aligned}$$

(22)

where $y_t$ represents the target Q value, $\theta$ is the network parameter. The stochastic gradient descent (SGD) algorithm is used on $L(\theta )$ to update the target DNN parameters.

Algorithm 2 DQN-based FCS Selection Algorithm

Simulation and analysis

Parameter and simulation environment settings

We consider the charging strategy arrangement for long-distance travel of EV. For both the electric and computation resource sharing schemes are considered in this paper, we construct a simulation model, we generate a series of points randomly, and connect them with line segments to simulate roads, each point can be connected with at most four points around the top, bottom, left and right, using line segments to simulate roads. Multiple independent FCSs are deployed in points randomly. We assign a number to each line segment to represent the distance, measured in km. Actually, the current simulation model can be improved to each future realistic models. According to the current market mainstream, the EV battery capacity is 60Ah with maximal driving range 540km, the FCS fast charging rate is 40kw/h, the costs of FCS charging are divided into three types according to different time periods, $[\tau _i^{p_v}$,$\tau _i^{p_f}$,$\tau _i^{p_p}]$ corresponding to [0.55, 1.03, 1.4](CNY/kwh) [6]. FCS connects the local VECNs, and the computation resource sharing prices are in the interval of [0.6, 1.5](CNY/min), different computation resource prices simulate the levels of demand at FCSs. The accepted road congested degree is $\sigma _{th}=0.6$, and the traffic conditions in each road segment are random deployed, $\sigma _{i,t}\in (0,1)$. $\eta _1=\eta _2=0.5$. The generated random map of the simulation model is shown in Fig. 4. Other main simulation parameters are shown in Table 2.

Fig. 4

Random map of simulation

Table 2 Simulation parameters

Results and analysis

EV route planning

The start and the end points are determined Then, the EV route is derived by the $A^{*}$ algorithm, and the result obtained is shown in Fig. 5. Then $A^{*}$ algorithm outputs the coordinates of the points passed in the path as: $[(-398,-212),(-312,-185),(-301,-104), (-218,-106),(-105,-87),(-87,14),(-86,104), (-13,105),(109,107), (184,83),(292,117),(293,197), (296,283),(292,393)].$

Fig. 5

$A^*$ algorithm to find the EV route

It means EV pass through 14 FCSs during the journey. The distances between each two FCSs in the journey are [90, 81, 83, 114, 102, 90, 73, 122, 78, 113, 80, 86, 110]km.

DQN-based FCS selection algorithm

To verify the performance of the proposed DQN-based FCS selection algorithm, four baseline schemes are proposed, as:

• RL-based EV charging algorithm [38, 39] : After EV route is determined, the RL method is applied to find the optimal FCSs.

• Soft Q-learning based EV charging [40, 41]: After EV route is determined, the soft Q-learning algorithm is applied to find the suitable and optimal FCSs.

• FCS selection with SoC: It is a normal scheme. EV selects FCS and staying time in FCS only the SoC constraint C1 is ensured. In this method, several trials are conducted and averaged as an evaluation result.

• FCS selection without reward: The computation resource sharing is not considered, the EV route and FCS selection are optimized jointly, based on SoC requirement only.

Fig. 6

Convergence of both the reward and total system cost F training results solved by DQN

Fig. 7

Convergence of both the reward and total system cost F training results solved by Q-learning

When the initial range of EV is 200 km at the start point, the convergence of charging strategies solved by DQN and Q-learning are shown in Fig. 6 and Fig. 7. To express the performance clearly, we divide the objective function of P1 the total system cost F into reward and cost, as

$$\begin{aligned} F={} & {} cost-reward,\nonumber \\ cost={} & {} \eta _1\left( T_w^{total}+T_r^{total}+T_c^{total}\right) \nonumber \\{} & {} +\eta _2\left( E_r^{total}+E_c^{total}\right) , \nonumber \\ reward={} & {} R^{total}. \end{aligned}$$

(23)

Figure 6 shows that during the training process, the reward obtained by sharing the EV computation resources grows slowly until it reaches convergence finally, while at the same time, the total cost spent by traveling the full path also gradually decreases. Besides, the total system cost F is negative when the training reaches convergence, which indicates that the EV achieves a positive gain when it travels under the proposed DQN-based algorithm, the benefits obtained by EV sharing their idle computation resources with FCSs are outweigh the total cost spent on the road. Comparing Figs. 6 and 7, the training results of Q-learning are not as better as DQN, because DQN introduces target Q network to update the target values, which makes it easier for the parameters to converge during the training process, and the training results are better.

Fig. 8

Total reward for computation resource sharing under different initial SoCs of EV

Figure 8 shows the total reward of EV obtained by sharing computation resources throughout the whole journey with different initial SoCs of EV. We use the initial cruising distance to express the initial SoC of EV. From Fig. 8, we can find that the rewards throughout the journey decrease as the initial SoC increases, the reason is that in the case of a more adequate initial SoC, the necessary charging time decreases, and both the time duration for resource sharing and the number of entering the FCS decrease. The EV under the proposed DQN-based algorithm obtains the best reward, because EV can make the optimal decision about the FCS selection and the staying time in the FCS. The EV under the FCS selection with SoC method obtains reward without a fixed pattern. The method considers SoC as the only constraint. When an EV passes through a FCS, only when the remaining SoC is not enough to reach the next FCS, it selects to charge and provide computation resources. We can find that when the EV SoC is sufficient, the algorithm is more inclined to let the EV choose to continue driving, which can shorten the whole journey time, be in line with daily user driving habits.

Fig. 9

The total costs under different number of congestion points deployed in the path

Figure 9 compares the system costs of EV for DQN-based, RL-based, soft Q-learning based algorithms, FCS selection with SoC and FCS selection without reward under different congestion points are randomly deployed in the whole path. The initial cruising distance of EV is set as 200km, and the length of the whole journey is 1000km. From Fig. 9, we can find that the system costs of EV for FCS selection with SoC and FCS selection without reward increase quickly as the number of congestion points increases. The reason is that the EV selects the charging stations, and only the SoC constraint is considered. When the computation resource sharing rewards are not considered, the costs of EV increase. When there are many congestion points exist, the EV may select stay and wait, the time cost increases. However, the advantages of the DQN, RL-based and soft Q-learning based algorithms are the EV can select to enter the nearest FCS, both the computation and electric resource sharing are considered. The traveling time, energy consumption and the rewards are balanced. And under the proposed DQN-based algorithm, the EV can find the best decisions when it passed each FCS.

Fig. 10

The total system cost F of EV under different total route distances

Figure 10 compares the total system cost F of EV under different total route lengths. The initial cruising distance of EV is set as 200km. We can find that the reward of EV under the proposed DQN-based algorithm is the best than others, and along with the increasing of lengths of the total route, the benefit gap increases. The reason is that the EV under the proposed algorithm has a large solution space to find the optimal decision. However, as the length of route increases, the benefits obtained from the Randomized method by providing computation resources are no longer able to meet the charge expenses of the whole journey, the income of EV becomes negative. Disordered charging selection affects the EV benefit. And the EV under the DQN-based, RL-based, soft Q-learning based algorithms can also achieve positive incomes.

Above all, under the proposed DQN-based FCS selection algorithm, the EV can select the optimal operations in a larger solution space, including the FCS selection and the waiting time. For both the electric and computation resource sharing schemes are considered, the performance of the proposed DQN-based algorithm is the best, compared with other three algorithms.

Conclusion

In this paper, we design a smart and efficient EV charging navigation scheme in a long-distance travel scenario, with the support of VECNs in ITS. Both the electric and computation resource sharing schemes are considered between the EV and FCSs. The EV own computation resource can compensate for the resource storage of VECN in FCS at peak times. The moving route, FCS selection and staying time in FCS are optimized jointly. We establish a two-stage solution algorithm, combined the $A^{*}$-based EV route planning and DRL-based FCS selection algorithms, to find the optimal EV charging navigation scheme. The geographical imbalance of VECN computation resources in path is solved effectively. Simulation results show that the proposed algorithm reduces the EV traveling costs, improves the traveling efficiency and makes reasonable use of global electric and computation resources.