CN115413044A - A Joint Allocation Method of Computing and Communication Resources in Industrial Wireless Network - Google Patents

Abstract

The present invention relates to industrial wireless networks, in particular to a method for joint allocation of computing and communication resources in industrial wireless networks, including the following steps: establishing an industrial wireless network with edge computing capabilities; establishing a problem prototype for joint allocation of computing and communication resources ;Problem conversion based on Markov decision process;Construction of dual-layer dual deep neural network model based on deep reinforcement learning;Offline training of neural network model until reward convergence;Online execution of computing and communication resource allocation, collaborative processing of heterogeneous industrial tasks. The present invention can support on-demand unloading of heterogeneous industrial tasks by jointly allocating computing and communication resources of industrial wireless networks, and realize "device-edge" resource collaboration; when meeting the deadline requirements of heterogeneous industrial tasks, "device- Under the premise of limiting the computing power, maximum transmit power, and peak interference power of "edge" devices, the total delay in processing heterogeneous industrial tasks is minimized, and collaborative manufacturing is supported.

Description

A Joint Allocation Method of Computing and Communication Resources in Industrial Wireless Network

technical field

The invention relates to the field of industrial wireless networks, in particular to a method for jointly allocating computing and communication resources of industrial wireless networks.

Background technique

With the continuous enhancement of 5G ultra-reliable low-latency communication (Ultra-Reliable Low-Latency Communication, URLLC) technology, 5G-based industrial wireless network capabilities are becoming stronger and stronger, which can support key industrial control tasks and complex production . However, although URLLC can guarantee the deterministic arrival of control commands before the deadline to a certain extent, complex industrial tasks usually require real-time calculations to complete decisions and generate control commands, which poses a huge challenge to industrial field devices with limited resources. The rapid development of Multi-access Edge Computing (MEC) technology and its combination with 5GURLLC provide an effective solution for solving real-time computing of complex production tasks and reducing latency. By jointly deploying MEC servers with industrial base stations, the computing power of industrial field devices can be supplemented and enhanced, and the real-time performance of processing complex tasks can be improved.

However, the inhomogeneous distribution of end-to-end resources in industrial wireless networks, the competition for resources due to massive device access, and the differences in QoS requirements for heterogeneous tasks pose further challenges to the on-demand allocation of computing and communication resources in the network, and become The bottleneck restricting the efficient application of MEC. In order to realize end-edge resource collaboration in industrial wireless networks, the academic community has proposed different resource allocation and computing offloading algorithms to balance computing and communication resource allocation in different scenarios, including single-user-multi-server, multi-user-single-server, and multi-user Scenarios such as user-multiple servers. Related methods aim at minimizing delay, minimizing energy consumption, maximizing throughput, etc., and mainly adopt Deep Reinforcement Learning (DRL) to solve complex problems such as computing decisions, unloading ratios, transmission power, communication bandwidth, and CPU resources. Coupled computing and communication resources for decision-making and allocation to deal with highly dynamic wireless network environments.

However, existing methods pay less attention to the high-concurrency access problem of heterogeneous industrial tasks with different QoS requirements, such as computing-intensive tasks of machine vision and delay-sensitive tasks of motion control. Especially in industrial scenarios, heterogeneous industrial tasks have different deadline requirements and interference restrictions, which have caused great difficulties in the allocation of computing and communication resources in industrial wireless networks.

Contents of the invention

The present invention is oriented to the collaborative processing of heterogeneous high-concurrency tasks in industrial wireless networks in a general multi-user-multi-server scenario, considering the time deadline requirements of heterogeneous tasks, the maximum transmission power of industrial terminals and their peak interference to other devices Power and other constraints, as well as the computing power of network end devices, a joint allocation method of industrial wireless network communication and computing resources based on deep reinforcement learning is proposed, which solves the problem that traditional resource allocation methods are difficult to deal with the state space explosion in a dynamic network environment , which minimizes the total processing delay of heterogeneous industrial tasks, and can support real-time collaborative processing of heterogeneous and highly concurrent industrial tasks such as computing-intensive and delay-sensitive.

The technical solution adopted by the present invention to achieve the above purpose is: a joint allocation method of computing and communication resources in industrial wireless networks, which realizes the collaborative allocation of computing and communication resources between industrial base stations and industrial terminals in industrial wireless networks based on deep reinforcement learning, including The following steps:

1) Establish an industrial wireless network with edge computing capabilities;

2) According to the deadline requirements of heterogeneous industrial tasks, construct an optimization problem about the joint allocation of computing and communication resources between industrial base stations and industrial terminals;

3) converting the optimization problem into a Markov decision process problem;

4) Construct a dual-layer dual deep neural network model based on deep reinforcement learning to solve the Markov decision process problem;

5) Offline training of dual-layer dual deep neural network model to obtain calculation and communication resource allocation results;

6) Industrial base stations and industrial terminals in the industrial wireless network perform computing and communication resource allocation online, perform wireless communication and task offloading, so as to collaboratively process heterogeneous industrial tasks and minimize delay.

The industrial wireless network with edge computing capability includes: N industrial base stations and M industrial terminals;

The industrial base station is configured with an edge computing server, which is used to provide computing resources for multiple industrial terminals and support scheduling of industrial terminals within its coverage;

The industrial terminal is used for local computing of heterogeneous industrial tasks, and supports offloading of heterogeneous industrial tasks to industrial base stations through wireless channels for edge computing.

For the task of a single industrial terminal, it can be unloaded, partially unloaded or fully unloaded to an industrial base station; d _m,n represents the calculation decision, d _m,n = 1 means that the industrial terminal m chooses the industrial base station n to unload, otherwise it does not unload ;

The transmission rate when the industrial terminal performs task offloading through the wireless channel is

表示工业基站n处的噪声，g_m,n和g_m',n分别表示从工业终端m、工业终端m'到工业基站n的信道功率增益，p_m、p_m'分别表示工业终端m、工业终端m'的发射功率。Among them, B _m,n represents the bandwidth,

Indicates the noise at industrial base station n, g _m,n and g _m',n respectively represent the channel power gain from industrial terminal m, industrial terminal m' to industrial base station n, p _m , p _m' represent industrial terminal m, The transmit power of the industrial terminal m'.

The optimization problem of the joint allocation of computing and communication resources between the industrial base station and the industrial terminal is

stC1:0≤p _m ≤P _max ,m=1,...M,

C2:

C3:0≤f _m,n ≤F _n ,

C4:

C5:0≤um _, n≤1,

C6:d _m,n ∈{0,1},

C7:

C8: 0≤τm _{, n≤wm}

为任务目标，即最小化总时延，τ_m,n表示处理工业终端m任务的时延，

分别表示所有工业基站和工业终端的客观计算决策、卸载比例和发射功率；in,

is the task goal, that is, to minimize the total delay, τ _m,n represents the delay in processing industrial terminal m tasks,

respectively represent the objective calculation decision, unloading ratio and transmit power of all industrial base stations and industrial terminals;

C1 and C2 are transmit power constraints; among them, P _max represents the maximum transmit power of the industrial terminal, I _p represents the peak interference power that the industrial terminal can tolerate, and g _m,m' represents the distance between the industrial terminal m and the industrial terminal m' channel power gain;

C3 and C4 are computing resource constraints; among them, f _m,n represent the computing resources allocated by industrial base station n to industrial terminal m, and F _n represent the total computing resources of industrial base station n, which are represented by the number of CPU cycles per unit time;

C5 is the unloading ratio constraint; among them, u _m,n represents the unloading ratio of industrial terminal m offloading tasks to industrial base station n, and its size is between 0 and 1;

C6 and C7 are the constraints of computing decisions; among them, d _{m, n} represent computing decisions, d _{m, n} = 1 means that industrial terminal m chooses industrial base station n to offload tasks; d _{m, n} = 0 means that industrial terminal m does not choose industrial Base station n performs task offloading; each industrial terminal can only select one industrial base station for task offloading, and cannot unload to multiple industrial base stations;

C8 is the task deadline constraint; among them, w _m represents the deadline of the task performed by the industrial terminal m, that is, the longest task processing time that the industrial terminal m can accept.

The task processing delay of the industrial terminal is determined by the edge computing delay and the local computing delay, and the calculation method is as follows:

由通信时延和计算时延决定，计算为The edge computing delay

Determined by the communication delay and calculation delay, calculated as

Among them, C _m represents the number of CPU cycles required by the industrial terminal m to calculate the unit data volume task, and T _m represents the task data volume to be processed by the industrial terminal m;

计算为The local computation delay

calculated as

The process of problem conversion based on the Markov decision process is as follows:

a) Establish a Markov decision model, including state vectors, action vectors, reward vectors and state transition functions;

The state vector is the state of the industrial terminal at time t, denoted as s(t)={T(t), C(t), w(t), v(t), g(t)};

Among them, T(t)={T _m (t)} _M represents the data size set of tasks performed by M industrial terminals, and C(t)={C _m (t)} _M represents the execution unit data of M industrial terminals The set of CPU computing cycles required by the quantity task, v(t)={v _m (t)} _M represents the task priority set of M industrial terminals, w(t)={w _m (t)} _M represents M The set of deadlines for performing tasks by industrial terminals, g(t)={g _m,n (t)} _M×N represents the set of channel power gains between M industrial terminals and N industrial base stations;

The action vector is the action of the industrial terminal at time t, denoted as a(t)={d(t), u(t), p(t)};

Among them, d(t)={d _m,n (t)} _M×N represents the calculation and decision result set of M industrial terminals performing task offloading to N industrial base stations; u(t)={u _m,n (t )} _M×N represents the proportion set of M industrial terminals unloading tasks to N industrial base stations, p(t)={p _m,n (t)} _M×N represents the transmission power set of M industrial terminals;

The reward vector is the reward obtained by the industrial terminal at time t, which is denoted as r(t)={r _m,n (t)} _M×N ;

Among them, the reward obtained by the industrial terminal m is r _m,n (t)=τ _m,n (t)+ρ(τ _m,n (t)-w _m (t)), ρ represents the reward and punishment coefficient of the task, given by task prioritization;

The state transition function is the probability of transitioning from state s(t) to state s(t+1) after executing action a(t) at time t, expressed as f(s(t+1)|s(t) ,a(t));

b) Determine the long-term cumulative reward function; the long-term cumulative reward is

Among them, t ₀ represents the previous moment, and γ∈[0,1] represents the discount coefficient, which is used to indicate the influence of past rewards on current rewards;

c) carry out problem conversion; The problem after described conversion is

max R _p (t)

s.t.C1,C2,C3,C4,C5,C6,C7,C8

In the case of satisfying constraints C1-C8, the long-term cumulative reward is maximized to obtain the best state transition probability, and then an effective delay minimization strategy is obtained.

The two-layer dual deep neural network model based on deep reinforcement learning includes two deep neural networks, which are respectively called the estimated Q network and the target Q network to form an agent; both have the same structure, but use different hyperparameters θ to Generate Q-values.

The offline training of the neural network model until the reward converges comprises the following steps:

a) Extract experience data E(t) from the experience pool as training data;

b) Input s(t) into the estimated Q network to generate action a(t) and Q value Q(s(t), a(t)|θ);

c) Execute a(t), convert s(t) into s(t+1), and get reward r(t);

d) Train the estimated Q network and update the hyperparameter θ of the target Q network in real time;

e) Store the obtained current state, action, reward and next state as experience E(t)={s(t), a(t), r(t), s(t+1)} into the experience pool;

f) Input s(t+1) to the target Q network, get a(t+1) and Q value Q′(s(t),a(t)|θ′), and calculate Q′(s(t) +1), a(t+1)|θ′);

g) updating θ with the stochastic gradient descent method; the stochastic gradient descent method updating θ is realized by the following formula:

为估计Q网络的损失函数，表示目标Q网络与估计Q网络的均方误差；in,

To estimate the loss function of the Q network, it represents the mean square error between the target Q network and the estimated Q network;

h) Execute experience replay, repeat the iterative steps a)-g) until the reward converges to a stable value, and obtain an effective delay minimization strategy, that is, calculation and communication about all industrial base stations and industrial terminals computing decisions, unloading ratios, and transmit power Resource allocation results.

The online execution of computing and communication resource allocation and collaborative processing of heterogeneous industrial tasks includes the following steps:

a) The state vector s(t) of all industrial terminals at the current moment t is used as the input of the agent after offline training, and the output action vector a(t) is obtained;

b) According to the obtained output action vector a(t), all industrial terminals process industrial tasks by allocating computing and communication resources according to the calculation decision, unloading ratio and transmission power in a(t).

The present invention has the following beneficial effects and advantages:

1. Aiming at the highly dynamic industrial network environment, as well as the difficulty in modeling and the explosion of the algorithm state space caused by the complex coupling of communication and computing multi-dimensional resources, the present invention proposes a combination of computing and communication resources for industrial wireless networks by using a deep reinforcement learning method The allocation method realizes offline training and online execution of resource allocation, and can ensure the real-time performance of industrial wireless networks.

2. The present invention is oriented towards the cooperative processing of heterogeneous and highly concurrent industrial tasks in industrial wireless networks, fully considering the time deadline requirements of heterogeneous tasks, the maximum transmission power of industrial terminals and the peak interference power to other devices, etc., As well as the computing power of the network end, a joint allocation method of industrial wireless network communication and computing resources based on deep reinforcement learning is proposed, which can meet the transmission quality requirements of heterogeneous industrial tasks in interference-limited industrial wireless networks, and support heterogeneous Collaborative processing of tasks.

Description of drawings

Fig. 1 is a flow chart of the method of the present invention;

FIG. 2 is a schematic diagram of a multi-user-multi-service industrial wireless network scenario for the present invention;

Fig. 3 is the deep neural network structural diagram that the present invention adopts;

Fig. 4 is a flowchart of deep reinforcement learning training in the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

The present invention is oriented to the "device-side" resource allocation of industrial wireless networks under the general scenario of multi-user-multi-server, and proposes a joint allocation method of industrial wireless network communication and computing resources based on deep reinforcement learning. The present invention can support on-demand unloading of heterogeneous industrial tasks by jointly allocating computing and communication resources of industrial wireless networks, and realize "device-edge" resource collaboration; when meeting the deadline requirements of heterogeneous industrial tasks, "device- Under the premise of limiting the computing power, maximum transmit power, and peak interference power of "edge" devices, the total processing delay of heterogeneous industrial tasks is minimized, and collaborative manufacturing is supported.

A method for joint allocation of industrial wireless network communication and computing resources proposed by the present invention includes the following steps: 1) establishing an industrial wireless network with edge computing capabilities; 2) establishing a problem prototype for joint allocation of computing and communication resources; 3) based on Markov decision process for problem conversion; 4) Construct dual-layer dual deep neural network model based on deep reinforcement learning; 5) Offline training of neural network model until reward convergence; 6) Online execution of computing and communication resource allocation, collaborative processing of heterogeneous industries Task. The overall process of the present invention is shown in Figure 1.

1) Establish an industrial wireless network with edge computing capabilities. As shown in Figure 2, the industrial wireless network includes N industrial base stations and M industrial terminals. Among them, the industrial base station is equipped with an edge computing server, which has strong real-time computing capabilities, can provide computing resources for multiple industrial terminals, and supports the scheduling of industrial terminals within its coverage to meet the computing and communication needs of industrial terminals; And communication capabilities, can process heterogeneous industrial tasks in real time, and support the offloading of heterogeneous industrial tasks to industrial base stations through wireless channels for processing.

According to the calculation and decision-making situation, the task of a single industrial terminal can be unloaded, partially or completely offloaded to the industrial base station, but it can only be offloaded to one industrial base station. Let d _m,n represent the calculation decision, d _m,n = 1 means that the industrial terminal m selects the industrial base station n for offloading, otherwise it does not offload.

When the industrial terminal performs task offloading through the wireless channel, the rate is

表示工业基站n处的噪声，g_m,n和g_m',n分别表示从工业终端m、工业终端m'到工业基站n的信道功率增益，p_m、p_m'分别表示工业终端m、工业终端m'的发射功率。Among them, B _m,n represents the bandwidth,

Indicates the noise at industrial base station n, g _m,n and g _m',n respectively represent the channel power gain from industrial terminal m, industrial terminal m' to industrial base station n, p _m , p _m' represent industrial terminal m, The transmit power of the industrial terminal m'.

2) Establish a problem prototype for the joint allocation of computing and communication resources

The optimization problem of end-side joint computing and communication resource allocation in industrial wireless network is

stC1:0≤p _m ≤P _max ,m=1,...M,

C2:

C3:0≤f _m,n ≤F _n ,

C4:

C5:0≤um _, n≤1,

C6:d _m,n ∈{0,1},

C7:

C8: 0≤τm _{, n≤wm}

分别表示所有工业基站和工业终端的客观计算决策、卸载比例和发射功率。

为任务目标，即最小化总时延。τ_m,n表示处理工业终端m的任务时延，由边缘计算时延和本地计算时延决定，计算方法如下：in,

Represent the objective calculation decision, unloading ratio and transmit power of all industrial base stations and industrial terminals, respectively.

is the mission goal, that is, to minimize the total delay. τ _m,n represents the task delay of processing industrial terminal m, which is determined by edge computing delay and local computing delay. The calculation method is as follows:

由通信时延和计算时延决定，计算为Edge Computing Latency

Determined by the communication delay and calculation delay, calculated as

Among them, C _m represents the number of CPU cycles required by the industrial terminal m to calculate the unit data volume task, and T _m represents the task data volume to be processed by the industrial terminal m.

计算为Local Computing Latency

calculated as

C1 and C2 are transmit power constraints. Among them, p _m represents the transmission power of industrial terminal m, P _max represents the maximum transmission power of industrial terminal, I _p represents the peak interference power that industrial terminal can tolerate, g _m,m' represents the distance between industrial terminal m and industrial terminal m' The channel power gain between.

C3 and C4 are computing resource constraints. Among them, f _{m, n} represent the computing resources allocated by industrial base station n to industrial terminal m, and F _n represent the total computing resources of industrial base station n, which are measured by the number of CPU cycles per unit time.

C5 is the constraint of unloading ratio. Among them, u _m,n represents the unloading ratio of industrial terminal m offloading tasks to industrial base station n, and its size is between 0 and 1.

C6 and C7 are constraints for computing decisions. Among them, d _m,n represents a calculation decision, d _m,n = 1 means that the industrial terminal m selects the industrial base station n for task offloading; d _m,n = 0 means that the industrial terminal m does not select the industrial base station n for task offloading.

C8 is the constraint of task deadline. Among them, w _m represents the deadline of the task performed by the industrial terminal m, that is, it is required that the processing delay of the task of the industrial terminal m should be completed before the maximum time.

3) The process of problem conversion based on Markov decision process is as follows:

a) Establish a Markov decision model, including state vectors, action vectors, reward vectors and state transition functions.

The state vector is the state of the industrial terminal at time t, expressed as s(t)={T(t), C(t), w(t), v(t), g(t)}. Among them, T(t)={T _m (t)} _M represents the data size set of tasks performed by M industrial terminals, and C(t)={C _m (t)} _M represents the execution unit data of M industrial terminals The set of CPU computing cycles required by the quantity task, v(t)={v _m (t)} _M represents the task priority set of M industrial terminals, w(t)={w _m (t)} _M represents M A set of deadlines for industrial terminals to perform tasks, g(t)={g _m,n (t)} _M×N represents a set of channel power gains between M industrial terminals and N industrial base stations.

The action vector is the action of the industrial terminal at time t, expressed as a(t)={d(t),u(t),p(t)}. Among them, d(t)={d _m,n (t)} _M×N represents the calculation and decision result set of M industrial terminals performing task offloading to N industrial base stations; u(t)={u _m,n (t )} _M×N represents the proportion set of M industrial terminals offloading tasks to N industrial base stations, p(t)={p _m,n (t)} _M×N represents the transmit power set of M industrial terminals.

The reward vector is the reward obtained by the industrial terminal at time t, expressed as r(t)={r _m,n (t)} _M×N . Among them, the reward obtained by the industrial terminal m is

r _m,n (t)=τ _m,n (t)+ρ(τ _m,n (t)-w _m (t))

Among them, ρ represents the reward and punishment coefficient of the task, which is determined by the priority of the task.

The state transition function is the probability of transitioning from state s(t) to state s(t+1) after executing action a(t) at time t, expressed as f(s(t+1)|s(t),a (t)).

b) Determine the long-term cumulative reward as

Among them, t ₀ represents the previous moment, and γ∈[0,1] represents the discount coefficient, which is used to indicate the influence of past rewards on current rewards.

c) Perform problem transformation

According to the long-term cumulative function, the optimization problem of computing and communication resource allocation is transformed into

max R _p (t)

s.t.C1,C2,C3,C4,C5,C6,C7,C8

In the case of satisfying constraints C1-C8, maximizing the long-term cumulative reward can obtain the best state transition probability, and then obtain effective calculation and communication resource allocation results, and minimize the delay.

4) Construct a dual-layer dual deep neural network model based on deep reinforcement learning, including two deep neural networks, called the estimated Q-network and the target Q-network, respectively. Both have the same structure, but adopt different hyperparameters θ to generate Q values. Both the estimated Q network and the target Q network adopt a dual architecture, as shown in Figure 3, which contains two branches, V(s) and A(s,a), which respectively indicate the state value of the current state and the advantages of each action.

5) offline training neural network model until the reward converges, as shown in Figure 4, including the following steps:

a) Extract data from the experience pool as training data;

b) Input s(t) into the estimated Q network to generate action a(t) and Q value Q(s(t), a(t)|θ);

c) Execute a(t), convert s(t) into s(t+1), and get reward r(t);

d) Train the estimated Q network and update the hyperparameter θ of the target Q network in real time;

e) Store the obtained current state, action, reward and next state as experience E(t)={s(t), a(t), r(t), s(t+1)} into the experience pool;

f) Input s(t+1) to the target Q network, get a(t+1) and Q value Q′(s(t),a(t)|θ′), and calculate Q′(s(t) +1), a(t+1)|θ′);

g) updating θ with the stochastic gradient descent method; the stochastic gradient descent method updating θ is realized by the following formula:

为估计Q网络的损失函数，表示目标Q网络与估计Q网络的均方误差；in,

To estimate the loss function of the Q network, it represents the mean square error between the target Q network and the estimated Q network;

h) Execute experience replay, repeat iteration steps a)-g) until the reward converges to a stable value, obtain effective calculation and communication resource allocation results, and minimize time delay.

6) Online execution of computing and communication resource allocation, collaborative processing of heterogeneous industrial tasks includes the following steps:

a) The state vector s(t) of all industrial terminals at the current moment t is used as the input of the agent after offline training, and the output action vector a(t) is obtained;

b) According to the obtained output action vector a(t), all industrial terminals process industrial tasks by allocating computing and communication resources according to the calculation decision, unloading ratio and transmission power in a(t).

Claims (9)

1. A method for joint allocation of computing and communication resources of an industrial wireless network, characterized in that, based on deep reinforcement learning, the collaborative allocation of computing and communication resources of industrial base stations and industrial terminals in industrial wireless networks may include the following steps: 1) Establish an industrial wireless network with edge computing capabilities; 2) According to the deadline requirements of heterogeneous industrial tasks, construct an optimization problem about the joint allocation of computing and communication resources between industrial base stations and industrial terminals; 3) converting the optimization problem into a Markov decision process problem; 4) Construct a dual-layer dual deep neural network model based on deep reinforcement learning to solve the Markov decision process problem; 5) Offline training of dual-layer dual deep neural network model to obtain calculation and communication resource allocation results; 6) Industrial base stations and industrial terminals in the industrial wireless network perform computing and communication resource allocation online, perform wireless communication and task offloading, so as to collaboratively process heterogeneous industrial tasks and minimize delay. 2. A method for joint allocation of computing and communication resources for an industrial wireless network according to claim 1, wherein the industrial wireless network with edge computing capabilities includes: N industrial base stations and M industrial terminals; The industrial base station is configured with an edge computing server, which is used to provide computing resources for multiple industrial terminals and support scheduling of industrial terminals within its coverage; The industrial terminal is used for local computing of heterogeneous industrial tasks, and supports offloading of heterogeneous industrial tasks to industrial base stations through wireless channels for edge computing. 3. The computing and communication resource joint allocation method of a kind of industrial wireless network according to claim 2, it is characterized in that, for the task of a single industrial terminal, by not unloading, partially unloading or all unloading to an industrial base station; with d _{m, n} represent calculation decisions, d _{m, n} = 1 means that the industrial terminal m selects the industrial base station n for unloading, otherwise it does not unload; The transmission rate when the industrial terminal performs task offloading through the wireless channel is

Among them, B _m,n represents the bandwidth,

Indicates the noise at industrial base station n, g _m,n and g _m',n respectively represent the channel power gain from industrial terminal m, industrial terminal m' to industrial base station n, p _m , p _m' represent industrial terminal m, The transmit power of the industrial terminal m'. 4. The computing and communication resource joint distribution method of a kind of industrial wireless network according to claim 1, is characterized in that, the optimization problem of the computing and communication resource joint distribution of described industrial base station and industrial terminal is

C3:0≤f _m,n ≤F _n ,

C5:0≤um _, n≤1, C6:d _m,n ∈{0,1},

C8: 0≤τm _{, n≤wm} in,

is the task goal, that is, to minimize the total delay, τ _m,n represents the delay in processing industrial terminal m tasks,

respectively represent the objective calculation decision, unloading ratio and transmit power of all industrial base stations and industrial terminals; C1 and C2 are transmit power constraints; among them, P _max represents the maximum transmit power of the industrial terminal, I _p represents the peak interference power that the industrial terminal can tolerate, and g _m,m' represents the distance between the industrial terminal m and the industrial terminal m' channel power gain; C3 and C4 are computing resource constraints; among them, f _m,n represent the computing resources allocated by industrial base station n to industrial terminal m, and F _n represent the total computing resources of industrial base station n, which are represented by the number of CPU cycles per unit time; C5 is the unloading ratio constraint; among them, u _m,n represents the unloading ratio of industrial terminal m offloading tasks to industrial base station n, and its size is between 0 and 1; C6 and C7 are the constraints of computing decisions; among them, d _{m, n} represent computing decisions, d _{m, n} = 1 means that industrial terminal m chooses industrial base station n to offload tasks; d _{m, n} = 0 means that industrial terminal m does not choose industrial Base station n performs task offloading; each industrial terminal can only select one industrial base station for task offloading, and cannot unload to multiple industrial base stations; C8 is the task deadline constraint; among them, w _m represents the deadline of the task performed by the industrial terminal m, that is, the longest task processing time that the industrial terminal m can accept. 5. A joint allocation method for computing and communication resources of an industrial wireless network according to claim 4, wherein the task processing delay of the industrial terminal is determined by the edge computing delay and the local computing delay, and the calculation Methods as below:

The edge computing delay

Determined by the communication delay and calculation delay, calculated as

Among them, C _m represents the number of CPU cycles required by the industrial terminal m to calculate the unit data volume task, and T _m represents the task data volume to be processed by the industrial terminal m; The local computation delay

calculated as

6. A method for jointly allocating computing and communication resources of an industrial wireless network according to claim 1, wherein the process of converting a problem based on a Markov decision process is as follows: a) Establish a Markov decision model, including state vectors, action vectors, reward vectors and state transition functions; The state vector is the state of the industrial terminal at time t, denoted as s(t)={T(t), C(t), w(t), v(t), g(t)}; Among them, T(t)={T _m (t)} _M represents the data size set of tasks performed by M industrial terminals, and C(t)={C _m (t)} _M represents the execution unit data of M industrial terminals The set of CPU computing cycles required by the quantity task, v(t)={v _m (t)} _M represents the task priority set of M industrial terminals, w(t)={w _m (t)} _M represents M The set of deadlines for performing tasks by industrial terminals, g(t)={g _m,n (t)} _M×N represents the set of channel power gains between M industrial terminals and N industrial base stations; The action vector is the action of the industrial terminal at time t, denoted as a(t)={d(t), u(t), p(t)}; Among them, d(t)={d _m,n (t)} _M×N represents the calculation and decision result set of M industrial terminals performing task offloading to N industrial base stations; u(t)={u _m,n (t )} _M×N represents the proportion set of M industrial terminals unloading tasks to N industrial base stations, p(t)={p _m,n (t)} _M×N represents the transmission power set of M industrial terminals; The reward vector is the reward obtained by the industrial terminal at time t, which is denoted as r(t)={r _m,n (t)} _M×N ; Among them, the reward obtained by the industrial terminal m is r _m,n (t)=τ _m,n (t)+ρ(τ _m,n (t)-w _m (t)), ρ represents the reward and punishment coefficient of the task, given by task prioritization; The state transition function is the probability of transitioning from state s(t) to state s(t+1) after executing action a(t) at time t, expressed as f(s(t+1)|s(t) ,a(t)); b) Determine the long-term cumulative reward function; the long-term cumulative reward is

Among them, t ₀ represents the previous moment, and γ∈[0,1] represents the discount coefficient, which is used to indicate the influence of past rewards on current rewards; c) carry out problem conversion; The problem after described conversion is max R _p (t) s.t.C1,C2,C3,C4,C5,C6,C7,C8 In the case of satisfying constraints C1-C8, the long-term cumulative reward is maximized to obtain the best state transition probability, and then an effective delay minimization strategy is obtained. 7. A method for jointly allocating computing and communication resources of an industrial wireless network according to claim 1, wherein said construction of a dual-layer dual deep neural network model based on deep reinforcement learning includes two deep neural networks, respectively It is called the estimated Q network and the target Q network, which constitute the agent; the two have the same structure, but use different hyperparameters θ to generate the Q value. 8. A method for jointly allocating computing and communication resources of an industrial wireless network according to claim 1, wherein the off-line training of the neural network model until the reward converges comprises the following steps: a) Extract experience data E(t) from the experience pool as training data; b) Input s(t) into the estimated Q network to generate action a(t) and Q value Q(s(t), a(t)|θ); c) Execute a(t), convert s(t) into s(t+1), and get reward r(t); d) Train the estimated Q network and update the hyperparameter θ of the target Q network in real time; e) Store the obtained current state, action, reward and next state as experience E(t)={s(t), a(t), r(t), s(t+1)} into the experience pool; f) Input s(t+1) to the target Q network, get a(t+1) and Q value Q′(s(t),a(t)|θ′), and calculate Q′(s(t) +1), a(t+1)|θ′); g) updating θ with the stochastic gradient descent method; the stochastic gradient descent method updating θ is realized by the following formula:

in,

To estimate the loss function of the Q network, it represents the mean square error between the target Q network and the estimated Q network; h) Execute experience replay, repeat the iterative steps a)-g) until the reward converges to a stable value, and obtain an effective delay minimization strategy, that is, calculation and communication about all industrial base stations and industrial terminals computing decisions, unloading ratios, and transmit power Resource allocation results. 9. A method for jointly allocating computing and communication resources of an industrial wireless network according to claim 1, wherein said online execution of computing and communication resource allocation, and collaborative processing of heterogeneous industrial tasks comprises the following steps: a) The state vector s(t) of all industrial terminals at the current moment t is used as the input of the agent after offline training, and the output action vector a(t) is obtained; b) According to the obtained output action vector a(t), all industrial terminals process industrial tasks by allocating computing and communication resources according to the calculation decision, unloading ratio and transmission power in a(t).

Images