CN114826449B - Map-assisted Internet of vehicles anti-interference communication method based on reinforcement learning - Google Patents

Abstract

A map-assisted Internet of vehicles anti-interference communication method based on reinforcement learning belongs to the fields of wireless communication, internet of vehicles and information safety. The method solves the problem of high-reliability and safe communication of the vehicle-mounted wireless equipment in an intelligent jammer attack environment, obtains information such as the current position of a vehicle, the vehicle density, the position and the size of a shelter and the like by using a map, estimates the channel state between the vehicle and a receiving vehicle, obtains the received signal power and the bit error rate of the first M data packets from the feedback information of the receiving vehicle, adopts a reinforcement learning algorithm to dynamically select the transmission power and the channel of the vehicle-mounted networking wireless communication equipment, and defends wireless interference attack without knowing an attack model of the jammer. The message transmission reliability of the vehicle-mounted wireless communication equipment in a high dynamic environment is effectively improved, and the communication energy consumption of the wireless equipment is reduced.

Description

Map-assisted Internet of vehicles anti-interference communication method based on reinforcement learning

Technical Field

The invention belongs to the field of wireless communication, internet of vehicles and information safety, and particularly relates to a map-assisted Internet of vehicles anti-interference communication method based on reinforcement learning.

Background

The communication equipment of the internet of vehicles has high moving speed, changeable network topology, higher requirements on transmission time delay and anti-interference capability in the application of traffic safety and the like, but electromagnetic signals of an interference machine and peripheral wireless transmitters can block the wireless communication of the vehicle-mounted communication equipment and passenger mobile equipment, the service quality of communication users is reduced, the communication energy consumption of the equipment is increased, and even serious traffic safety accidents are caused. Therefore, the anti-interference communication method of the Internet of vehicles has great significance for ensuring efficient and reliable data transmission between vehicles.

The wireless anti-interference relay technology of the internet of vehicles adopts the unmanned aerial vehicle node to forward the data packet of the wireless communication equipment so as to improve the reliability of data transmission. [ L.Xiao, X.Lu, D.xu, Y.Tang, L.Wang, and W.Zhuang, UAV Relay in VANETs against Smart Jamming with relationship Learning left, IEEE trans.Vehicular Technology, vol.67, no.5, pp.4087-4097, may 2018] proposes a Reinforcement Learning-based vehicle networking Relay power control algorithm, which forwards data packets of vehicle-mounted wireless devices by unmanned planes far away from an interference area, thereby improving the transmission quality of communication and reducing the transmission energy consumption of Relay unmanned planes. Chinese patent CN113055898A proposes an unmanned aerial vehicle position deployment and data chain anti-interference method based on air-ground cooperation Internet of vehicles, and based on an unmanned aerial vehicle dynamic position deployment algorithm, a multilevel spread spectrum and multicarrier modulation technology is adopted to relay and forward a data packet for a vehicle node, so that the error rate of data transmission is reduced.

Power control and channel selection may also be effective against internet of vehicle interference attacks. Chinese patent CN201811129485.9 proposes a communication anti-interference method based on a spectrum information matrix, which optimizes the transmission power and channel selection of a wireless communication device by using a reinforcement learning algorithm for deeply determining a policy gradient, and reduces the complexity of policy selection. [ S.Feng and S.Haykin, cognitive Risk Control for Anti-JammingV2V Communications in Autonomous Vehicle Networks, IEEE trans. Vehicle Technology, vol.68, no.10, pp.9920-9934, oct.2019] proposes an intelligent Anti-interference method for Vehicle networking based on reinforcement learning, jointly optimizes the transmitting power and transmission channel of Vehicle-mounted communication equipment to resist a wireless intelligent jammer based on a multi-arm gambling machine algorithm, and improves the transmission quality of the Vehicle-mounted wireless communication equipment.

Disclosure of Invention

The invention aims to provide a map-assisted anti-interference communication method for the Internet of vehicles based on reinforcement learning, which dynamically optimizes the transmitting power and channel selection of wireless communication equipment of the Internet of vehicles by using a reinforcement learning algorithm so as to adapt to frequent changes of a signal transmission environment caused by high-dynamic characteristics of the Internet of vehicles and solve the problem of high-reliability and safe communication of the wireless equipment of the Internet of vehicles in an attack environment of an intelligent jammer.

The invention comprises the following steps:

step 1: recording the available transmitting power and the number of channels of the wireless communication equipment of the Internet of vehicles as N and C respectively, and recording the transmitting power

Optional channel->

X＝{[iP _max /N，j]} _{1≤i≤N，1≤j≤C} ；

And 2, step: constructing a neural network A and a network B, and respectively recording network parameters thereof as omega ₁ And omega ₂ (ii) a Initializing signal to interference and noise ratio (SINR) gamma ⁽⁰⁾ Bit Error Rate (BER) ρ ⁽⁰⁾ Reception power

Exploration ratio epsilon = epsilon _max ；

And step 3: acquiring a real-time map gamma to obtain information such as the position, the size and the like of a shelter;

and 4, step 4: at the k-th time slot, the vehicle position Y is obtained from a roadside station or the like ^(k) Current vehicle density n ^(k) Obtaining the channel state h with the roadside station through a channel estimation algorithm ^(k) According to the vehicle position Y ^(k) Calculating the distance D between the station and the roadside ^(k) From a real-time map Γ and vehicle position Y ^(k) Determining a channel type k ^(k) Wherein the channel types are divided into line-of-sight communication links (k) ^(k) = 1) and a non line-of-sight communication link (κ) ^(k) ＝-1)；

And 5: constructing a current state vector

Step 6: will state vector s ^(k) Inputting the input into a neural network A to obtain an estimated value Q(s) of a state-action pair ^(k) ，x ^(k) ) (ii) a Choosing the one with the largest Q with a probability of 1-epsilon _i Value of transmitted power

And channel->

To +>

Randomly selecting any transmitting power and channel; sending vehicle on channel +>

Up to power->

Sending the data packet to a roadside station;

and 7: enabling roadside stations to measure received signal power

The bit error rate of M data packets before statistics is greater than or equal to>

And quantized to L order to estimate the SINR gamma ^(k) Forming feedback information>

Feeding back to the vehicle-mounted wireless equipment through a control channel and the like;

and 8: estimating the current energy consumption E ^(k) And calculating:

wherein the content of the first and second substances,

is an indication function: when the variable is true, the value is 1, otherwise, the value is 0;

and step 9: using historical anti-interference experience e ^(k) ＝{s ^(k) ，x ^(k) ，u ^(k) Storing the data into a buffer pool H, and randomly taking out a historical experience e from the buffer pool H after Z time slots ⁽ⁱ⁾ And apply the state vector s ⁽ⁱ⁾ Inputting the neural network B, and recording the output of the neural network B as

Step 10: updating weight parameters of neural network A by adopting stochastic gradient descent algorithm

Namely, it is

And every c time slots apart, order

Step 11: determining a channel type k ^(k) Whether or not it is equal to- κ ^(k-1) If the exploration rate is epsilon = epsilon _max ；

Step 12: judging whether the exploration rate epsilon is larger than epsilon _min If so, then ε = ε - ε ₀ ；

Step 13, repeating steps 3-12 until | Q(s) is satisfied ^(k+1) ，x ^(k+1) )-Q(s ^(k) ，x ^(k) ) If | is less than 0.01, the algorithm converges.

Compared with the prior art, the invention has the following outstanding advantages:

the invention solves the problem of high-reliability safe communication of vehicle-mounted wireless equipment in an intelligent jammer attack environment, obtains information such as the current position of a vehicle, the density of the vehicle, the position and the size of a shelter and the like by using a map, estimates and receives the channel state between the vehicles, obtains the received signal power and the bit error rate of the first M data packets from the feedback information of the received vehicle, dynamically optimizes and optimizes the transmission power and channel selection of the wireless communication equipment of the internet of vehicles by adopting a reinforcement learning algorithm, and defends wireless interference attack without knowing an attack model of the jammer; the vehicle node predicts the future radio frequency spectrum environment and channel state information by using a local map containing buildings, road topology and traffic information, and continuously optimizes the vehicle networking transmission strategy according to the information such as the communication distance between surrounding vehicles and roadside stations, vehicle density, channel gain, bit error rate, received power and the like so as to adaptively resist dynamic interference signals and effectively reduce the transmission power and the bit error rate of the vehicles. The invention effectively improves the message transmission reliability of the vehicle-mounted wireless communication equipment in a high dynamic environment and reduces the communication energy consumption of the wireless equipment.

Drawings

Fig. 1 shows transmission energy consumption of the anti-interference communication method according to the embodiment of the present invention under the line-of-sight communication condition.

Fig. 2 shows transmission energy consumption of the anti-interference communication method according to the embodiment of the present invention under a non-line-of-sight communication condition.

Fig. 3 is a bit error rate of the anti-interference communication method according to the embodiment of the present invention under the line-of-sight communication condition.

Fig. 4 is a bit error rate of the anti-interference communication method according to the embodiment of the present invention under the non-line-of-sight communication condition.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments will be further described with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

The embodiment of the invention comprises the following steps:

step 1: the available transmission power and the number of channels of the wireless communication equipment of the internet of vehicles are respectively N =5 and C =4, and the transmission power is recorded

Selectable channel->

X＝{[20，0]，[20，1]…[100，4]}。

Step 2:constructing a neural network A and a network B which are respectively composed of 4 full-connection layers and initial network parameters omega of the neural network A and the network B ₁ ＝ω ₂ =0. The first full junction layer consists of 6 neurons, the second and third full junction layers consist of 64 neurons, and the fourth full junction layer consists of 30 neurons. Let initial signal to interference and noise ratio (SINR) gamma ⁽⁰⁾ Bit Error Rate (BER) ρ ⁽⁰⁾ Reception power

Are all zero, and the exploration rate epsilon = epsilon _max ＝0.9。

And step 3: and acquiring the real-time map gamma to obtain the information such as the position, the size and the like of the shelter.

And 4, step 4: at the k-th time slot, the vehicle position Y is obtained from a roadside station or the like ^(k) Current vehicle density n ^(k) Obtaining the channel state h to the roadside station through a channel estimation algorithm ^(k) According to the vehicle position Y ^(k) Calculating the distance D between the station and the roadside ^(k) From a real-time map Γ and vehicle position Y ^(k) Determining channel type k ^(k) Wherein the channel types are divided into line-of-sight communication links (k) ^(k) = 1) and a non line-of-sight communication link (κ) ^(k) ＝-1)。

And 5: constructing state vectors

And 6: the state vector s ^(k) Inputting the input into a neural network A to obtain an estimated value Q(s) of a state-action pair ^(k) ，x ^(k) ) (ii) a Choosing the one with the largest Q with a probability of 1-epsilon _i Value of transmitted power

And channel->

To be->

Is randomly selectedDesired transmit power and channel. The sending vehicle is on channel->

On power>

And sending the data packet to the roadside station.

And 7: enabling roadside stations to measure received signal power

Counting bit error rate rho of M =10 data packets before counting ^(k) Quantizing the signal to L =5 order, and estimating the signal-to-interference-and-noise ratio gamma of the received signal ^(k) Forming the feedback information->

And feeds back to the vehicle-mounted wireless equipment through a control channel and the like.

And 8: estimating the current energy consumption E ^(k) And (3) calculating:

wherein the content of the first and second substances,

is an indication function: and when the variable is true, the value is 1, otherwise, the value is 0. Let c ₁ ＝0.2，c ₂ ＝1。

And step 9: using historical anti-interference experience e ^(k) ＝{s ^(k) ，x ^(k) ，u ^(k) Storing the historical experience into a buffer pool H, and randomly taking out a historical experience e from the buffer pool H after Z =10 time slots ⁽ⁱ⁾ And apply the state vector s ⁽ⁱ⁾ Inputting the neural network B, and recording the output of the neural network B as

Step 10: updating with a random gradient descent algorithmWeight parameter of network A

Namely that

And every c time slots apart, order

Where δ =0.1, c =100.

Step 11: determining a channel type k ^(k) Whether or not it is equal to-kappa ^(k-1) If so, the search rate ε = ε _max . Wherein epsilon _max ＝0.9。

Step 12: judging whether the exploration rate epsilon is larger than epsilon _min If so, the search rate ε = ε - ε ₀ . Wherein epsilon _min ＝0.1，ε ₀ ＝0.001。

Step 13: repeating steps 3-12 until | Q(s) is satisfied ^(k+1) ，x ^(k+1) )-Q(s ^(k) ，x ^(k) ) If | is less than 0.01, the algorithm converges.

According to the invention, information such as a map, a vehicle position and vehicle density, parameters such as a channel state vector and communication performance of the Internet of vehicles and the like are utilized, a reinforcement learning algorithm is adopted to dynamically optimize the transmission power and channel selection of the wireless communication equipment of the Internet of vehicles, so that wireless interference attack is prevented, the communication service quality of the wireless communication equipment of the vehicles is effectively improved, and the communication error rate and energy consumption are reduced. As can be seen from FIGS. 1 to 4, the embodiment of the invention can effectively reduce the transmission power consumption and the error rate of the anti-interference communication of the Internet of vehicles under the line-of-sight communication condition and the non-line-of-sight communication condition.

The above-described embodiments are merely preferred embodiments of the present invention, and should not be considered as limiting the scope of the invention. All equivalent changes and modifications made within the scope of the present invention shall fall within the scope of the present invention.

Claims (1)

1. A map-assisted Internet of vehicles anti-interference communication method based on reinforcement learning is characterized by comprising the following steps:

step 1: recording the available transmission power and the number of channels of the wireless communication equipment of the Internet of vehicles as N and C respectively

Optional channel->

X＝{[iP _max /N，]} _{1≤i≤N，1≤j≤C} ；

Step 2: constructing a neural network A and a network B, and respectively recording network parameters thereof as omega ₁ And ω ₂ (ii) a Initializing the SINR gamma ⁽⁰⁾ Bit error rate ρ ⁽⁰⁾ Reception power

Exploration ratio epsilon = epsilon _max ；

And 3, step 3: acquiring a real-time map gamma to obtain the position and size information of the shelter;

and 4, step 4: at the kth time slot, the vehicle position Y is obtained from the roadside station channel ^(k) Current vehicle density n ^(k) Obtaining the channel state h to the roadside station through a channel estimation algorithm ^(k) According to the vehicle position Y ^(k) Calculating the distance D between the station and the roadside ^(k) From real-time map Γ and vehicle location Y ^(k) Determining a channel type k ^(k) Wherein the channel types are divided into line-of-sight communication links κ ^(k) =1 and non line-of-sight communication link κ ^(k) ＝-1；

And 5: constructing a current state vector

Step 6: will state vector s ^(k) Inputting the input into a neural network A to obtain an estimated value Q(s) of a state-action pair ^(k) ，x ^(k) ) (ii) a With a probability of 1-epsilon to select the one with the maximumQ _i Value of transmission power

And channel->

To be->

Randomly selecting any transmitting power and channel; the sending vehicle is on channel->

Up to power->

Sending the data packet to a roadside station;

and 7: enabling roadside stations to measure received signal power

Carrying out statistics on bit error rate rho of M data packets before ^(k) Quantized to L order and used for estimating signal-to-interference-and-noise ratio gamma ^(k) Forming the feedback information->

Feeding back to the vehicle-mounted wireless equipment through a control channel;

and 8: estimating the current energy consumption E ^(k) And calculating:

wherein the content of the first and second substances,

is an indication function: when the variable is true, the value is 1, otherwise, the value is 0;

step (ii) of9: using historical anti-interference experience e ^(k) ＝{s ^(k) ，x ^(k) ，u ^(k) Storing the data into a buffer pool H, and randomly taking out a historical experience e from the buffer pool H after Z time slots ⁽ⁱ⁾ And apply the state vector s ⁽ⁱ⁾ Inputting the neural network B, and recording the output of the neural network B as

Step 10: updating weight parameters of the neural network A by adopting a stochastic gradient descent algorithm

Namely, it is

And every c time slots apart, order

Step 11: determining a channel type k ^(k) Whether or not it is equal to-kappa ^(k-1) When k is ^(k) ＝-κ ^(k-1) If the search rate is larger than the predetermined value, the search rate is smaller than the search rate ε = ε _max ；

Step 12: judging whether the exploration rate epsilon is larger than epsilon _min If so, then ε = ε - ε ₀ ；

Step 13: repeating steps 3-12 until | Q(s) is satisfied ^(k+1) ，x ^(k+1) )-Q(s ^(k) ，x ^(k) ) And | is less than 0.01, namely the algorithm converges.

Images