KR102257536B1 - A Method and Apparatus for Distributed Congestion Control of VANET using Reinforcement Learning Based on Neural Network Model - Google Patents

Abstract

The present invention relates to a method and an apparatus for distributed congestion control of a VANET using a reinforcement learning based neural network model. The distributed congestion control method of the VANET using the reinforcement learning based neural network model according to an embodiment of the present invention comprises the steps of: (a) calculating at least one among a channel busy ratio (CBR) of a wireless channel for a V2X communication service, and a channel occupancy ratio (CR), a movement speed and a transmission power of a vehicle; and (b) applying at least one among the CBR of the wireless channel for the V2X communication service, and the CR, the moving speed and the transmission power of the vehicle to the reinforcement learning-based neural network model to determine the maximum CR and the maximum transmission power of the vehicle. The present invention enables a V2X terminal to be optimally adapted to wave resource allocation.

Description

{A Method and Apparatus for Distributed Congestion Control of VANET using Reinforcement Learning Based on Neural Network Model}

The present invention relates to a VANET (Vehicular Ad-hoc network), and more particularly, to a method and apparatus for distributed congestion control of VANET using a neural network model based on reinforcement learning.

Collision Avoidance (CA) Between vehicles To support safety services or autonomous driving, the V2X (Vehicular to Everything) communication terminal of each vehicle, OBU (On Board unit), sends a message containing various information to a number of surrounding objects. Simultaneously broadcast transmission to (automobiles, people, roadside units (RSU), etc.).

In order to transmit messages, each vehicle performs competition-based channel access.If the number of competing vehicles increases, the probability of transmitted messages colliding increases, and accordingly, CA safety service or autonomous driving support service It can increase the likelihood of failure.

Therefore, when the number of vehicles increases and the contention for channel access becomes severe, each vehicle reduces the transmission rate of messages by itself to reduce the total amount of messages generated per second to reduce the probability of collision between messages, or to receive messages by lowering the transmission power. Technology that reduces the number of competing vehicles can be used by reducing the spatial extent of the competition. This technology is called Distributed Congestion Control (DCC) and is standardized for V2X communication technology using IEEE 802.11p physical and MAC (Medium Access Control) technology, and standardization is in progress for 3GPP C-V2X communication technology. In progress.

This conventional DCC technology measures CBR (Channel Busy Ratio), a parameter representing the channel load (CL), which is the total amount of messages generated by vehicles competing for channel access, and according to the CBR measured in each vehicle. By individually adjusting the channel occupancy ratio (CR), which is the channel capacity used by each vehicle, the total amount of channel load generated by competing vehicles for channel access can be adjusted.

The prior art uses a method of mapping _{the maximum value (CR limit} ) of radio resources that can be used by each vehicle according to the section to which the CBR measured in each vehicle belongs as shown in FIG. 1. The mapping table of the CBR value of FIG. 1 and the corresponding CR _limit value is derived by the following process.

First, the ratio of the radio resources in which the RSSI (Received Signal Strength Indicator) indicating the strength of the received signal exceeds a predetermined criterion is measured by CBR among the total radio resources in the frequency and time dimensions that can be used by the V2X receiver of each vehicle.

Using the measured CBR value, the number of competing vehicles, N _sta , is estimated as shown in Equation 1 below.

Since the channel usage indicated by CBR is proportional to the number of vehicles performing V2X communication using radio resources, N _sta can be expressed as an arbitrary function f() using CBR as an input value.

_{The maximum CR limit} of the channel capacity that each vehicle can use can be _{obtained by dividing CR total_limit} , the maximum value of the total amount of radio resources, by N _sta , as shown in Equation 2 below.

Therefore, the most important process of determining the accuracy of the prior art for determining _{the maximum CR limit} of the channel capacity that each vehicle can use using the measured CBR is the process of determining _{N sta from the measured CBR of <Equation 1>.} to be.

However, depending on the type of road (intersection, straight line, road width), vehicle movement speed, and the required amount of radio wave resources according to QoS required for V2X service, the function f() for determining _{N sta by receiving CBR is changed.}

However, since the prior art only defines one f(), it is not possible to determine _{a CR limit} applicable to various road environments, vehicle driving conditions, and various QoS of V2X services. In addition, in order to adapt to various road environments, driving conditions, etc. using the prior art, since numerous functions f() and corresponding mapping tables must be defined according to various environments, practical application is not possible.

[Patent Document 1] Korean Patent Registration No. 10-2002807

The present invention was created to solve the above-described problem, and an object of the present invention is to provide a method and apparatus for controlling distributed congestion of a VANET using a neural network model based on reinforcement learning.

In addition, the present invention uses a reinforcement learning-based neural network, in consideration of various road environments, vehicle operation conditions, and different QoS of various V2X communication services in VANET (Vehicular Ad-hoc network), and allocates radio resources by itself to a vehicle V2X terminal. It is an object of the present invention to provide a distributed congestion control method and apparatus that optimally adapts to.

The objects of the present invention are not limited to the objects mentioned above, and other objects that are not mentioned will be clearly understood from the following description.

In order to achieve the above objectives, VANET's distributed congestion control method using a reinforcement learning-based neural network model includes (a) a channel busy ratio (CBR) of a wireless channel for a V2X communication service, and a channel occupancy ratio of a vehicle ( calculating at least one of channel occupancy ratio, CR), movement speed, and transmission power; And (b) applying at least one of a channel congestion ratio of a radio channel for the V2X communication service, a channel occupancy ratio of the vehicle, a movement speed, and a transmission power to a neural network model based on reinforcement learning, and the maximum channel occupancy ratio of the vehicle. And determining a maximum transmission power. It may include.

In an embodiment, in the step (b), at least one of a channel congestion ratio of a radio channel for the V2X communication service, a channel occupancy ratio of the vehicle, a movement speed, and a transmission power is applied to the reinforcement learning-based neural network model. It may include; calculating a result value of the reinforcement learning-based neural network model.

In an embodiment, step (b) may include determining a maximum channel occupancy ratio and a maximum transmission power of the vehicle according to a result value of the reinforcement learning-based neural network model.

In an embodiment, the step (b) includes: immediately determining a degree that the channel congestion ratio approaches a target target channel congestion ratio as a compensation value according to the maximum occupancy ratio and the maximum transmission power of the vehicle, and the transmission rate ( throughput) and time delay (latency) may include determining a degree of achievement for a target value as a future compensation value.

In an embodiment, the step (b) may include training the reinforcement learning-based neural network model by using the current total compensation value obtained by combining the immediate compensation value and the future compensation value.

In an embodiment, the step (b) includes determining whether a difference between the current total compensation value and the previous total compensation value is greater than a threshold value, and the current total compensation value and the previous total compensation value. When the difference is greater than the threshold value, the message for the V2X communication service may be transmitted according to the maximum channel occupancy ratio and the maximum transmission power of the vehicle.

In an embodiment, the VANET distributed congestion control apparatus using the reinforcement learning-based neural network model includes a channel busy ratio (CBR) of a wireless channel for a V2X communication service, and a channel occupancy ratio (CR) of the vehicle. , Movement speed, and transmission power, and at least one of the channel congestion ratio of the wireless channel for the V2X communication service, the channel occupancy ratio of the vehicle, the movement speed and the transmission power is applied to the reinforcement learning-based neural network model Thus, it may include a; a control unit for determining the maximum channel occupancy ratio and the maximum transmission power of the vehicle.

In an embodiment, the control unit applies at least one of a channel congestion ratio of a wireless channel for the V2X communication service, a channel occupancy ratio of the vehicle, a movement speed, and a transmission power to the reinforcement learning-based neural network model, and the reinforcement learning It is possible to calculate the result value of the underlying neural network model.

In an embodiment, the controller may determine a maximum channel occupancy ratio and a maximum transmission power of the vehicle according to a result value of the reinforcement learning-based neural network model.

In an embodiment, the control unit immediately determines a degree that the channel congestion ratio is close to a target target channel congestion ratio as a compensation value, and compensates for a degree of achievement of a target value of throughput and time delay in the future. Can be determined by value.

In an embodiment, the controller may train the reinforcement learning-based neural network model by using a current total compensation value obtained by combining the immediate compensation value and the future compensation value.

In an embodiment, the control unit determines whether a difference between the current total compensation value and the previous total compensation value is greater than a threshold value, and the difference between the current total compensation value and the previous total compensation value When is greater than the threshold value, a communication unit for transmitting a message for the V2X communication service according to the maximum channel occupancy rate and maximum transmission power of the vehicle; may be further included.

The distributed congestion control method for VANET based on reinforcement learning according to an embodiment of the present invention estimates the channel congestion degree CBR in the vehicle V2X terminal, and determines the current vehicle's radio resource share CR, movement speed, transmission power, and V2X service QoS. Using a reinforcement learning-based neural network that takes the channel congestion CBR, radio resource share CR, movement speed, transmission power, and V2X service QoS as inputs, the maximum CR _limit and transmission power of the radio resource share that the vehicle can use. The maximum value P _limit can be determined.

The VANET distributed congestion control device using a reinforcement learning-based neural network according to an embodiment of the present invention includes a storage device that stores variables of the neural network and actions and compensation for later learning, and the channel congestion degree CBR of the current channel and the own difference ( ego vehicle)'s current radio resource occupancy CR, transmission power P, movement speed v, and a data collection device that estimates QoS of V2X service, and the channel congestion CBR, radio resource occupancy CR, movement speed, transmission power, V2X service QoS It may include a control device that determines the _{maximum value CR limit} of the radio resource share that the vehicle can use and the maximum transmission power value P _limit by using a reinforcement learning-based neural network taking as an input.

Detailed matters for achieving the above objects will become apparent with reference to embodiments to be described later in detail together with the accompanying drawings.

However, the present invention is not limited to the embodiments disclosed below, but may be configured in various different forms, so that the disclosure of the present invention is complete and those of ordinary skill in the technical field to which the present invention pertains ( Hereinafter, it is provided in order to completely inform the scope of the invention to the "normal engineer").

According to an embodiment of the present invention, in consideration of various road environments, vehicle operation states, and different QoS of various V2X communication services in a vehicle ad-hoc network (VANET), a vehicle V2X terminal can optimally adapt radio resource allocation by itself. have.

The effects of the present invention are not limited to the above-described effects, and the potential effects expected by the technical features of the present invention will be clearly understood from the following description.

1 is a diagram showing a mapping table of a conventional CBR and a CR _{limit according thereto.}
2 is a diagram illustrating a distributed congestion control apparatus of VANET using a neural network model based on reinforcement learning according to an embodiment of the present invention.
3 is a diagram illustrating a method of operating a distributed congestion control apparatus of VANET using a neural network model based on reinforcement learning according to an embodiment of the present invention.
4 is a diagram showing a functional configuration of a control unit of a distributed congestion control apparatus according to an embodiment of the present invention.
5 is a diagram illustrating a method of operating a control unit of a distributed congestion control apparatus for learning a neural network model based on reinforcement learning according to an embodiment of the present invention.
6 is a diagram illustrating a detailed distributed congestion control operation method of a distributed congestion control apparatus of VANET using a previously learned reinforcement learning-based neural network model according to an embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail.

Various features of the invention disclosed in the claims may be better understood in view of the drawings and detailed description. The apparatus, method, preparation method, and various embodiments disclosed in the specification are provided for illustration purposes. The disclosed structural and functional features are intended to enable a person skilled in the art to specifically implement various embodiments, and are not intended to limit the scope of the invention. The disclosed terms and sentences are intended to describe various features of the disclosed invention in an easy to understand manner, and are not intended to limit the scope of the invention.

In describing the present invention, when it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted.

Hereinafter, a method and apparatus for controlling distributed congestion of VANET using a neural network model based on reinforcement learning according to an embodiment of the present invention will be described.

The present invention uses a reinforcement learning-based neural network, in consideration of various road environments, vehicle operation conditions, and different QoS of various V2X communication services in a VANET (Vehicular Ad-hoc network), and optimizes the allocation of radio resources by itself to a vehicle V2X terminal. It is an object of the present invention to provide a method and apparatus for distributed congestion control that is adapted to.

2 is a diagram illustrating a distributed congestion control apparatus 100 of a vehicle ad hoc network (VANET) using a neural network model based on reinforcement learning according to an embodiment of the present invention.

Referring to FIG. 2, the distributed congestion control apparatus 100 may be integrated with a vehicle providing a V2X communication service and configured as a single entity, or may be separated and configured as a separate entity. For example, the distributed congestion control apparatus 100 may be located inside a vehicle to everything (V2X) communication terminal of each vehicle.

The distributed mixing control apparatus 100 may include a control unit 101, a communication unit 102, a storage unit 103, and a vehicle network interface unit 104. In this case, the communication unit 102 may include a V2X receiving unit 105 and a V2X transmitting unit 106.

The control unit 101 transmits the channel busy ratio (CBR) of the current wireless channel and the current channel occupancy ratio (CR) of the ego vehicle using the information of the V2X receiver 105. Power P can be estimated.

In one embodiment, the channel congestion rate may be referred to as a channel congestion degree or a term having a technical meaning equivalent thereto. In addition, the channel occupancy ratio may be referred to as radio resource occupancy or a term having a technical meaning equivalent thereto.

The control unit 101 may estimate the moving speed v of the own vehicle and the quality of service (QoS) of the V2X service through the vehicle network interface unit 104.

The control unit 101 can be used by each vehicle using a reinforcement learning-based neural network model that inputs the channel congestion ratio CBR of the wireless channel, the channel occupancy ratio CR of the vehicle, the movement speed v, the transmission power P, and the QoS of the V2X service. The maximum channel occupancy ratio CR _limit and the maximum transmission power P _limit can be determined.

In an embodiment, the control unit 101 may include at least one processor or a micro processor, or may be a part of a processor. Also, the control unit 101 may be referred to as a communication processor (CP). The controller 101 may control the operation of the distributed congestion control apparatus 100 according to various embodiments of the present disclosure.

The storage unit 103 stores the variables of the reinforcement learning-based neural network model and actions ( _{determining and performing the maximum channel occupancy ratio CR limit} and the maximum transmission power P _limit ) and compensation values (achievement of the target performance) for later learning. I can.

In one embodiment, the storage unit 103 may be formed of a volatile memory, a nonvolatile memory, or a combination of a volatile memory and a nonvolatile memory. In addition, the storage unit 103 may provide stored data according to the request of the control unit 101.

The communication unit 102 may transmit the V2X service message using the _{maximum channel occupancy ratio CR limit} and the maximum transmission power P _limit determined by performing distributed congestion control.

In one embodiment, the communication unit 102 may include at least one of a wired communication module and a wireless communication module. All or part of the communication unit 102 may be referred to as a'transmitter', a'receiver', or a'transceiver'.

V2X service messages transmitted from each distributed congestion control device 100 may be transmitted through a VANET wireless channel. In this case, a channel congestion ratio CBR (Channel Busy Ratio) may be determined by a channel load (CL), which is the total amount of messages generated by the V2X transmitter 106 of each vehicle.

The CBR may be estimated by the V2X receiving unit 105 and the control unit 101 of each distributed congestion control device 100 for transmission of the next step message.

Referring to FIG. 2, the distributed congestion control apparatus 100 may include a control unit 101, a communication unit 102, a storage unit 103, and a vehicle network interface unit 104. In various embodiments of the present invention, since the configurations described in FIG. 2 are not essential, the distributed congestion control apparatus 100 may have more or fewer configurations than those described in FIG. 2. I can.

3 is a diagram illustrating a method of operating a distributed congestion control apparatus 100 for VANET using a neural network model based on reinforcement learning according to an embodiment of the present invention.

Referring to FIG. 3, step S201 is a step of calculating at least one of a current channel congestion rate of a wireless channel for a V2X communication service, a current channel occupancy rate of a vehicle, a current moving speed of the vehicle, and a transmission power.

Step S203, by applying at least one of the current channel congestion rate of the wireless channel for the V2X communication service, the channel occupancy rate of the vehicle, the movement speed, and the transmission power to the reinforcement learning-based neural network model, the maximum of the vehicle for the next message transmission. This is the step of determining the channel occupancy ratio and the maximum transmission power.

In one embodiment, a result of a reinforcement learning-based neural network model by applying at least one of a current channel congestion rate of a wireless channel for V2X communication, a current channel occupancy rate of a vehicle, a moving speed, and a transmission power to a reinforcement learning-based neural network model. Can be calculated.

In an embodiment, the maximum channel occupancy ratio and the maximum transmission power of the vehicle may be determined according to the result value of the reinforcement learning-based neural network model.

In an embodiment, according to the maximum occupancy ratio and the maximum transmission power of the vehicle, a degree close to CBR_target, which is a target channel congestion ratio of the channel congestion ratio CBR, may be immediately determined as a compensation value.

In one embodiment, depending on the maximum occupancy ratio and the maximum transmission power of the vehicle, whether or not the required throughput and the time delay (latency) is achieved or the degree of achievement of the current value for the target value is determined as a future compensation value. I can.

In an embodiment, a reinforcement learning-based neural network model may be trained by using a total compensation value obtained by combining a current immediate compensation value and a future compensation value.

In one embodiment, the maximum channel occupancy ratio and the maximum transmission power of the vehicle are determined according to the congestion level of the VANET using a previously learned reinforcement learning-based neural network model, and the difference between the current total compensation value and the previous total compensation value (difference) When is greater than the threshold value, a message for a V2X communication service may be transmitted according to a predetermined maximum channel occupancy rate and maximum transmission power of the vehicle.

That is, according to the present invention, by using a reinforcement learning-based neural network model, each vehicle individually determines the maximum channel occupancy ratio CR _limit and the maximum transmission power P _limit that each vehicle can use according to the congestion level of the VANET. The channel congestion ratio can be close to the target value, and the target performance can be achieved in the transmission rate and delay time of the vehicle in the long term.

4 is a diagram showing a functional configuration 300 of the control unit 101 of the distributed congestion control apparatus 100 according to the present invention according to an embodiment of the present invention.

Referring to FIG. 4, the functional configuration 300 of the control unit 101 of the distributed congestion control device 100 is a state input unit 310, a value network 320, an action decision unit 330, and a reward grant unit 340. It may include.

In the state selection unit 310, the control unit 101 connects with the communication unit 102 and the vehicle network interface unit 104, the channel congestion ratio CBR of the V2X communication wireless channel, the current channel occupancy ratio CR of the vehicle, the movement speed v, The transmission power P is calculated and input to the value network 320.

In the value network 320, the control unit 101 inputs the channel congestion ratio CBR of the V2X communication wireless channel, the current channel occupancy ratio CR of the vehicle, the movement speed v, and the transmission power P to be applied to the reinforcement learning-based neural network model. I can.

For example, a neural network model based on reinforcement learning may include a deep Q-network (DQN). DQN may refer to an algorithm that performs reinforcement learning in a wider state space by adding artificial neural network technology to Q-learning technology.

In the conventional Q-learning technology, when the state space is limited, as shown in FIG. 1, the Q value representing the value of each state can be stored to effectively learn, but not only the CBR of the wireless channel, but also the CR, movement speed, and transmission of the vehicle. If the state space increases, such as power, it is not efficient because it has to store the Q values for a very large number of combinations. Here, the Q value may mean the result of the neural network model.

On the other hand, the DQN according to the present invention solves the problem by approximating a function that determines the Q value with an artificial neural network without storing individual Q values. Q _t can be extracted as the resulting value.

In the behavior determination unit 330, the control unit 101 determines _{the maximum channel occupancy ratio CR limit} and the maximum transmission power P _limit that can be used by each vehicle according to _{the result value Q t of the value network 320 approximated by the artificial neural network.} You can determine the action a _t to do.

In the reward granting unit 340, the controller 101 may observe two rewards (eg, immediate reward and future reward) received from the VANET environment for the _{action a t. Action a t} may or may not be performed according to the increase or decrease of the reward value. The reward value may be stored in the storage unit 103 together with _{the action a t for learning.}

Here, the immediate compensation is an immediate compensation generated for an action determined by the action determination unit 330, and the future compensation may mean a compensation for a future environment caused by the action.

In one embodiment, the action determination process 330 may be expressed as Equation 3 below.

는 감가율(discount factor)로 0에서 1사이의 값을 가지며 0에 가까우면 현재, 1에 가까울수록 미래에 대한 보상을 나타낼 수 있다.

는 학습율로 0에서 1사이의 값을 가지며 Q 값의 학습율을 결정한다. here,

Is a discount factor and has a value between 0 and 1, and the closer to 0 represents the present, and the closer to 1 represents the reward for the future.

Is the learning rate and has a value between 0 and 1 and determines the learning rate of the Q value.

In the compensation granting unit 340, the channel congestion ratio CBR of the radio channel for V2X communication may be considered as an immediate compensation value for the action determined by the action determiner 330. _{That is, the channel congestion ratio based on the maximum channel occupancy ratio CR limit} and the maximum transmission power P _limit determined by the behavior determination unit 330 that each vehicle can use is immediately compensated for the degree close to CBR_target, the target channel congestion ratio. Can be given by value.

In addition, in the reward granting unit 340, the degree of QoS achievement of the V2X service used by the own vehicle may be considered as a future compensation value for the behavior determined by the action determination unit 330. That is, whether or not a throughput and a time delay required for success of the V2X service used by the own vehicle are achieved, or a degree of achievement of a current value for a target value may be given as a compensation value.

The transmission rate may be measured as a packet delivery rate (PDR), which is a transmission rate for a packet that has been successfully received. In addition, the time delay required for packet transmission may be measured as an IPG (Inter Packet Gap), which is a time interval between successfully received packets. Since the PDR and IPG are performance evaluation indicators that cannot be measured instantaneously in each vehicle, they can be used as future compensation values.

In one embodiment, in the value network 320, the reinforcement learning-based neural network model may be trained according to the procedure of training a basic DQN of FIG. 5 to be described later. The control unit 101 of the distributed congestion control apparatus 100 may learn the reinforcement learning-based neural network model at each periodic training period using actions and rewards stored in the storage unit 103 and update the DQN network.

In addition, the controller 101 may perform learning while performing distributed congestion control while the vehicle is running in an actual environment. Additionally, the control unit 101 may use certain virtual environment simulation data or data collected in an actual environment in order to pre-learn various VANET environments.

That is, the control unit 101 according to the present invention performs a series of actions (determining the channel occupancy ratio and transmission power) according to the state of the environment (the congestion level of VANET) using the reinforcement learning-based neural network model. It aims to achieve the target value in the instantaneous channel congestion rate, and achieves the target performance in the transmission rate and delay time in the long term.

5 is a diagram illustrating a method of operating the controller 101 for learning a neural network model based on reinforcement learning according to an embodiment of the present invention.

Referring to FIG. 5, step S401 is _{a step of determining a current state s t} (CBR, CR, v, P) in the state input unit 310. That is, the channel congestion ratio CBR of the vehicle, the channel occupancy ratio CR, the moving speed of the vehicle, and the transmission power can be calculated and determined.

Step S403 is a step of determining _{the result value Q t} by using the reinforcement learning-based neural network model in the value network 320. For example, the Q _t value can be determined using DQN.

Step S405 is a step of determining an action a _t (CR _limit , P _limit ) of determining the maximum channel occupancy ratio CR _limit and the maximum transmission power P _{limit according to the Q t value in the action determination unit 330.} That is, the current maximum channel occupancy ratio CR _limit and the maximum transmission power P _limit can be determined.

Step S407 is a step of determining the _{reward R t} according to the action a _t (CR _limit , P _limit ) in the reward granting unit 340.

In step S409, it is determined whether the learning process of the reinforcement learning-based neural network model has ended.

In step S411, when the training of the reinforcement learning-based neural network model is not finished, the current state s _t , the behavior, a _t , and the corresponding compensation R _t are stored in the storage unit 103.

t+1), S401 단계로 진행하며, 이 과정을 학습이 종료될 때까지 반복하는 단계이다. Step S413 is for the next learning (t

It proceeds to steps t+1) and S401, and repeats this process until the learning is completed.

In this way, the distributed congestion control apparatus 100 may provide a DQN for determining _{the maximum channel occupancy ratio CR limit} and the maximum transmission power P _{limit of each vehicle through the distributed congestion control.}

The distributed congestion control apparatus 100 may perform learning while performing distributed congestion control in a real environment, and may also learn using constant sample data generated through simulation of a real environment or a virtual environment in order to learn in advance. .

FIG. 6 is a diagram showing a detailed distributed congestion control operation method of a VANET distributed congestion control apparatus 100 using a reinforcement learning-based neural network model previously learned through a learning process as shown in FIG. 5 according to an embodiment of the present invention. to be.

Referring to FIG. 6, step S501 is _{a step of determining a current state s t} (CBR, CR, v, P) in the state input unit 310. That is, the channel congestion ratio CBR of the vehicle, the channel occupancy ratio CR, the moving speed of the vehicle, and the transmission power can be determined.

_{In step S503, the result value Q t} is determined using the reinforcement learning-based neural network model previously learned in the value network 320. _{For example, the value of Q t} may be determined by using the DQN on which the learning has been completed.

Step S505 is a step of determining an action a _t (CR _limit , P _limit ) for determining the maximum channel occupancy ratio CR _limit and the maximum transmission power P _{limit according to the Q t value in the action determination unit 330.} That is, the current maximum channel occupancy ratio CR _limit and the maximum transmission power P _limit can be determined.

Step S507 is a step of observing the _{reward R t} according to the action a _t (CR _limit , P _limit ) in the reward granting unit 340.

보다 큰지 여부를 판단하는 단계이다. 일 실시예에서, V2X 통신 무선채널 혼잡도 CBR와 목표 CBR인 CBR_target의 차이, 즉

이 감소하는지 여부를 결정할 수 있다. 이를 통해, 분산혼잡제어에 따라 CBR이 지속적으로 CBR_target으로 근접하고 있는지를 판단할 수 있다.In step S509, the difference between _{the current reward R t} and the previous reward R _t-1 according to the current action is a threshold value.

It is a step to determine whether it is greater than or not. In one embodiment, the difference between the V2X communication radio channel congestion CBR and the target CBR, CBR_target, that is,

You can decide whether it decreases or not. Through this, it can be determined whether the CBR is continuously approaching the CBR_target according to the distributed congestion control.

In step S511, when the difference between the current compensation and the previous compensation is greater than the threshold, changing the maximum channel occupancy ratio CR _limit and the maximum transmission power P _{limit of each vehicle according to the action a t} (CR _limit , P _{limit ).} to be.

t+1), 이 과정을 분산혼잡제어가 종료될 때까지 반복한다. Step S513 proceeds to the next step (t

t+1), this process is repeated until the distributed congestion control is finished.

In step S515, when the difference between the current compensation and the previous compensation is not greater than a threshold value, it is determined whether the distributed congestion control (DCC) has ended.

t+1), 이 과정을 분산혼잡제어가 종료될 때까지 반복한다. Step S517, if the distributed congestion control is not finished, proceeds to the next step (t

t+1), this process is repeated until the distributed congestion control is finished.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art will be able to make various changes and modifications without departing from the essential characteristics of the present invention.

Accordingly, the embodiments disclosed in the present specification are not intended to limit the technical idea of the present invention, but are intended to be described, and the scope of the present invention is not limited by these embodiments.

The scope of protection of the present invention should be interpreted by the claims, and all technical ideas within the scope equivalent thereto should be understood as being included in the scope of the present invention.

100: distributed mixing control device
101: control unit
102: communication department
103: storage unit
104: vehicle network interface unit
105: V2X receiver
106: V2X transmitter
300: control unit
310: status input unit
320: value network
330: action decision section
340: reward granting unit

Claims (12)

Calculating at least one of a channel busy ratio (CBR) of a wireless channel for a V2X communication service, a channel occupancy ratio (CR) of a vehicle, a movement speed, and a transmission power;
Calculating a result value of the reinforcement learning-based neural network model by applying at least one of a channel congestion ratio of the radio channel, a channel occupancy ratio of the vehicle, a moving speed, and a transmission power to a reinforcement learning-based neural network model;
Determining a maximum channel occupancy ratio and a maximum transmission power of the vehicle according to the result of the reinforcement learning-based neural network model;
Immediately determining a degree that the channel congestion ratio is close to a target target channel congestion ratio as a compensation value according to the maximum occupancy ratio and maximum transmission power of the vehicle; And
Determining a degree of achievement of a target value of a throughput and a time delay of the radio channel as a future compensation value;
Containing,
Distributed congestion control method of VANET using reinforcement learning-based neural network model.
delete delete delete The method of claim 1,
After the step of calculating the at least one,
Training the reinforcement learning-based neural network model by using the total current compensation value obtained by combining the immediate compensation value and the future compensation value;
Further comprising,
Distributed congestion control method of VANET using reinforcement learning-based neural network model.
The method of claim 5,
After the step of calculating the at least one,
Determining whether a difference between the current total compensation value and the previous total compensation value is greater than a threshold value; And
Transmitting a message for the V2X communication service according to a maximum channel occupancy ratio and a maximum transmission power of the vehicle when the difference between the current total compensation value and the previous total compensation value is greater than a threshold value;
Further comprising,
Distributed congestion control method of VANET using reinforcement learning-based neural network model.
Calculate at least one of a channel busy ratio (CBR) of a wireless channel for a V2X communication service, a channel occupancy ratio (CR) of a vehicle, a moving speed, and a transmission power,
Applying at least one of a channel congestion ratio of the radio channel, a channel occupancy ratio of the vehicle, a moving speed, and a transmission power to a reinforcement learning-based neural network model to calculate a result value of the reinforcement learning-based neural network model,
Determine the maximum channel occupancy ratio and the maximum transmission power of the vehicle according to the result of the reinforcement learning-based neural network model,
In accordance with the maximum occupancy ratio and the maximum transmission power of the vehicle, the degree to which the channel congestion ratio approaches a target target channel congestion ratio is immediately determined as a compensation value,
A control unit determining a degree of achievement of a target value of a throughput and a time delay of the radio channel as a future compensation value;
Containing,
Distributed congestion control device of VANET using reinforcement learning-based neural network model.
delete delete delete The method of claim 7,
The control unit,
Training the reinforcement learning-based neural network model using the current total compensation value combined with the immediate compensation value and the future compensation value,
Distributed congestion control device of VANET using reinforcement learning-based neural network model.
The method of claim 11,
The control unit,
Determine whether a difference between the current total compensation value and the previous total compensation value is greater than a threshold value,
A communication unit for transmitting a message for the V2X communication service according to a maximum channel occupancy ratio and a maximum transmission power of the vehicle when a difference between the current total compensation value and the previous total compensation value is greater than a threshold value;
Further comprising,
Distributed congestion control device of VANET using reinforcement learning-based neural network model.

Images