KR20220087717A - Method and device for controlling resource reallocation in vehicle network - Google Patents

Abstract

The disclosed technology relates to a method and apparatus for controlling resource reallocation of a vehicle network, the method comprising: receiving, by a device, location information, allocated resource information, and a reward for a plurality of vehicles through a vehicle network; The device calculates a first probability for resource reallocation of the vehicle based on the location information and the reward, and inputs the resource information and reward to the reinforcement learning model to obtain a second probability for the resource reallocation calculating; and determining, by the device, a policy of the vehicle network based on the calculation results for the first probability and the second probability.

Description

Method and device for controlling resource reallocation of vehicle network {METHOD AND DEVICE FOR CONTROLLING RESOURCE REALLOCATION IN VEHICLE NETWORK}

The disclosed technology relates to a method and apparatus for controlling resource reallocation in a vehicle network.

Representative standards of vehicle networks using the conventional sub 6 GHz band include C-V2X and IEEE 802.11p. The C-V2X standard includes C-V2X mode 4 based on autonomous scheduling and C-V2X mode 3 that controls scheduling in the network.

In C-V2X mode 4, when each vehicle transmits a certain number of packets, based on sensing information, it is determined whether to use the existing resource block as it is or to update it with another resource block to communicate. . C-V2X mode 3 is a technique for directly allocating a resource block to each vehicle based on information sensed by the base station.

On the other hand, as the autonomous driving level of vehicles increases and inter-vehicle applications diversify, the need for MAC research that can meet higher data transmission rates is increasing. Accordingly, research on directional transmission using Abobe-6GHz, that is, millimeter wave, is being intensively conducted.

On the other hand, in the case of C-V2X mode 4, there is a problem in that the efficiency of resource selection is lower than that of the method that does not utilize the existing infrastructure and uses the infrastructure, but there is an advantage that there is no additional latency caused by going through the infrastructure. . On the other hand, in the case of C-V2X mode 3, there is a disadvantage in that an additional delay time occurs, but a data transmission gain occurs through efficient resource selection.

As the band recently moved from the sub 6GHz band to the Abobe-6GHz band, in the case of C-V2X mode 3, the data rate drops when all vehicles use directional antennas because resources are allocated based on the information sensed by the base station. can occur And in the case of C-V2X mode 4, even if millimeter wave is used, there is no significant difference from the previous sub 6GHz band.

Korean Patent Publication No. 10-2020-0096096

The disclosed technology is to provide a method and apparatus for controlling resource reallocation of a vehicle network.

A first aspect of the disclosed technology to achieve the above technical problem is the device receiving location information, allocated resource information, and a reward for a plurality of vehicles through a vehicle network, the device receiving the location information and the reward based on the calculating a first probability for the resource reallocation of the vehicle, inputting the resource information and a reward to a reinforcement learning model, and calculating a second probability for the resource reallocation, and the device performs the first An object of the present invention is to provide a method for controlling resource reallocation of a vehicle network, comprising determining a policy of the vehicle network based on a probability and a calculation result for the second probability.

A second aspect of the disclosed technology to achieve the above technical task is an antenna for receiving location information, allocated resource information and rewards for a plurality of vehicles, and a plurality of reinforcement learning to determine a policy for the plurality of vehicles A memory storing a model and inputting the location information and a reward to a first reinforcement learning model among the plurality of reinforcement learning models calculates a first probability for resource reallocation of the plurality of vehicles, and the plurality of reinforcement learning models of the second reinforcement learning model, calculating a second probability for the resource reallocation by inputting the resource information and the reward, and determining a vehicle network policy based on the calculation results for the first probability and the second probability An object of the present invention is to provide an apparatus for controlling resource reallocation of a vehicle network including a processor.

Embodiments of the disclosed technology may have effects including the following advantages. However, since it does not mean that the embodiments of the disclosed technology should include all of them, it should not be understood that the scope of the disclosed technology is limited thereby.

According to an embodiment of the disclosed technology, a method and apparatus for controlling resource reallocation of a vehicle network calculates the reallocation possibility and probability of a resource block allocated to a vehicle through a plurality of reinforcement learning models to determine a vehicle network policy with high reliability has the effect of

In addition, there is an effect of maintaining a high data transmission rate in consideration of the communication environment of the vehicle network.

In addition, there is an effect of reducing the amount of data transmission by using the distributed and stored crit model.

1 is a diagram illustrating a resource reallocation control process of a vehicle network according to an embodiment of the disclosed technology.
2 is a flowchart of a method for controlling resource reallocation of a vehicle network according to an embodiment of the disclosed technology.
3 is a block diagram of an apparatus for controlling resource reallocation of a vehicle network according to an embodiment of the disclosed technology.
4 is a diagram illustrating using a reinforcement learning model of a device according to an embodiment of the disclosed technology.
5 is a diagram illustrating the use of a reinforcement learning model distributed over a plurality of vehicles according to an embodiment of the disclosed technology.
6 is a diagram illustrating determination of a resource block candidate group.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention.

Terms such as first, second, A, and B may be used to describe various components, but the components are not limited by the above terms, and only for the purpose of distinguishing one component from other components. is used only as For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In terms of terms used herein, the singular expression is to be understood as including the plural expression unless the context clearly dictates otherwise. And terms such as "comprising" mean that the specified feature, number, step, operation, component, part, or a combination thereof exists, but one or more other features or number, step operation component, part It should be understood as not excluding the possibility of the presence or addition of or combinations thereof.

Prior to a detailed description of the drawings, it is intended to clarify that the classification of the constituent parts in the present specification is merely a division according to the main function each constituent unit is responsible for. That is, two or more components to be described below may be combined into one component, or one component may be divided into two or more for each more subdivided function.

In addition, each of the constituent units to be described below may additionally perform some or all of the functions of other constituent units in addition to the main function it is responsible for. Of course, it may be carried out by being dedicated to it. Accordingly, the existence or non-existence of each component described through the present specification should be interpreted functionally.

1 is a diagram illustrating a resource reallocation control process of a vehicle network according to an embodiment of the disclosed technology. Referring to FIG. 1 , a plurality of vehicles and a base station are included as nodes in a vehicle network. For example, vehicle A, vehicle B, and vehicle C may transmit/receive sensing data to each other through a network. In addition, sensing data can be transmitted not only to the vehicle but also to the base station. The basic network structure may be in the form of mode 3 of cellular-V2X (C-V2X). However, the internally processed data framework can be based on C-V2X mode 4. That is, C-V2X mode 3 and mode 4 may be mixed. The base station is equipped with a device for reallocating resources to each vehicle. The device 110 stores a reinforcement learning model for calculating the resource reallocation probability.

On the other hand, the device for controlling the resource reallocation may use a roadside base station (Road Side Unit, RSU) installed in the infrastructure on the road as well as the base station on the ground. The device receives location information, allocated resource information, and rewards for a plurality of vehicles through a vehicle network. The vehicle location information refers to a value sensed by an antenna or sensor mounted in the vehicle. For example, when a plurality of vehicles pass through a specific section, a sensing result collected within a detection radius of a sensor mounted on each vehicle body may be used as location information. And, the resource information means information on a resource block currently allocated to the vehicle and information on the size of the resource block. And the reward means a reward for the environment (Environment) of the vehicle network. That is, location information about a plurality of vehicles, allocated resource information, and rewards may be used as parameters for determining a policy of a vehicle network.

Meanwhile, the device may calculate a resource reallocation probability for each vehicle by using the received network parameter. Here, the resource for the vehicle means a resource block for communication. The device may calculate whether to maintain the resource block previously assigned to the plurality of vehicles or to allocate a new resource block. In this case, rather than simply calculating the probability for resource reallocation using the reinforcement learning model, the result of estimating the resource reallocation possibility based on vehicle location information can also be used. As an embodiment, a first probability for resource reallocation of a vehicle may be calculated based on location information and a reward of the vehicle. Then, the second probability for resource reallocation may be calculated by inputting resource information and a reward to the reinforcement learning model. In addition, a policy of the vehicle network may be determined based on the calculation result of the first probability and the second probability.

Meanwhile, the device includes a reinforcement learning model for calculating the resource reallocation probability. Reinforcement learning models can use actor-critic networks. The device may use the Critik network possessed by the device according to the communication state of the vehicle network, or may use the Critik network stored in the vehicle. That is, the Critik network can be distributed and stored in vehicles and devices. For example, if the communication environment of the vehicle network is good, the crit network stored by the device may be used, and if the communication environment of the vehicle network is poor, the result value of the clinic network mounted in the vehicle may be received. In the former case, it may be a model of a centralized type in the device, and in the latter case, it may be a model of a type of distributed (Decentralized) type in vehicles and devices. Here, the criterion for determining whether the communication environment of the vehicle network is good or bad may be determined according to the possibility that data interference may occur because a large number of vehicles are clustered around a specific vehicle. For example, if the interference in the communication environment is greater than or equal to the threshold value, the Critik network stored in the plurality of vehicles may be used, and if the interference in the communication environment is less than the threshold value, the Critik network stored in the device may be used. Therefore, it is possible to maintain the accuracy according to the policy decision by adjusting the data rate according to the communication environment.

Meanwhile, as described above, a plurality of vehicles and devices use C-V2X (Cellular Vehicle-to-Everything) based communication. A plurality of vehicles may transmit location information, allocated resource information, and rewards as network parameters to the device through the uplink of C-V2X. And, the device may transmit the policy through the downlink of C-V2X.

On the other hand, if the device determines that the reallocation probability of the current resource block is high according to the result of calculating the resource reallocation probability, the device maintains the currently allocated state, and if it is determined that the reallocation probability is low, a random among a plurality of resource block candidates The one determined to be a new resource block for the vehicle may be determined. In the former case, it is predicted that the data rate will not decrease, and in this case, the current state may be maintained. Of course, when the next network parameter is received, the above-described calculation process is performed again. At this time, if the resource reallocation probability decreases, the resource block may be newly updated.

Meanwhile, when the device calculates that the resource reallocation probability of the vehicle is low, the device may determine a new resource block based on the RSSI value measured before 1000 ms within the sensing window. For example, an average of the RSSI values may be calculated for each resource block, and a value of the lower 20% may be determined as a plurality of resource block candidate groups. Among them, one randomly determined resource block is determined as a resource block to be re-allocated to the vehicle.

On the other hand, as described above, the device determines a policy based on the result of calculating the first probability and the second probability for the location resource reallocation of the vehicle. As an embodiment, an average of values obtained by multiplying the result of calculating the first probability and the result of calculating the second probability by a weight may be determined as a policy of the vehicle network. In this case, when the reinforcement learning model is not sufficiently trained in advance, the reliability of the result value can be increased by adjusting the weight multiplied by the calculation result. For example, at the beginning of learning, the weight for the result of calculating the first probability may be set to a high value, and at the end of learning, the weight for the result of calculating the second probability may be set to a high value. Therefore, it is possible to increase the network reliability by adjusting the data transmission amount in consideration of the communication state of the vehicle network, and to compensate for the unstable result value of reinforcement learning by adjusting the weight.

2 is a flowchart of a method for controlling resource reallocation of a vehicle network according to an embodiment of the disclosed technology. Referring to FIG. 2 , the method 200 for controlling resource reallocation of a vehicle network includes receiving a vehicle network parameter ( 210 ), calculating a first probability and a second probability for resource reallocation of the vehicle, respectively ( 220 ) and determining (230) a policy of the vehicle network. The resource reallocation control method 200 of the vehicle network may be sequentially performed through a device.

In operation 210 , the device receives network parameters for a plurality of vehicles through a vehicle network. The network parameters include vehicle location information, resource information allocated to the vehicle, and rewards.

In step 220, the device calculates a first probability for resource reallocation of the vehicle based on the vehicle location information and the reward included in the network parameter. Then, a second probability for resource reallocation is calculated by inputting resource information and a reward to the reinforcement learning model. Calculating the first probability is estimating whether resources will be reallocated in consideration of the location where the vehicle will move in the future based on the sensing data of the vehicle, and calculating the second probability is whether to use the currently allocated resource block as it is or to use another resource block. It means calculating the probability of assigning. The device may calculate the second probability using the reinforcement learning model. Of course, the reinforcement learning model can also be used in the calculation of the first probability.

In step 230, the device determines a policy of the vehicle network based on the calculation results of each of the first probability and the second probability. The policy may be determined as a value obtained by calculating a weighted average of the calculation results of the first probability and the second probability. For example, the policy may be determined by multiplying the calculation result of the first probability by a weight of a specific value, multiplying the calculation result of the second probability by different weights, and taking the average of the two result values. The weight of the weight may vary depending on the learning degree of the reinforcement learning model. For example, at the beginning of learning, the weight for the calculation result of the first probability may be set to a high value, and the weight for the calculation result of the second probability may be set to be high in the late learning period.

3 is a block diagram of an apparatus for controlling resource reallocation of a vehicle network according to an embodiment of the disclosed technology. Referring to FIG. 3 , an apparatus 300 for controlling resource reallocation of a vehicle network includes an antenna 310 , a memory 320 , and a processor 330 .

The antenna 310 receives location information for a plurality of vehicles, allocated resource information, and a reward. The antenna is a device capable of communicating with a directional antenna mounted on a vehicle. The antenna 310 may use an antenna mounted on a base station or a frequency sensor.

The memory 320 stores a plurality of reinforcement learning models to determine a policy for a plurality of vehicles. The memory 320 is implemented as a device such as a hard disk having a predetermined storage space. The memory 320 has a capacity to store the reinforcement learning model and network parameters.

The processor 330 calculates a first probability for resource reallocation of a plurality of vehicles by inputting location information and a reward to the first reinforcement learning model among the plurality of reinforcement learning models. Then, by inputting resource information and a reward to the second reinforcement learning model among the plurality of reinforcement learning models, a second probability for resource reallocation is calculated. For the calculation of the first probability and the calculation of the second probability, different reinforcement learning models may be used. For example, a model such as Deep Q-network (DQN) may be used to calculate the first probability. Of course, like the second reinforcement learning model, it is also possible to use an actor-critic network. When the result value is output through the two reinforcement learning models, the processor 330 calculates each weighted average and determines it as a policy of the network.

Meanwhile, the above-described resource reallocation control apparatus 300 may be implemented as a program (or application) including an executable algorithm that can be executed in a device such as a computer. The program may be provided by being stored in a temporary or non-transitory computer readable medium.

The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, and the like, and can be read by a device. Specifically, the various applications or programs described above are CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), EPROM (Erasable PROM, EPROM) Alternatively, it may be provided by being stored in a non-transitory readable medium such as an EEPROM (Electrically EPROM) or flash memory.

Temporarily readable media include Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (Enhanced) SDRAM, ESDRAM), Synchronous DRAM (Synclink DRAM, SLDRAM) and Direct Rambus RAM (DRRAM)

4 is a diagram illustrating using a reinforcement learning model of a device according to an embodiment of the disclosed technology. Referring to FIG. 4 , a vehicle network including a plurality of vehicles and a roadside base station may be formed, and the data framework structure of the vehicle network may be a C-V2X mode 4 structure. Network parameters transmitted through the uplink may vary according to the communication state of the vehicle network. In the case of FIG. 4 , since the communication state of the vehicle network is in a good state, location information, resource information, and a reward may be transmitted as network parameters necessary for the A2C algorithm operation through the uplink.

5 is a diagram illustrating the use of a reinforcement learning model distributed over a plurality of vehicles according to an embodiment of the disclosed technology. In the case of FIG. 4, it is assumed that the communication state of the vehicle network is good, but in the case of FIG. 5, it is assumed that the communication state is uneven. In this case, it is possible to distribute the critique network to the vehicle and receive the result value output through the critique network of the vehicle as a network parameter. That is, depending on the communication state, the structure of the reinforcement learning model may be in a form that is concentrated on the device or a form that is distributed between the device and the vehicle.

In addition, in FIG. 5 , on the premise that the device estimation method is not used because each vehicle has a critical model, the value and resource information (Rsc Info) output from the vehicle's critical network instead of exchanging location information or rewards are not provided. ) by uploading the value, you can see the effect of reducing the communication amount. Such a distributed A2C technique can take different amounts of information to be uploaded depending on the communication conditions such as the channel environment for each vehicle. Of course, the estimation method may be applied only to vehicles with good communication environments.

Meanwhile, an equation including both techniques can be summarized as shown in Equation 1 below.

[Equation 1]

For example, it is possible to aim for relatively high performance even in the early stages of running by changing the resource blocks of vehicles that are likely to cause or receive interference because there are many nearby vehicles.

6 is a diagram illustrating determination of a resource block candidate group. Referring to FIG. 6 , the basic data framework of the vehicle network is based on the C-V2X mode 4 framework. When each vehicle transmits a certain number of packets during sensing and data transmission, it is introduced into the selection window and is applied to the existing communication according to the resource reallocation possibility (1-P, P=[0,0.8]). It is decided whether to use the used resource block as it is or to randomly reallocate it from among new resource block candidates. At this time, the received signal strength indication (RSSI) values measured before 1000 ms from the sensing window are averaged for each resource block, and a lower 20% value among them is designated as a new resource block candidate group. One randomly determined among them is determined as a new resource block.

Although the method and apparatus for controlling resource reallocation of a vehicle network according to an embodiment of the disclosed technology have been described with reference to the embodiment shown in the drawings for better understanding, this is merely exemplary, and those of ordinary skill in the art It will be understood that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the disclosed technology should be defined by the appended claims.

Claims (16)

receiving, by a device, location information, allocated resource information, and a reward for a plurality of vehicles through a vehicle network;
The device calculates a first probability for resource reallocation of the vehicle based on the location information and the reward, and inputs the resource information and reward to the reinforcement learning model to obtain a second probability for the resource reallocation calculating; and
and determining, by the device, a policy of the vehicle network based on the calculation results for the first probability and the second probability. The method of claim 1,
The reinforcement learning model is a resource reallocation control method of a vehicle network including an actor network and a crit network. The method of claim 1,
The reinforcement learning model includes an actor network, and a resource reallocation control method of a vehicle network operated by receiving a value output from the vehicle's critical network. 3. The method of claim 2,
The crit network is stored in the plurality of vehicles and the device, respectively, and if the interference of the communication environment is greater than or equal to a threshold value, the crit network stored in the plurality of vehicles is used; A method for controlling resource reallocation of a vehicle network using a stored crit network. The method of claim 1,
The plurality of vehicles and the device use C-V2X (Cellular Vehicle-to-Everything) based communication,
The plurality of vehicles transmit the location information, allocated resource information, and reward as network parameters to the device through the uplink of the C-V2X,
The device transmits the policy through the downlink of the C-V2X resource reallocation control method of the vehicle network. The method of claim 1,
A resource reallocation control method of a vehicle network for transmitting a sensing result collected within a detection radius of a sensor mounted on a vehicle body to the device as the location information when the plurality of vehicles pass through a specific section. The method of claim 1,
The device is a resource reallocation control method of a vehicle network, characterized in that the RSU (Road Side Unit) installed in a base station on the ground or infrastructure on the road. The method of claim 1, wherein calculating the reassignment probability comprises:
When the device calculates that the reallocation probability of the current resource block is high, it maintains the currently allocated state, and when the device calculates that the reallocation probability is low, a randomly determined one of a plurality of resource block candidate groups is selected as a new resource for the vehicle. A method of controlling resource reallocation of a vehicle network determined by blocks. 9. The method of claim 8,
The device calculates an average for each resource block of RSSI values measured before 1000 ms within a sensing window, and determines a value of the lower 20% as the plurality of resource block candidate groups. The method of claim 1,
The device determines the policy by calculating a weighted average of the estimation result and the calculation result,
A resource reallocation control method of a vehicle network in which a weight for the estimation result is set to a high value at the beginning of learning and a weight for the calculation result is set to a high return in the late learning period. An antenna for receiving location information, allocated resource information, and rewards for a plurality of vehicles;
a memory for storing a plurality of reinforcement learning models to determine a policy for the plurality of vehicles; and
Input the location information and the reward to a first reinforcement learning model among the plurality of reinforcement learning models to calculate a first probability for resource reallocation of the plurality of vehicles, and a second reinforcement learning model among the plurality of reinforcement learning models A processor configured to calculate a second probability for the resource reallocation by inputting the resource information and a reward to the , and determine a policy of a vehicle network based on the calculation results for the first probability and the second probability; Vehicle including a Resource reallocation control device in the network. 12. The method of claim 11,
The second reinforcement learning model is an apparatus for controlling resource reallocation of a vehicle network including an actor network and a critique network. 12. The method of claim 11,
The second reinforcement learning model includes an actor network, and an apparatus for controlling resource reallocation of a vehicle network that operates by receiving a value output from a critical network of the plurality of vehicles. 13. The method of claim 12,
The crit network is stored in the plurality of vehicles and the memory, respectively, and when the interference of the communication environment is greater than or equal to a threshold value, the crit network stored in the plurality of vehicles is used; A resource reallocation control device of the vehicle network using the stored crit network. 12. The method of claim 11,
The processor maintains the currently allocated resource block when it is calculated that the resource reallocation probability is high, and selects a randomly determined one of a plurality of resource block candidate groups when it is calculated that the resource reallocation probability is low for the plurality of vehicles. Resource reallocation control device of vehicle network determined by resource block. 16. The method of claim 15,
The processor calculates an average for each resource block of RSSI values measured before 1000 ms within a sensing window, and determines a value of a lower 20% of the RSSI values as the plurality of resource block candidate groups.

Images