KR20220112890A - Device and method for allocating resource in vehicle to everything communication based on non-orhthogonal multiple access - Google Patents

Abstract

Disclosed are a device and a method for allocating resources in vehicle communication based on non-orthogonal multiple access. In accordance with an embodiment of the present invention, the device for allocating resources includes: a reinforcement learning part performing reinforcement learning for modifying a policy such that the total data throughput between at least two vehicles and at least one base station can rise, while changing channels allocated to a plurality of partitions and positions of the vehicles; and a resource allocation part generating a resource allocation table in which one of the channels is allocated to each of the partitions according to a policy determined by the reinforcement learning part, and transmitting the resource allocation table to the vehicles and the base station. Therefore, the present invention is capable of efficiently allocating resources without causing an excessive delay.

Description

DEVICE AND METHOD FOR ALLOCATING RESOURCE IN VEHICLE TO EVERYTHING COMMUNICATION BASED ON NON-ORHTHOGONAL MULTIPLE ACCESS

The present invention relates to an apparatus and method for allocating resources in non-orthogonal multiple access based vehicular communication, and in particular, in non-orthogonal multiple access based vehicular communication that can efficiently allocate resources without excessive delay by applying deep reinforcement learning. It relates to an apparatus and method for allocating resources.

Technologies such as Vehicle to Everything (V2X) and Non-Orthogonal Multiple Access (NOMA) are currently being commercialized.

Vehicle communication (V2X) is a technology in which a moving vehicle communicates with a base station or other vehicle. Attenuation, change and It is important to perform stable communication in response to interference.

The non-orthogonal multiple access method (NOMA) is a technology for improving resource efficiency by transmitting signals to two or more receiving terminals simultaneously through the same channel (frequency). It cancels other signals through successful interference cancellation (SIC) and decodes only the own signal.

In recent years, communication traffic in vehicle communication is rapidly increasing due to the development of autonomous driving and in-vehicle infotainment. A problem is occurring. Accordingly, in order to improve data throughput, a technique for applying a non-orthogonal multiple access method to vehicle communication is being considered.

In this case, a wireless resource allocation technique for allocating channels to communication devices, in particular, allocating the same channel in a non-orthogonal multiple access method and determining the strength of transmission power is very important.

When radio resource allocation is properly performed, interference between devices in the system can be minimized and the data throughput of the entire system can be maximized by using the channel with the highest transmission efficiency.

On the other hand, Deep Reinforcement Learning (DRL) is a type of machine learning in the field of artificial intelligence. It is generally optimized by applying an artificial neural network to the existing reinforcement learning algorithm. It is a technique suitable for solving problems with non-convex characteristics that are difficult to derive algorithmic or mathematical approaches.

In deep reinforcement learning, concepts such as Agent, Environment, State, Action, and Reward are used. The agent learns to find out the current state by observing the environment, and learns to predict how much reward can be obtained by taking an action in the current state. Learn (Policy). If you use the deep reinforcement learning model that has been trained, you can perform an action that returns the optimal reward in the current state without additional learning according to your own policy. This can be advantageous when the user in the system can share the same model in a distributed manner, so the delay in the process of agreeing actions can also be reduced.

Since vehicle communication has high variables such as vehicle movement, there are various difficulties in introducing a non-orthogonal multiple access method. In particular, there is a great difficulty in resource allocation. In particular, if the number of users (or vehicles) in the area increases, it may be difficult to calculate the resource allocation problem in a short time, and difficulty may occur in meeting the transmission efficiency condition. Therefore, there is a need for an efficient resource allocation technique that satisfies the required data throughput in the system and does not cause delay even in consideration of various variables of vehicle communication.

An object of the present invention is to provide an apparatus and method for allocating resources in non-orthogonal multiple access method-based vehicle communication that can efficiently allocate resources without excessive delay by applying deep reinforcement learning.

The resource allocating apparatus in non-orthogonal multiple access method-based vehicle communication according to an embodiment of the present invention changes channels allocated to a plurality of partitions and positions of at least two or more vehicles while changing the positions of the vehicles and at least one or more base stations. Create a resource allocation table in which one of the channels is allocated to each of the partitions according to a policy determined by a reinforcement learning unit and a reinforcement learning unit that performs reinforcement learning to modify a policy so that data throughput is increased, and and a resource allocator for transmitting the resource allocation table to the vehicles and the base station.

According to an embodiment, the channels may be divided by at least one of a frequency and a signal strength.

According to an embodiment, the reinforcement learning measures the data throughput as a reward after performing an action to change the channels allocated to the partitions, and modifies the policy so that the total data throughput is increased. Step (a) repeating the learning process n times (n is a natural number greater than or equal to 2) n times (n is a natural number greater than or equal to 2), and step (a) is repeated m times (m is a natural number greater than or equal to 2) while moving the positions of the vehicles according to the driving information set for each vehicle. ) repeating step (b) and (c) repeating step (b) p times (p is a natural number equal to or greater than 2) while changing the initial positions of the vehicles.

According to an embodiment, the step (a) may include allocating the channels to the partitions based on the resource allocation table generated according to the policy, observing communication between the vehicles and the base station to determine the total data throughput and updating the weight of the neural network so that a loss, which is a difference between a result value of a predetermined objective function and the compensation, is minimized.

According to an embodiment, the driving information may include the traveling direction and speed of each of the vehicles.

According to an embodiment, the total data throughput may include a data throughput in an uplink from each of the vehicles to another vehicle or the base station and a data throughput in a downlink from the base station to the vehicles.

, 여기서, N은 상기 채널들의 집합, V는 상기 차량들의 집합, B는 상기 기지국의 집합,

는 n번째 채널을 통한 i번째 차량으로부터 다른 차량으로의 상향 링크에서의 제1 데이터 처리량,

는 상기 n번째 채널을 통한 상기 i번째 차량으로부터 상기 기지국으로의 상향 링크에서의 제2 데이터 처리량,

는 상기 n번째 채널을 통한 j번째 기지국으로부터 제1 차량으로의 하향 링크에서의 제3 데이터 처리량,

는 상기 n번째 채널을 통한 상기 j번째 기지국으로부터 제2 차량으로의 하향 링크에서의 제4 데이터 처리량을 나타낸다.According to an embodiment, the total data throughput is calculated by the following equation,

, where N is the set of channels, V is the set of vehicles, B is the set of base stations,

is the first data throughput in the uplink from the i-th vehicle to another vehicle through the n-th channel,

is a second data throughput in the uplink from the i-th vehicle to the base station through the n-th channel,

is the third data throughput in the downlink from the j-th base station to the first vehicle through the n-th channel,

denotes the fourth data throughput in the downlink from the j-th base station to the second vehicle through the n-th channel.

, 여기서,

는 상기 i번째 차량에 상기 n번째 채널의 할당 여부를 나타내는 값,

는 상기 i번째 차량과 상기 다른 차량 사이의 상기 n번째 채널의 제1 SINR(signal to interference and noise ratio) 값을 나타낸다.According to an embodiment, the first data throughput is calculated according to the following equation,

, here,

is a value indicating whether the n-th channel is allocated to the i-th vehicle,

denotes a first signal to interference and noise ratio (SINR) value of the n-th channel between the i-th vehicle and the other vehicle.

, 여기서,

는 상기 i번째 차량에 상기 n번째 채널의 할당 여부를 나타내는 값,

는 상기 i번째 차량과 상기 기지국 사이의 상기 n번째 채널의 제2 SINR 값을 나타낸다.According to an embodiment, the second data throughput is calculated according to the following equation,

, here,

is a value indicating whether the n-th channel is allocated to the i-th vehicle,

denotes the second SINR value of the n-th channel between the i-th vehicle and the base station.

, 여기서,

는 상기 j번째 기지국에 상기 n번째 채널의 할당 여부를 나타내는 값,

는 상기 n번째 채널을 통한 상기 j번째 기지국으로부터 상기 제1 차량으로의 하향 링크에서의 제3 SINR 값을 나타낸다.The third data throughput is calculated according to the following equation,

, here,

is a value indicating whether the n-th channel is allocated to the j-th base station,

denotes a third SINR value in a downlink from the j-th base station to the first vehicle through the n-th channel.

, 여기서,

는 상기 j번째 기지국에 상기 n번째 채널의 할당 여부를 나타내는 값,

는 상기 n번째 채널을 통한 상기 j번째 기지국으로부터 상기 제2 차량으로의 하향 링크에서의 제4 SINR 값을 나타낸다.According to an embodiment, the fourth data throughput is calculated according to the following equation,

, here,

is a value indicating whether the n-th channel is allocated to the j-th base station,

denotes a fourth SINR value in a downlink from the j-th base station to the second vehicle through the n-th channel.

A resource allocation method in non-orthogonal multiple access scheme-based vehicle communication according to an embodiment of the present invention performs an action of changing channels allocated to a plurality of partitions, and then between at least two or more vehicles and at least one or more base stations Step (a) of measuring the total data throughput as a reward and repeating the learning process of modifying the policy so that the total data throughput is increased n times (n is a natural number greater than or equal to 2); Step (b) repeating step (a) m times (m is a natural number greater than or equal to 2) while moving the locations of the vehicles according to the information, and repeating step (b) p times (p is 2) while changing the initial locations of the vehicles or more) repeating steps (c) and (d) allocating the channels to the partitions according to the policy determined through steps (a) to (c).

According to an embodiment, the step (a) may include allocating the channels to the partitions based on the resource allocation table generated according to the policy, observing communication between the vehicles and the base station to determine the total data throughput and updating the weight of the neural network so that a loss, which is a difference between a result value of a predetermined objective function and the compensation, is minimized.

According to an embodiment, the channels may be divided by at least one of a frequency and a signal strength.

According to an embodiment, the driving information may include the traveling direction and speed of each of the vehicles.

According to an embodiment, the total data throughput may include a data throughput in an uplink from each of the vehicles to another vehicle or the base station and a data throughput in a downlink from the base station to the vehicles.

, 여기서, N은 상기 채널들의 집합, V는 상기 차량들의 집합, B는 상기 기지국의 집합,

는 n번째 채널을 통한 i번째 차량으로부터 다른 차량으로의 상향 링크에서의 제1 데이터 처리량,

는 상기 n번째 채널을 통한 상기 i번째 차량으로부터 상기 기지국으로의 상향 링크에서의 제2 데이터 처리량,

는 상기 n번째 채널을 통한 j번째 기지국으로부터 제1 차량으로의 하향 링크에서의 제3 데이터 처리량,

는 상기 n번째 채널을 통한 상기 j번째 기지국으로부터 제2 차량으로의 하향 링크에서의 제4 데이터 처리량을 나타낸다.According to an embodiment, the total data throughput is calculated by the following equation,

, where N is the set of channels, V is the set of vehicles, B is the set of base stations,

is the first data throughput in the uplink from the i-th vehicle to another vehicle through the n-th channel,

is a second data throughput in the uplink from the i-th vehicle to the base station through the n-th channel,

is the third data throughput in the downlink from the j-th base station to the first vehicle through the n-th channel,

denotes the fourth data throughput in the downlink from the j-th base station to the second vehicle through the n-th channel.

, 여기서,

는 상기 i번째 차량에 상기 n번째 채널의 할당 여부를 나타내는 값,

는 상기 i번째 차량과 상기 다른 차량 사이의 상기 n번째 채널의 제1 SINR(signal to interference and noise ratio) 값을 나타낸다.According to an embodiment, the first data throughput is calculated according to the following equation,

, here,

is a value indicating whether the n-th channel is allocated to the i-th vehicle,

denotes a first signal to interference and noise ratio (SINR) value of the n-th channel between the i-th vehicle and the other vehicle.

, 여기서,

는 상기 i번째 차량에 상기 n번째 채널의 할당 여부를 나타내는 값,

는 상기 i번째 차량과 상기 기지국 사이의 상기 n번째 채널의 제2 SINR 값을 나타낸다.According to an embodiment, the second data throughput is calculated according to the following equation,

, here,

is a value indicating whether the n-th channel is allocated to the i-th vehicle,

denotes the second SINR value of the n-th channel between the i-th vehicle and the base station.

, 여기서,

는 상기 j번째 기지국에 상기 n번째 채널의 할당 여부를 나타내는 값,

는 상기 n번째 채널을 통한 상기 j번째 기지국으로부터 상기 제1 차량으로의 하향 링크에서의 제3 SINR 값을 나타낸다.The third data throughput is calculated according to the following equation,

, here,

is a value indicating whether the n-th channel is allocated to the j-th base station,

denotes a third SINR value in a downlink from the j-th base station to the first vehicle through the n-th channel.

, 여기서,

는 상기 j번째 기지국에 상기 n번째 채널의 할당 여부를 나타내는 값,

는 상기 n번째 채널을 통한 상기 j번째 기지국으로부터 상기 제2 차량으로의 하향 링크에서의 제4 SINR 값을 나타낸다.According to an embodiment, the fourth data throughput is calculated according to the following equation,

, here,

is a value indicating whether the n-th channel is allocated to the j-th base station,

denotes a fourth SINR value in a downlink from the j-th base station to the second vehicle through the n-th channel.

A resource allocation apparatus and method in non-orthogonal multiple access scheme-based vehicle communication according to an embodiment of the present invention can efficiently allocate resources without excessive delay by applying deep reinforcement learning.

In order to more fully understand the drawings recited in the Detailed Description of the Invention, a detailed description of each drawing is provided.
1 is a conceptual diagram illustrating a wireless communication system according to an embodiment of the present invention.
2 is a block diagram illustrating an apparatus for allocating a resource according to an embodiment of the present invention.
3 is a flowchart illustrating a reinforcement learning process of the reinforcement learning unit shown in FIG. 2 .
FIG. 4 is a flowchart illustrating the step S200 shown in FIG. 3 in more detail.
FIG. 5 is a diagram illustrating a virtual system for simulating the reinforcement learning process shown in FIG. 3 .
6 is a graph illustrating a change in loss in an initial stage of learning while reinforcement learning is performed.
7 is a graph illustrating a change in loss at a late learning stage while reinforcement learning is performed.
8 is a graph illustrating a change in a discounted reward while reinforcement learning is performed.

Specific structural or functional descriptions for the embodiments according to the concept of the present invention disclosed in this specification are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention are It may be implemented in various forms and is not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one element from other elements, for example, without departing from the scope of the present invention, a first element may be called a second element, and similarly The second component may also be referred to as the first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Other expressions describing the relationship between elements, such as "between" and "immediately between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers. , it should be understood that it does not preclude the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings.

1 is a conceptual diagram illustrating a wireless communication system according to an embodiment of the present invention.

Referring to FIG. 1 , the wireless communication system 10 includes a base station 100 and vehicles 200 - 1 to 200 - 4 (hereinafter, referred to as 200 when commonly referred to). Although one base station 100 and four vehicles 200-1 to 200-4 are illustrated in FIG. 1, the technical spirit of the present invention is not limited thereto. That is, the wireless communication system 10 may include at least one base station and at least two or more vehicles.

The base station 100 and the vehicles 200 - 1 to 200 - 4 may communicate with each other in a non-orthogonal multiple access method.

For example, the base station 100 may transmit signals to the first vehicle 200 - 1 and the second vehicle 200 - 2 with different signal strengths at the same frequency (downlink channel NOMA1). . At this time, the base station 100 is the first vehicle 200-1 and the second vehicle 200- according to the distance from the base station 100 to the first vehicle 200-1 and the second vehicle 200-2, respectively. 2), you can adjust the strength of the transmitted signal. The first vehicle 200-1 and the second vehicle 200 are such that the difference between the distance between the base station 100 and the first vehicle 200-1 and the distance between the base station 100 and the second vehicle 200-2 is large. It may be desirable to select -2). The base station 100 may determine the communication range of the downlink channel NOMA1 based on a vehicle that is farther from the base station 100 among the first vehicle 200 - 1 and the second vehicle 200 - 2 .

Also, the third vehicle 200 - 3 may transmit a signal to the base station 100 and the fourth vehicle 200 - 4 by varying the signal strength at the same frequency (uplink channel NOMA2). At this time, the third vehicle 200-3 is the base station 100 and the fourth vehicle 200- according to the distances from the third vehicle 200-3 to the base station 100 and the 42nd vehicle 200-4, respectively. 4), you can adjust the strength of the transmitted signal. The fourth vehicle 200-4 is selected so that the difference between the distance between the third vehicle 200-3 and the base station 100 and the distance between the third vehicle 200-3 and the fourth vehicle 200-4 is large. It may be desirable to The third vehicle 200 - 3 may determine the communication range of the uplink channel NOMA2 based on a vehicle having a greater distance from the base station 100 among the base station 100 and the fourth vehicle 200 - 4 .

In the present invention, an area corresponding to a road is divided into a plurality of sections HF. The compartments HF may be defined in a preset shape, and most simply, may be defined in a rectangular shape.

In FIG. 1 , a road is illustrated as a two-lane reciprocating lane, and sections HF are defined to include the entire width of the reciprocating two-lane, but the technical spirit of the present invention is not limited thereto. For example, each of the sections HF may be set to include only one lane.

In addition, in each of the compartments HF, a resource, for example, a frequency, etc. for wireless communication is allocated, and there is only one communication device, eg, one vehicle 200, in each of the compartments HF. Since it is desirable (when there are a plurality of vehicles, one frequency is allocated to the plurality of vehicles, there is a possibility that a frequency collision may occur), the size of the partitions HF may be defined in consideration of this. That is, the size of each of the compartments HF may be preferably set to be smaller than the size of the vehicle 200 .

As shown in FIG. 1 in the wireless communication system 10, when only a small number of base stations 100 and vehicles 200 exist, channels are provided to the base stations 100 and vehicles 200 without a large delay in various ways. However, when there are a plurality of base stations 100 and vehicles 200, channels cannot be appropriately allocated using a predetermined algorithm.

The resource allocating apparatus 300 according to an embodiment of the present invention allocates resources to be used by the base station 100 and the vehicles 200 in the wireless communication system 10 according to a policy determined through deep reinforcement learning.

The resource allocating apparatus 300 may perform a resource allocation method according to an embodiment of the present invention as a kind of computing system. That is, the resource allocation method of the present invention may be performed by a computer.

The resource allocating apparatus 300 may be provided in a computing system connected to the base station 100 . For example, the resource allocating apparatus 300 may implement its function in the base station 100 or may be implemented in a server connected to the base station 100 .

In the present specification, a channel may mean a resource for wireless communication divided by at least one of a frequency and a signal strength.

The structure and operation of the resource allocating apparatus 300 will be described in more detail with reference to FIGS. 2 and 3 .

FIG. 2 is a block diagram illustrating an apparatus for allocating resources according to an embodiment of the present invention, and FIG. 3 is a flowchart illustrating a reinforcement learning process of the reinforcement learning unit shown in FIG. 2 .

2 and 3 , the resource allocating apparatus 300 may include a reinforcement learning unit 310 and a resource allocating unit 330 .

The reinforcement learning unit 310 changes the channels allocated to the compartments HF and the positions of the vehicles 200 in the direction of increasing the total data throughput between the vehicles 200 and the base station 100 . ) can be modified by reinforcement learning.

In the reinforcement learning, the environment is the length of the X-axis and the Y-axis of the wireless communication system 10, the length, width, and location (coordinates) of the road, the properties of the wireless link (uplink, downlink), vehicle It can be defined as the moving speed of the devices 200 , etc., the action can be defined as allocating a channel to each of the compartments HF, and the reward can be defined as the overall in the wireless communication system 10 . It can be defined as data throughput, policy can be defined as a criterion (neural network) for performing optimal resource allocation in a given state, and loss can be defined as an agent in a given state. It can be defined as the difference between the expected reward for performing a given behavior and the measured reward for actually performing the corresponding behavior.

The reinforcement learning unit 310 may perform reinforcement learning while repeating the learning process of the minimum unit in three steps.

In the present specification, for convenience of description, a repeating unit is defined as an epoch, an ear, and an episode, in a small order. The upper step may mean repeating the lower step a set number of times while changing the condition. For example, epoch means the learning process of the minimum unit, 1 epoch means that the epoch is repeated n times (n is a natural number greater than or equal to 2), and 1 episode means that the epoch is repeated m times (m is a natural number greater than or equal to 2). ) is repeated. The episode is repeated p times (p being a natural number greater than or equal to 2).

Reinforcement learning unit 310, as a pre-processing of deep reinforcement learning, defines the zones HF, or defines the number of repetitions of the learning step, that is, the number of repetitions of epochs, errata and episodes, or the base station 100 and the vehicle. It is possible to set the position of the 200.

Steps S100 to S500 of FIG. 3 may be performed by the reinforcement learning unit 310 , specifically, an agent of the reinforcement learning unit. In this specification, for convenience of description, the agent is collectively referred to as the reinforcement learning unit 310 .

As a first step of deep reinforcement learning, the reinforcement learning unit 310 may observe the state of the wireless communication system 10 (S100). For example, the reinforcement learning unit 310 may observe the positions of the base station 100 and the vehicle light 200 in the wireless communication system 10 .

Reinforcement learning unit 310, as a learning process of a minimum unit, the total data throughput between the base station 100 and the vehicles 200 is increased while changing the channels allocated to the compartments HF based on the observed state. It is possible to modify the policy in the direction (S200). In other words, the reinforcement learning unit 310 measures the total data throughput as a reward and maximizes the reward while changing the channels allocated to the compartments HF for a predetermined number of learning times based on the observed state. Policies can be updated to minimize losses.

A process by which the reinforcement learning unit 310 performs the learning process of the minimum unit will be described in more detail with reference to FIG. 4 .

FIG. 4 is a flowchart illustrating the step S200 shown in FIG. 3 in more detail.

Referring to FIG. 4 , the reinforcement learning unit 310 may generate a resource allocation table according to a policy and allocate channels to the partitions HF based on the generated resource allocation table ( S201 ).

The resource allocation table may be defined as a two-dimensional array having the number of the partitions HF as an index and the channel number to be used in each of the partitions HF as an element.

Allocating one channel to one partition may be most preferable in terms of reducing system complexity. This is because when one channel is allocated to one compartment when actual wireless communication is performed after reinforcement learning, the vehicles 200 can communicate by determining a channel according to their location without an additional confirmation/verification process.

The reinforcement learning unit 310 may measure the total data throughput in the wireless communication system 10 by observing the communication between the vehicles 200 and the base station 100 ( S203 ).

The total data throughput is the data throughput in the uplink from any one of the vehicles 200 to another of the vehicles 200 or the base station 100 and the data throughput in the downlink from the base station 100 to the vehicles 200 . of data throughput.

The total data throughput may be calculated by Equation 1 below, which may serve as an objective function in deep reinforcement learning.

[Equation 1]

는 상기 채널들 중에서 n번째 채널을 통해 차량들(200) 중에서 i번째 차량으로부터 차량들(200) 중에서 다른 차량으로의 상향 링크에서의 제1 데이터 처리량을 나타낸다.

는 상기 n번째 채널을 통한 상기 i번째 차량으로부터 복수의 기지국들 중에서 대응하는 기지국으로의 상향 링크에서의 제2 데이터 처리량을 나타낸다.

는 상기 n번째 채널을 통한 복수의 기지국들 중에서 j번째 기지국으로부터 차량들(200) 중에서 제1 차량으로의 하향 링크에서의 제3 데이터 처리량,

는 상기 n번째 채널을 통한 상기 j번째 기지국으로부터 차량들(200) 중에서 제2 차량으로의 하향 링크에서의 제4 데이터 처리량을 나타낸다.In Equation 1, N represents a set of channels, V represents a set of vehicles 200 , and B represents a set of base stations 100 .

denotes a first data throughput in an uplink from the i-th vehicle among the vehicles 200 to another vehicle among the vehicles 200 through the n-th channel among the channels.

denotes a second data throughput in an uplink from the i-th vehicle through the n-th channel to a corresponding base station among a plurality of base stations.

is a third data throughput in a downlink from a j-th base station among a plurality of base stations through the n-th channel to a first vehicle among vehicles 200;

denotes a fourth data throughput in a downlink from the j-th base station to a second vehicle among the vehicles 200 through the n-th channel.

The first data throughput may be calculated according to Equation 2 below.

[Equation 2]

는 상기 i번째 차량에 상기 n번째 채널의 할당 여부를 나타내는 값이다. 즉,

이 '1'이면 상기 i번째 차량에 상기 n번째 채널이 할당된 것을 나타내고, 반대로,

이 '0'이면 상기 i번째 차량에 상기 n번째 채널이 할당되지 않은 것을 나타낸다.

는 상기 i번째 차량과 상기 다른 차량 사이의 상기 n번째 채널의 제1 SINR(signal to interference and noise ratio) 값을 나타낸다.in Equation 2

is a value indicating whether the n-th channel is allocated to the i-th vehicle. in other words,

If this is '1', it indicates that the n-th channel is allocated to the i-th vehicle, and conversely,

If this value is '0', it indicates that the n-th channel is not allocated to the i-th vehicle.

denotes a first signal to interference and noise ratio (SINR) value of the n-th channel between the i-th vehicle and the other vehicle.

The second data throughput may be calculated according to Equation 3 below.

[Equation 3]

는 상기 i번째 차량에 상기 n번째 채널의 할당 여부를 나타내는 값이다.

는 상기 i번째 차량과 상기 복수의 기지국들 중에서 상기 대응하는 기지국 사이의 상기 n번째 채널의 제2 SINR 값을 나타낸다.in Equation 3

is a value indicating whether the n-th channel is allocated to the i-th vehicle.

denotes a second SINR value of the n-th channel between the i-th vehicle and the corresponding base station among the plurality of base stations.

The third data throughput may be calculated according to Equation 5 below.

[Equation 5]

는 상기 j번째 기지국에 상기 n번째 채널의 할당 여부를 나타내는 값이다.

는 상기 n번째 채널을 통한 상기 j번째 기지국으로부터 상기 제1 차량으로의 하향 링크에서의 제3 SINR 값을 나타낸다.In Equation 5,

is a value indicating whether the n-th channel is allocated to the j-th base station.

denotes a third SINR value in a downlink from the j-th base station to the first vehicle through the n-th channel.

The fourth data throughput may be calculated according to Equation 6 below.

[Equation 6]

는 상기 j번째 기지국에 상기 n번째 채널의 할당 여부를 나타내는 값이다.

는 상기 n번째 채널을 통한 상기 j번째 기지국으로부터 상기 제2 차량으로의 하향 링크에서의 제4 SINR 값을 나타낸다.In Equation 6,

is a value indicating whether the n-th channel is allocated to the j-th base station.

denotes a fourth SINR value in a downlink from the j-th base station to the second vehicle through the n-th channel.

The first SINR value may be calculated according to Equation 7 below.

[Equation 7]

는 상기 i번째 차량으로부터 상기 다른 차량으로 송신하는 상향 링크 신호의 세기(전력 크기)를 나타낸다. s는 차량들(200) 중에서 s번째 차량을 나타내며,

는 상기 s번째 차량에 상기 n번째 채널의 할당 여부를 나타내는 값이다.

는 상기 s번째 차량으로부터 상기 s번째 차량이 NOMA 방식으로 통신하는 특정 차량으로 송신하는 신호의 세기를 나타내며,

는 상기 s번째 차량으로부터 상기 s번째 차량이 NOMA 방식으로 통신하는 기지국으로 송신하는 신호의 세기를 나타낸다. t는 복수의 기지국들(100) 중에서 t번째 기지국을 나타내며,

는 상기 t번째 기지국에 상기 n번째 채널의 할당 여부를 나타내는 값이다.

는 상기 t번째 기지국으로부터 상기 t번째 기지국이 NOMA 방식으로 통신하는 차량들 중에서 어느(가까운) 하나로 송신하는 신호의 세기를 나타내며,

는 상기 t번째 기지국으로부터 상기 t번째 기지국이 NOMA 방식으로 통신하는 차량들 중에서 다른(먼) 하나로 송신하는 신호의 세기를 나타낸다.

는 잡음 세기 상수를 나타낸다.in Equation 7

denotes the strength (power level) of an uplink signal transmitted from the i-th vehicle to the other vehicle. s represents the s-th vehicle among the vehicles 200,

is a value indicating whether the n-th channel is allocated to the s-th vehicle.

represents the strength of a signal transmitted from the s-th vehicle to a specific vehicle that the s-th vehicle communicates with in the NOMA method,

denotes the strength of a signal transmitted from the s-th vehicle to a base station communicating with the s-th vehicle in the NOMA method. t represents the t-th base station among the plurality of base stations 100,

is a value indicating whether the n-th channel is allocated to the t-th base station.

represents the strength of a signal transmitted from the t-th base station to any (closer) one of the vehicles that the t-th base station communicates with in the NOMA method,

denotes the strength of a signal transmitted from the t-th base station to another (far) one of the vehicles communicating by the t-th base station in the NOMA method.

is the noise intensity constant.

이라고 함은 k번째 채널을 통해 l번째 송신기와 m번째 수신기 사이의 채널 이득을 의미하며, 다음의 수학식 8에 의해 정의된다.H stands for the channel gain between the transmitter and the receiver,

denotes a channel gain between the l-th transmitter and the m-th receiver through the k-th channel, and is defined by Equation 8 below.

[Equation 8]

는 상기 k번째 채널에서 상기 l번째 송신기와 상기 m번째 수신기 사이의 fast fading component를 나타내는데, 이는 complex Gaussian 분포를 따르며

이다.

은 상기 l번째 송신기와 상기 m번째 수신기 사이의 log-normal shadowing 랜덤 변수를 나타내며, 표준편차

를 가진다.

은 상기 l번째 송신기와 상기 m번째 수신기 사이의 path-loss 상수를 나타낸다.

은 상기 l번째 송신기와 상기 m번째 수신기 사이의 거리를 나타낸다.

은 상기 l번째 송신기의 path-loss exponent를 나타낸다.In Equation 8,

denotes a fast fading component between the l-th transmitter and the m-th receiver in the k-th channel, which follows a complex Gaussian distribution,

to be.

denotes a log-normal shadowing random variable between the l-th transmitter and the m-th receiver, standard deviation

have

denotes a path-loss constant between the l-th transmitter and the m-th receiver.

is the distance between the l-th transmitter and the m-th receiver.

denotes the path-loss exponent of the l-th transmitter.

The second SINR value may be calculated according to Equation 9 below.

[Equation 9]

이

으로,

이

으로,

이

으로 변경된 것을 제외하고는 실질적으로 동일하므로 중복되는 설명은 생략한다.

는 상기 i번째 차량이 상기 기지국으로 송신하는 상향 링크 신호의 세기를 나타낸다.Equation 9 is from Equation 7

this

by,

this

by,

this

Since it is substantially the same except for the change to , the overlapping description will be omitted.

denotes the strength of an uplink signal transmitted by the i-th vehicle to the base station.

The third SINR value may be calculated according to Equation 10 below.

[Equation 10]

이

으로,

이

으로,

이

으로 변경된 것을 제외하고는 실질적으로 동일하므로 중복되는 설명은 생략한다.

는 상기 j번째 기지국이 상기 제1 차량으로 송신하는 하향 링크 신호의 세기를 나타낸다.Equation 10 is obtained from Equation 7

this

by,

this

by,

this

Since it is substantially the same except for the change to , the overlapping description will be omitted.

denotes the strength of a downlink signal transmitted by the j-th base station to the first vehicle.

The fourth SINR value may be calculated according to Equation 11 below.

[Equation 11]

이

으로,

이

으로,

이

으로 변경된 것을 제외하고는 실질적으로 동일하므로 중복되는 설명은 생략한다.

는 상기 j번째 기지국이 상기 제2 차량으로 송신하는 하향 링크 신호의 세기를 나타낸다.Equation 11 is obtained from Equation 7

this

by,

this

by,

this

Since it is substantially the same except for the change to , the overlapping description will be omitted.

denotes the strength of a downlink signal transmitted by the j-th base station to the second vehicle.

The reinforcement learning unit 310 may update the weight of the neural network so that the loss is minimized (S205).

The reinforcement learning unit 310 may change the weight of the neural network so that a loss that is a difference between a value calculated by the objective function and an actual simulated or measured value can be minimized.

Steps S201 to S205 shown in FIG. 4 represent the learning process of the minimum unit in the present invention.

Referring back to FIGS. 2 and 3 , the reinforcement learning unit 310 counts the number of repetitions of the epoch S200, which is a learning process of the minimum unit, and repeats it n times (n is a natural number greater than or equal to 2) (NO branch of S210) .

When the epoch is repeated n times and 1 Era is finished (YES branch of S220 ), the reinforcement learning unit 310 may move the positions of the vehicles 200 according to the driving information set for each vehicle 200 ( S300 ). ).

According to an embodiment, the driving information may include the traveling direction and speed of each of the vehicles 200 .

After moving the vehicles 200 , the reinforcement learning unit 310 starts reinforcement learning of S200 to S210 , that is, a new erasure. The reinforcement learning unit 310 counts the number of repetitions of the error and repeats it m times (m is a natural number equal to or greater than 2) (NO branch of S310).

When Erra is repeated m times and one episode ends (YES branch of S310 ), the reinforcement learning unit 410 initializes the positions of the vehicles 200 . Here, initialization means randomly determining the positions of the vehicles 200 , and may include adding or removing vehicles 200 .

After initializing the positions of the vehicles 200 , the reinforcement learning unit 310 starts reinforcement learning of S200 to S310 , that is, a new episode. The reinforcement learning unit 310 counts the number of repetitions of the episode and repeats it p times (p is a natural number greater than or equal to 2) (NO branch of S410).

After the episode is repeated p times (NO branch of S410), the reinforcement learning unit 310 derives a resource allocation model according to the policy of the last version (S500). For example, the reinforcement learning unit 310 may generate a resource allocation table according to the strategy and output the generated resource allocation table to the resource allocation unit 330 .

The resource allocation unit 330 may transmit the resource allocation table received from the reinforcement learning unit 310 to the base station 100 and the vehicles 200 . The resource allocator 330 may receive the final resource allocation table from the reinforcement learning unit 310 as well as receive the resource allocation table during reinforcement learning.

The base station 100 and the vehicles 200 may transmit data by selecting a channel to be used for wireless communication by themselves based on the resource allocation table.

As the resource allocation table is determined through such a deep reinforcement learning process and used in the wireless communication system 10, the base station 100 and the vehicles 200 can perform wireless communication without a large delay such as an available channel search time. And according to the characteristics of deep reinforcement learning, very efficient and collision-free channel assignment is possible.

5 to 8 show the process and results of simulating the technical idea of the present invention.

FIG. 5 is a diagram illustrating a virtual system for simulating the reinforcement learning process shown in FIG. 3 , FIG. 6 is a graph illustrating a change in loss in the initial stage of learning while reinforcement learning is performed, and FIG. 7 is reinforcement learning It is a graph showing a change in loss in the late learning stage while this is being performed, and FIG. 8 is a graph showing a change in a discounted reward while reinforcement learning is performed.

5 to 8, as shown in Figure 5, both the X-axis and the Y-axis in the wireless communication system 10 in an area of 2,000 [m] or more 9 base stations (MBS) and a plurality of vehicles (VUE) was set to exist. In such a setting, epochs were simulated as repeating thousands of times, errata repeating hundreds of times, and episodes repeating dozens of times.

As shown in FIG. 6 , in the early stages of learning (first episode, first epoch, first epoch), the loss, which is the difference between the result of the objective function and the actual simulation result, was found to be relatively large and even divergent.

However, as shown in FIG. 7 . In the middle of the learning (first episode, 40th epoch, first epoch), the loss quickly converges to '0'.

In addition, as shown in FIG. 8 , in the latter half of learning (after performing several episodes), the discounted reward has a high value and continuously rises even in an arbitrary error.

Through such simulation results, it can be recognized that resource allocation in non-orthogonal multiple access method-based vehicle communication can be efficiently performed through deep reinforcement learning according to the present invention.

Although the present invention has been described with reference to one embodiment shown in the drawings, this is merely exemplary, and those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

10; wireless communication system
100; base station
200; vehicle
300; resource allocation device
310; reinforcement learning department
330; resource allocator

Claims (21)

Reinforcement learning unit for performing reinforcement learning to modify the policy (policy) to increase the total data throughput between the vehicles and at least one base station while changing the locations of at least two or more vehicles and channels allocated to a plurality of compartments; and
Non-orthogonal including a resource allocation unit that generates a resource allocation table in which one of the channels is allocated to each of the partitions according to a policy determined by the reinforcement learning unit, and transmits the resource allocation table to the vehicles and the base station Resource allocation device in multi-access method-based vehicle communication. According to claim 1,
The channel is a resource allocating apparatus in a non-orthogonal multiple access method-based vehicle communication that is distinguished by at least one of a frequency and a signal strength. According to claim 1,
The reinforcement learning is
After performing an action to change the channels allocated to the compartments, the data throughput is measured as a reward and the learning process of modifying the policy so that the total data throughput is increased n times (n is 2) (a) step of repeating (a) or more natural numbers;
(b) repeating step (a) m times (m is a natural number equal to or greater than 2) while moving the positions of the vehicles according to the driving information set for each vehicle; and
and (c) repeating step (b) p times (p is a natural number greater than or equal to 2) while changing the initial positions of the vehicles. 4. The method of claim 3,
The step (a) is,
allocating the channels to the partitions based on a resource allocation table generated according to the policy;
measuring the overall data throughput by observing communication between the vehicles and the base station; and
A resource allocation apparatus in non-orthogonal multiple access method-based vehicle communication, comprising the step of updating a weight of a neural network so that a loss, which is a difference between a result value of a predetermined objective function and the compensation, is minimized. 4. The method of claim 3,
The driving information is an apparatus for allocating resources in vehicle communication based on a non-orthogonal multiple access method including the traveling direction and speed of each of the vehicles. According to claim 1,
In the non-orthogonal multiple access scheme-based vehicle communication, the total data throughput includes a data throughput in an uplink from each of the vehicles to another vehicle or the base station and a data throughput in a downlink from the base station to the vehicles. of resource allocation device. According to claim 1,
The total data throughput is calculated by Equation 1 below,
[Equation 1]

where N is the set of channels, V is the set of vehicles, B is the set of the base station,

is the first data throughput in the uplink from the i-th vehicle to another vehicle through the n-th channel,

is a second data throughput in the uplink from the i-th vehicle to the base station through the n-th channel,

is the third data throughput in the downlink from the j-th base station to the first vehicle through the n-th channel,

is a resource allocation apparatus in non-orthogonal multiple access scheme-based vehicle communication indicating a fourth data throughput in a downlink from the j-th base station to a second vehicle through the n-th channel. 8. The method of claim 7,
The first data throughput is calculated according to Equation 2 below,
[Equation 2]

here,

is a value indicating whether the n-th channel is allocated to the i-th vehicle,

is a resource allocation apparatus in non-orthogonal multiple access scheme-based vehicle communication indicating a first signal to interference and noise ratio (SINR) value of the n-th channel between the i-th vehicle and the other vehicle. 8. The method of claim 7,
The second data throughput is calculated according to the following Equation 3,
[Equation 3]

here,

is a value indicating whether the n-th channel is allocated to the i-th vehicle,

is a resource allocation apparatus in non-orthogonal multiple access scheme-based vehicle communication indicating a second SINR value of the n-th channel between the i-th vehicle and the base station. 8. The method of claim 7,
The third data throughput is calculated according to the following Equation 5,
[Equation 4]

here,

is a value indicating whether the n-th channel is allocated to the j-th base station,

is a resource allocation apparatus in non-orthogonal multiple access scheme-based vehicle communication indicating a third SINR value in a downlink from the j-th base station to the first vehicle through the n-th channel. 8. The method of claim 7,
The fourth data throughput is calculated according to Equation 6 below,
[Equation 6]

here,

is a value indicating whether the n-th channel is allocated to the j-th base station,

is a resource allocation apparatus in non-orthogonal multiple access scheme-based vehicle communication indicating a fourth SINR value in a downlink from the j-th base station to the second vehicle through the n-th channel. After performing an action of changing channels allocated to a plurality of compartments, the total data throughput between at least two or more vehicles and at least one or more base stations is measured as a reward, and the policy (policy) to increase the total data throughput (a) repeating the learning process to correct ) n times (n is a natural number greater than or equal to 2);
(b) repeating step (a) m times (m is a natural number equal to or greater than 2) while moving the positions of the vehicles according to the driving information set for each vehicle;
(c) repeating step (b) p times (p is a natural number greater than or equal to 2) while changing the initial positions of the vehicles; and
(d) allocating the channels to the compartments according to the policy determined through the steps (a) to (c). 13. The method of claim 12,
The step (a) is,
allocating the channels to the partitions based on a resource allocation table generated according to the policy;
measuring the overall data throughput by observing communication between the vehicles and the base station; and
A method for allocating resources in a non-orthogonal multiple access scheme-based vehicle communication, comprising: updating a weight of a neural network so that a loss, which is a difference between a result value of a predetermined objective function and the compensation, is minimized. 13. The method of claim 12,
The channel is a resource allocation method in a non-orthogonal multiple access method-based vehicle communication that is distinguished by at least one of a frequency and a signal strength. 13. The method of claim 12,
The method of allocating resources in vehicle communication based on a non-orthogonal multiple access method, wherein the driving information includes a traveling direction and speed of each of the vehicles. 13. The method of claim 12,
In the non-orthogonal multiple access scheme-based vehicle communication, the total data throughput includes a data throughput in an uplink from each of the vehicles to another vehicle or the base station and a data throughput in a downlink from the base station to the vehicles. resource allocation method. 13. The method of claim 12,
The total data throughput is calculated by Equation 1 below,
[Equation 1]

where N is the set of channels, V is the set of vehicles, B is the set of the base station,

is the first data throughput in the uplink from the i-th vehicle to another vehicle through the n-th channel,

is a second data throughput in the uplink from the i-th vehicle to the base station through the n-th channel,

is the third data throughput in the downlink from the j-th base station to the first vehicle through the n-th channel,

represents a fourth data throughput in a downlink from the j-th base station to a second vehicle through the n-th channel. A resource allocation method in non-orthogonal multiple access scheme-based vehicle communication. 18. The method of claim 17,
The first data throughput is calculated according to Equation 2 below,
[Equation 2]

here,

is a value indicating whether the n-th channel is allocated to the i-th vehicle,

is a resource allocation method in non-orthogonal multiple access scheme-based vehicle communication indicating a first signal to interference and noise ratio (SINR) value of the n-th channel between the i-th vehicle and the other vehicle. 18. The method of claim 17,
The second data throughput is calculated according to the following Equation 3,
[Equation 3]

here,

is a value indicating whether the n-th channel is allocated to the i-th vehicle,

is a resource allocation method in non-orthogonal multiple access scheme-based vehicle communication indicating the second SINR value of the n-th channel between the i-th vehicle and the base station. 18. The method of claim 17,
The third data throughput is calculated according to the following Equation 5,
[Equation 4]

here,

is a value indicating whether the n-th channel is allocated to the j-th base station,

is a method for allocating resources in non-orthogonal multiple access scheme-based vehicle communication indicating a third SINR value in a downlink from the j-th base station to the first vehicle through the n-th channel. 18. The method of claim 17,
The fourth data throughput is calculated according to Equation 6 below,
[Equation 5]

here,

is a value indicating whether the n-th channel is allocated to the j-th base station,

is a method for allocating resources in non-orthogonal multiple access scheme-based vehicle communication indicating a fourth SINR value in a downlink from the j-th base station to the second vehicle through the n-th channel.

Images