KR20220009308A - Packet Transmission Decision Apparatus And Packet Transmission Schedule Decision Method - Google Patents

Abstract

An apparatus for determining packet transmission includes: a pre-processing unit which generates status information for each determination time based on behavior information that has transmitted a packet and feedback information received through packet transmission; a data management unit which generates redefined status information by using the status information and accumulated status information; a value calculating unit which calculates a present value of the packet transmission and a future value of the packet transmission by using the redefined status information; a Q-value calculating unit which calculates a Q-value for the packet transmission by summing the present and future values; and a decision unit which determines a packet transmission schedule at the next determination time based on the Q-value.

Description

Packet Transmission Decision Apparatus And Packet Transmission Schedule Decision Method

The present invention relates to an apparatus for determining packet transmission and a method for determining a packet transmission schedule, and more particularly, to an apparatus for determining packet transmission and packet transmission for improving fairness between user terminals by calculating a Q-Value using status information according to packet transmission How to decide on a schedule.

The conventional machine learning-based multi-channel access method is a centralized radio resource management method in which a single agent collects information of all users and allocates channels. Recently, machine learning-based multi-channel access methods applicable in a distributed environment have been proposed, but there is a limitation in radio resource management because a fixed set of users is assumed.

In addition, the conventional random access technique corresponds to a group of techniques based on optimization and decision algorithms. Therefore, there is a limit in adapting to the communication environment changing in real time, and optimization also has a limit due to the complexity of the calculation.

In addition, in conventional machine learning-based radio resource management techniques, the centralized agent collects learning data and then allocates radio resources to multiple users through learning. Therefore, it is difficult to support the Massive Connectivity required for 5G and Beyond 5G cellular communication. In addition, the recently proposed machine learning-based distributed radio resource management technology group also assumes a time-invariant set of users, and there is a limit to adapting to the real-time changing Internet of Things environment because a lot of learning data and a long learning time are required.

Republic of Korea Patent Publication No. 10-2019-0127480 (Method and apparatus for determining transmission point and link adaptation scheme in a communication system, Korea Electronics and Telecommunications Research Institute, 2019.11.13)

The technical problem to be achieved by the present invention is to solve these problems or limitations, and the present invention provides a packet transmission determining device and packet transmission that improve fairness between user terminals by calculating a Q-Value using status information according to packet transmission. It provides a method for determining a schedule.

The apparatus for determining packet transmission of the present invention includes a pre-processing unit that generates status information for each determination time based on behavior information of packet transmission and feedback information received through packet transmission, status information redefined using status information and accumulated status information A data management unit that creates a value, a value calculation unit that calculates the present value and future value of packet transmission using the redefined state information, and a Q that calculates the Q-Value for packet transmission by adding up the present value and future value It may include a -Value calculator and a decision part that determines a packet transmission schedule at the next decision time based on the Q-Value.

The value calculation unit can calculate the present value and future value by inputting the redefined state information into the deep Q-Network prepared in advance.

The value calculating unit may include a present value calculating unit comprising a plurality of Deep Q-Networks corresponding to the maximum number of allowable user terminals.

The present value calculating unit may be characterized in that the present value is calculated by using only the number of deep Q-Networks corresponding to the size of the decision time among the plurality of Deep Q-Networks.

The present value calculator may be characterized by normalizing the present values calculated from the plurality of deep Q-Networks.

The value calculation unit may include a future value calculation unit that calculates a future value that is a value of packet transmission after the determination time at which the present value is determined by using the redefined state information.

The determination unit may determine a transmission channel corresponding to a timeslot within a determination time in which the highest Q-Value is calculated among Q-Values as a packet transmission schedule at a next determination time.

The determining unit may apply zero-padding to a timeslot not determined by the packet transmission schedule to match the size of the packet transmission schedule and the size of the determination time.

The data management unit may be characterized by time-series learning and aggregation of the state information and the redefined state information according to the elapse of a decision time using a Long Short-Term Memory (LSTM) model.

The packet transmission schedule determination method of the present invention includes a pre-processing step in which the pre-processing unit generates status information at each determination time based on the packet transmission behavior information and the feedback information received through packet transmission, and the data management unit collects the status information and accumulated status information. A data management step that generates the redefined state information using the It may include a Q-Value calculation step of calculating a Q-Value for packet transmission by summing values, and a determining step of determining a packet transmission schedule at a next determination time by a decision unit based on the Q-Value.

The apparatus for determining packet transmission and the method for determining a packet transmission schedule according to the present invention can ensure a data rate for each user terminal and fairness between user terminals.

In addition, the apparatus for determining packet transmission and the method for determining a packet transmission schedule of the present invention can reduce computational complexity compared to the existing centralized machine learning and distributed machine learning.

In addition, the apparatus for determining packet transmission and the method for determining a packet transmission schedule of the present invention can be utilized as access technology in various IoT-related fields or 5G and Beyond 5G cellular communication systems.

1 is a diagram illustrating a random access communication system according to an embodiment.
2 is a diagram illustrating an operation of a random access communication system according to an embodiment.
3 is a block diagram illustrating a configuration of a user terminal according to an embodiment.
4 is a diagram illustrating a packet transmission schedule according to an embodiment.
5 is a block diagram illustrating a configuration of an apparatus for determining packet transmission according to an embodiment.
6 is a block diagram illustrating a configuration of a present value calculator according to an exemplary embodiment.
7 is a diagram illustrating a packet transmission schedule determination process of an apparatus for determining packet transmission according to an embodiment.
8 is a flowchart illustrating a method of determining a packet transmission schedule using the apparatus for determining packet transmission according to an embodiment.
9A is a diagram illustrating the fairness of packet transmission between active user terminals according to the lapse of time according to an embodiment.
9B is a diagram illustrating a data rate and a target data rate for each active user terminal according to an embodiment.

Hereinafter, the present invention will be described in detail so that those skilled in the art can easily understand and reproduce the invention through the embodiments described with reference to the accompanying drawings. However, when it is determined that a detailed description of a known function or configuration related to the invention may unnecessarily obscure the gist of the embodiments of the invention, the detailed description thereof will be omitted.

The terms used below are terms selected in consideration of functions in the embodiment, and the meaning of the terms may vary depending on the intention or custom of a user or operator. Therefore, the meanings of terms used in the examples to be described below follow the definitions when they are specifically defined below, and when there is no specific definition, they should be interpreted as meanings generally recognized by those skilled in the art. In addition, the same reference numerals or reference numerals in each drawing indicate parts or components that perform substantially the same functions.

Hereinafter, an apparatus for determining packet transmission and a method for determining a packet transmission schedule according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings and contents described therein.

1 is a diagram illustrating a random access communication system according to an embodiment of the present invention.

Referring to FIG. 1 , a random access communication system 1 according to an embodiment of the present invention may include at least one user terminal 10 and a repeater 20 .

When there is a packet to be transmitted, the user terminal 10 may access the relay 20, transmit the packet in a random timeslot, and release the connection with the relay 20 when the packet transmission is completed.

Since the plurality of user terminals 10 repeatedly perform connection and disconnection with the repeater 20 for each timeslot, the number of active user terminals connected to the repeater 20 may change according to each timeslot.

The repeater 20 may generally mean an access point (AP) that communicates with the user terminal 10 , and thus may receive a packet transmitted from the user terminal 10 . The repeater 20 may receive the packet transmitted by the user terminal 10 for each time slot, and broadcast feedback information according to the reception of the packet.

Specifically, when only one packet is received in one timeslot, the repeater 20 may decode the received packet and broadcast an ACK signal to the user terminal 10 side.

However, when a plurality of packets are received in one timeslot, the repeater 20 may broadcast the NAK signal to the user terminal 10 without decoding the corresponding packets.

2 is a diagram illustrating an example in which a random access communication system according to an embodiment of the present invention operates.

Referring to FIG. 2, FIG. 2 shows an example of operation of a random access communication system in the case where there are 5 active user terminals and 2 transmission channels. Here, a user terminal that accesses a repeater for packet transmission at each decision time (i=1,2,??,n) is defined as an active user terminal, and the decision time will be described later.

Looking at the packet transmission operation according to time lapse, active user terminal 1 may transmit a packet to the relay through transmission channel 2 in timeslot 1, and the repeater receives only packets of active user terminal 1 in transmission channel 2 of timeslot 1 Therefore, after decoding the corresponding packet, the ACK signal can be broadcast to the active user terminal 1 side.

Active user terminal 2 may transmit a packet to the repeater through transport channel 1 in timeslot 1, and since the repeater has only received packets of active user terminal 1 in transport channel 1 of timeslot 1, after decoding the packet, active user terminal An ACK signal may be broadcast to one side.

Although the plurality of active user terminals did not participate in packet transmission in timeslot 2, the repeater may broadcast the NAK signal to the plurality of active user terminals. Here, the NAK signal broadcast in timeslot 2 may be a NAK signal meaning that no user terminal accesses the repeater.

Active user terminals 2 and 4 may transmit packets to the relay through transport channel 2 in timeslot 3, and since the repeater has received a plurality of packets in transport channel 2 of timeslot 3, active user terminals without decoding the packets You can broadcast NAK signals to the 2nd and 4th sides. Here, the NAK signal broadcast in timeslot 3 may be a NAK signal indicating that a collision between packets has occurred.

Since the above operation is also performed in the subsequent determination time and timeslot, a description thereof will be omitted.

3 is a block diagram illustrating the configuration of a user terminal according to an embodiment of the present invention.

Referring to FIG. 3 , the user terminal 10 may include a communication unit 110 , a process 120 , and a storage unit 130 .

The communication unit 110 may transmit a packet to the repeater and receive feedback information according to whether the repeater has received the packet.

The process 120 may generate a packet to be transmitted to the repeater and control the communication unit 110 to transmit the packet according to the packet transmission schedule determined by the packet transmission determining device 121 .

In more detail, the user terminal 10 may include a packet transmission determining device 121 for determining a timeslot and a transmission channel to transmit a packet in a random access communication system. The packet transmission determining device 121 may be provided as a separate hardware configuration in the user terminal 10 , but is not limited thereto and may also be configured in a software form driven by the process 120 of the user terminal 10 . can That is, the user terminal 10 and the packet transmission determining device 121 may be configured as one device.

The packet transmission determining apparatus 121 may determine a packet transmission schedule to transmit a packet at each determined time.

Here, the packet transmission schedule may be composed of a timeslot within a determined time and a transmission channel to be used for packet transmission, and the packet transmission schedule may be managed in the form of a specific matrix.

2 and 4, in the random access system when the size of the decision time is 5 and the number of transmission channels is 2 as shown in FIG. 2, the packet transmission schedule 30 as shown in FIG. 4 is the size of the decision time of 5 It may be composed of a matrix of a corresponding size, and may be generated in a form in which the number of a transmission channel to be used for packet transmission is stored for each timeslot.

In this case, a timeslot in which the number of a transport channel to be used for packet transmission is not stored may be processed as zero-padding, but is not limited thereto, and may be processed as a special value indicating that packet transmission is not performed. can

As an embodiment, when the number of active user terminals is n, the packet transmission determining apparatus 121 may determine the size of the determination time to be n hours, and may determine the packet transmission schedule 30 every n hours. When the determination time is determined according to the active user terminal, the user terminal 10 receives the number of active user terminals for each preset timeslot from the repeater, or utilizes the ACK signal and NAK signal broadcast by the repeater to the active user terminal number can be estimated.

The storage unit 130 may perform a role of storing a packet to be transmitted to the repeater or storing a packet generated by the process 120 .

Hereinafter, the configuration and operation of the user terminal including the apparatus for determining packet transmission according to an embodiment of the present invention will be described in detail with reference to the drawings.

5 is a block diagram showing the configuration of an apparatus for determining packet transmission according to an embodiment of the present invention.

The packet transmission determining device 121 calculates a Q-Value for each time slot within a decision time based on the behavior information, reward information, and status information of the user terminal, and according to the priority of the calculated Q-Value, the packet of the next decision time A transmission schedule may be determined.

Here, the behavior information, reward information, and status information of the user terminal refer to information used in the process of determining the packet transmission schedule or in each configuration of the packet transmission determining device 121, which will be described below. First, behavior information and reward information and the definition of status information will be described in advance.

Behavior information refers to information that the user terminal takes to transmit packets, and the behavior information may include information on whether or not packets are transmitted for each time slot and information on a transmission channel used for packet transmission.

로 표현될 수 있다. 여기서,

는 각 활성 사용자 단말,

는 결정시간, N은 전송 채널,

는 결정시간 i에서의 활성 사용자 단말의 수를 의미한다.Behavior information according to an embodiment of the present invention may be composed of a set of behaviors for the active user terminal to transmit packets in units of decision time, and the behavior information

can be expressed as here,

is each active user terminal,

is the decision time, N is the transmission channel,

denotes the number of active user terminals at decision time i.

In other words, the behavior information is information for the active user terminal to transmit packets during the determined time, and may indicate information on packet transmission schedules of timeslots belonging to the current determined time.

For example, when a packet transmission schedule including behavior information is given as shown in FIG. 4, the active user terminal transmits a packet through transport channel 2 in timeslot 1, and transmits a packet through transport channel 1 in timeslot 2, , may mean that packets are not transmitted in timeslots 3 to 5.

The reward information may be obtained from feedback information transmitted by the repeater by transmitting a packet according to the behavior information of the active user terminal. Here, the feedback information may include, but is not limited to, an ACK signal indicating completion of packet transmission and a NAK signal indicating packet collision.

The apparatus for determining packet transmission 121 according to an embodiment of the present invention sets 1 when receiving an ACK signal from the repeater according to packet transmission, -1 when receiving a NAK signal indicating collision occurrence, and ( = when no packet is transmitted), the compensation information can be determined as 0. Accordingly, compensation information for each time slot of the active user terminal can be expressed by Equation 1 below.

는 활성 사용자 단말,

는 시간슬롯을 의미할 수 있다.in Equation 1

is the active user terminal,

may mean a timeslot.

로 표현될 수 있다.However, the compensation information of the present invention may be composed of a set of compensation information determined according to the feedback information acquired through the action of the active user terminal in units of determination time, and the compensation information is

can be expressed as

As described above, by generating compensation information by different compensation values for each packet transmission success, packet transmission failure, and packet transmission idle state, the calculation accuracy of Q-Value, which will be described below, can be further improved.

로 표현될 수 있다.The status information may mean information including behavior information and reward information at a previous decision time. According to an embodiment of the present invention, since the behavior information and the reward information are configured in a matrix corresponding to the size of the decision time, the state information is

can be expressed as

Hereinafter, an apparatus for determining packet transmission based on the behavior information, reward information and status information defined above will be described in detail.

The packet transmission determination apparatus 121 may include a preprocessor 122 , a data management unit 123 , a value calculation unit 124 , a Q-Value calculation unit 127 , and a determination unit 128 .

를 생성하고, 생성된 상태정보가 인공지능을 통한 처리에 용이하도록 전처리를 수행하여 재정의된 상태정보

를 생성할 수 있다.The pre-processing unit 122 provides status information at the current decision time.

created state information, and pre-processing is performed to facilitate processing of the generated state information through artificial intelligence.

can create

를 생성하기 위하여 이에 대한 행동정보 및 보상정보를 생성할 수 있다.Specifically, the pre-processing unit 122 states information

In order to generate , it is possible to generate behavior information and reward information for it.

Here, the behavior information may be generated by the communication unit in the user terminal monitoring whether or not packets are transmitted, the pre-processing unit 122 may receive the corresponding behavior information from the communication unit, and the pre-processing unit 122 may transmit the packet transmission schedule of the previous transmission time. may be brought as behavior information, but is not limited thereto.

의 각 요소에 대하여 One-Hot 인코딩을 적용하여,

을 도출할 수 있다. 여기서,

는 각 활성 사용자 단말,

는 결정시간, N은 전송 채널,

는 결정시간 i에서의 활성 사용자 단말의 수를 의미한다.The pre-processing unit 122 is inputted behavior information

By applying One-Hot encoding to each element of

can be derived. here,

is each active user terminal,

is the decision time, N is the transmission channel,

denotes the number of active user terminals at decision time i.

When the ACK signal and the NAK signal are received through the communication unit, the pre-processing unit 122 may generate compensation information by matching each case as in Equation 1 based on this.

를 생성할 수 있다.Accordingly, the pre-processing unit 122 is state information based on the behavior information and the reward information.

can create

를 기초로, 활성 사용자 단말의 수

와 최대 허용 사용자 단말 수

에 따라 재정의된 상태정보

를 생성할 수 있다.Data management unit 123 state information

Based on the number of active user terminals

and the maximum number of user terminals allowed

Status information redefined according to

can create

와 최대 허용 사용자 단말 수

가 동일한 경우, 데이터관리부(123)는 상태정보

를 다음의 수학식 2와 같이 재정의하여 생성할 수 있다.Specifically, the number of active user terminals

and the maximum number of user terminals allowed

If is the same, the data management unit 123 is the status information

can be created by redefining as in Equation 2 below.

는 재정의된 상태정보,

는 활성 사용자 단말,

는 결정시간,

은 전송 채널의 수,

는 최대 허용 사용자 단말 수,

는 One-Hot 인코딩을 적용한 행동정보,

는 보상정보를 의미할 수 있다.in Equation 2

is the overridden state information,

is the active user terminal,

is the decision time,

is the number of transmission channels,

is the maximum number of allowed user terminals,

is behavior information applied with One-Hot encoding,

may mean compensation information.

가 최대 허용 사용자 단말 수

미만일 경우, 데이터관리부(123)는 상태정보

를 다음의 수학식 3과 같이 재정의할 수 있다.In addition, the number of active user terminals

is the maximum number of user terminals allowed

If less than, the data management unit 123 state information

can be redefined as in Equation 3 below.

는 재정의된 상태정보,

는 활성 사용자 단말,

는 결정시간,

은 전송 채널의 수,

는 최대 허용 사용자 단말 수,

는 시간슬롯의 수,

는 One-Hot 인코딩을 적용한 행동정보,

는 보상정보를 의미할 수 있다.in Equation 3

is the overridden state information,

is the active user terminal,

is the decision time,

is the number of transmission channels,

is the maximum number of allowed user terminals,

is the number of timeslots,

is behavior information applied with One-Hot encoding,

may mean compensation information.

가 최대 허용 사용자 단말 수

미만일 경우,

만큼 상태정보의 크기가 작아지므로, 데이터관리부(123)는 추가적으로 0을 삽입하는 제로패딩을 적용하여 수학식 3의

및

와 같이 크기를 맞춰 재정의된 상태정보

를 생성할 수 있다.However, the number of active user terminals

is the maximum number of user terminals allowed

If less than

Since the size of the state information is reduced as much as possible, the data management unit 123 applies zero padding to additionally insert 0 in Equation 3

and

Redefined state information to fit the size like

can create

또는 재정의된 상태정보

를 누적하여 저장할 수 있다. 즉, 데이터관리부(123)는 매 결정시간마다 생성되는 상태정보를 지속적으로 누적하여 저장할 수 있다.In addition, the data management unit 123 state information

or overridden status information

can be accumulated and stored. That is, the data management unit 123 may continuously accumulate and store the state information generated at every determination time.

Here, if the data management unit 123 continuously accumulates the accumulated state information and the redefined state information at every determination time, the number becomes enormous and problems such as data loss and learning errors may occur.

Accordingly, the data management unit 123 is configured in the form of a memory called an LSTM (Long Short-Term Memory) block, and the LSTM data in which the accumulated state information and the state information generated by the pre-processing unit 122 are aggregated according to the elapse of a decision time. Time series data can be processed and trained, and the output of the corresponding LSTM data can be efficiently adjusted.

In this way, the data management unit 123 introduces the LSTM model and aggregates the state information according to the elapse of the decision time, so that the packet transmission determining device 121 uses the past state information to estimate the current state information. accuracy can be improved. The LSTM data aggregated in the data management unit 123 in this way may be used as an input value of the value calculation unit 124 thereafter.

The value calculating unit 124 may include a present value calculating unit 125 and a future value calculating unit 126 .

를 입력하여 패킷전송의 현재가치를 산출할 수 있다.The present value calculation unit 125 provides state information redefined by artificial intelligence provided in advance.

can be input to calculate the present value of packet transmission.

입력을 통하여 패킷전송의 현재가치를 산출할 수 있다. 여기서 현재가치는 현재 결정시간만을 고려할 때 각 시간슬롯 및 전송 채널이 갖는 Q-Value를 의미할 수 있다.The present value calculation unit 125 may be composed of at least one Deep Q-Network (DQN), and state information redefined in a pre-prepared Deep Q-Network

The present value of packet transmission can be calculated through input. Here, the present value may mean a Q-Value of each timeslot and a transmission channel when only the current determination time is considered.

)로 구성될 수 있으며, 여기서 Deep Q-Network(125-1 내지 125-

)의 수는 최대 허용 사용자 단말 수

와 동일할 수 있다.Specifically, the present value calculation unit 125 is a plurality of Deep Q-Network (125-1 to 125-) as shown in FIG.

), where Deep Q-Network (125-1 to 125-

) is the maximum number of allowed user terminals.

can be the same as

)의 개수를 조정할 수 있다. 결정시간의 크기는 활성 사용자 단말의 수에 따라 달라질 수 있으므로, 기존의 Q-Value 산출 방법으로는 패킷전송에 대한 현재가치를 산출하는 것에 어려움이 있다.In addition, the present value calculation unit 125 is a Deep Q-Network (125-1 to 125-) used to calculate the present value according to the determination time.

) can be adjusted. Since the size of the decision time may vary depending on the number of active user terminals, it is difficult to calculate the present value for packet transmission using the existing Q-Value calculation method.

만큼의 Deep Q-Network(125-1 내지 125-

)로 구성되어, 활성 사용자 단말 수를 고려한 현재가치를 산출함으로써 현재가치 산출의 정확도를 향상할 수 있으며, 현재 활성 사용자 단말 수에 한하여 인공지능을 활용함으로써 패킷전송 결정 장치(121)의 향상된 효율을 기대할 수 있다.Therefore, the present value calculation unit 125 is at least the maximum number of allowable user terminals.

Deep Q-Network (125-1 to 125-

), the accuracy of the present value calculation can be improved by calculating the present value considering the number of active user terminals, and the improved efficiency of the packet transmission determining device 121 is improved by using artificial intelligence only for the number of currently active user terminals. can be expected

)는 활성 사용자 단말 수의 범위인

로 정의될 수 있으며, 각 Deep Q-Network(125-1 내지 125-

)는 패킷전송의 현재가치를 산출하는 어드벤티지 행동 브렌치(Advantage Action Branch) 및 산출된 패킷전송의 현재가치를 정규화하는 정규화 행동 브렌치(Normalization Action Branch)로 구성될 수 있다.Specifically, a plurality of Deep Q-Networks (125-1 to 125-) included in the present value calculation unit 125

) is the range of the number of active user terminals.

It can be defined as, and each Deep Q-Network (125-1 to 125-

) may be composed of an advantage action branch that calculates the present value of packet transmission and a normalization action branch that normalizes the calculated present value of packet transmission.

를 입력 받아, j번째 행동정보

의 현재가치인

를 산출할 수 있다.The advantage behavior branch of the j-th present value calculator (that is, the j-th Deep Q-Network) is the redefined state information

receives the input, the j-th behavior information

is the present value of

can be calculated.

,

,

, ?? ,

,

,

를 j번째 행동정보

에 대해 정규화하여, 최종적으로 정규화된 패킷전송의 현재가치

를 산출할 수 있다.After that, the normalized behavior branch of the j-th present value calculation unit is the present value for each j-th behavior information.

,

,

, ?? ,

,

,

is the j-th behavior information

By normalizing for , the present value of the finally normalized packet transmission

can be calculated.

를 입력하여 패킷전송의 미래가치를 산출할 수 있다. 여기서, 미래가치는 현재가치 산출부(125)에서 산출되는 패킷전송의 현재가치 이후의 패킷전송을 통해 획득할 수 있는 잠재적인 가치를 의미하는 것으로, 현재가치 산출부(125)에서 현재가치가 산출된 결정시간 이후의 패킷전송에 대한 가치를 의미할 수 있다.The future value calculation unit 126 provides state information redefined in artificial intelligence prepared in advance.

can be input to calculate the future value of packet transmission. Here, the future value means a potential value that can be obtained through packet transmission after the present value of packet transmission calculated by the present value calculation unit 125, and the present value is calculated by the present value calculation unit 125. It can mean the value of packet transmission after the determined decision time.

To describe the future value in more detail with reference to FIG. 2 , the present value calculating unit 125 performs a packet after the decision time i=2 (more specifically, the interval between the decision time i=2 and the decision time i=3) If the present value for transmission is calculated, then the future value can be the value for packet transmission after the decision time i=3.

Even if the present value of the current packet transmission is calculated as a high value, the future value of the packet transmission can be calculated as a low value. can

Accordingly, the value calculating unit 124 may calculate the final Q-Value by comprehensively considering the present and future values of packet transmission.

를 산출할 수 있다. 다만, 앞서 언급했던 것처럼 상태정보

는 결정시간 i마다 크기가 다르기 때문에, 미래가치 산출부(126)는 상태정보

를 대체하는 고정된 크기를 갖는 재정의된 상태정보

를 적용한 미래가치

를 산출할 수 있다.Specifically, the future value calculation unit 126 calculates the future value of packet transmission separately from the present value of packet transmission calculated by the present value calculation unit 125 .

can be calculated. However, as mentioned above, status information

Since the size of is different for each determination time i, the future value calculation unit 126 displays the state information

Overridden state information with a fixed size that replaces

future value applied

can be calculated.

The Q-Value calculator 127 may calculate the Q-Value for each timeslot and transmission channel within the determined time by adding up the present value of packet transmission and the future value of packet transmission.

와 미래가치 산출부(126)에서 산출된 패킷전송의 미래가치

의 조합으로 표현되는 Q-Value를 산출할 수 있고,

와 같이 표현할 수 있다.Specifically, the Q-Value calculator 127 calculates the present value of packet transmission calculated by the present value calculator 125 .

and the future value of packet transmission calculated by the future value calculation unit 126

It is possible to calculate the Q-Value expressed by the combination of

can be expressed as

As described above, the Q-Value calculator 127 may calculate a Q-Value for each timeslot and each transmission channel within the determination time, and an example thereof is shown in FIG. 7A .

The determination unit 128 may determine a transmission channel usable in each timeslot within the next determination time based on the Q-Value for each time slot and each transmission channel within the determination time calculated by the Q-Value calculator 127. have.

The determination unit 128 determines the largest Q- It is possible to select a value and determine the transmission channel N in the timeslot T including the corresponding Q-Value as the transmission channel in the timeslot T within the next determination time.

Afterwards, when the largest Q-Value in the N x T matrix is determined as the transmission channel, the determiner 128 selects the largest Q-Value again from the N x T-1 matrix except for the matrix including the corresponding Q-Value, , it is possible to determine the transport channel N in the timeslot T including the corresponding Q-Value as the transport channel in the timeslot T within the next determination time.

However, the determination unit 128 may repeat the process of determining the transmission channel in the timeslot including the largest Q-Value as the transmission channel of the next determination time as described above by the number of transmission channels N.

가 패킷전송을 시도할 수 있는 전송 채널의 수 N이 주어지면, 각 활성 사용자 단말은

확률로 패킷을 전송할 수 있기 때문에, 다음 결정시간의 전송 채널로 결정되는 수는 주어진 전송 채널의 수 N과 같을 수 있다.Specifically, the number of active user terminals during the determination time i in the process of determining the transmission channel

Given the number of transport channels N can attempt to transmit a packet, each active user terminal

Since the packet can be transmitted with a probability, the number determined as the transmission channel at the next decision time may be equal to the number N of the given transmission channel.

In addition, the determination unit 128 applies zero padding to the remaining timeslots for which the Q-Value is not selected, and treats the transmission channel of the corresponding timeslot as 0, thereby determining the size of the transmission channel set at the next determination time and the determination time. can fit the size.

Finally, the determiner 128 may determine a packet transmission schedule in the form of a 1 x T matrix in which the transmission channel for each timeslot is determined within the next determination time (ie, the transmission channel set at the next determination time mentioned above).

In addition, when the determination unit 128 determines the packet transmission schedule for the next determination time, the active user terminals transmit the packet according to the packet transmission schedule at the next determination time, and when the packet transmission is completed, the determination unit 128 again The process of determining the packet transmission schedule for the next determined time can be repeated and updated.

The operation of the determination unit will be described once again with reference to FIG. 7 showing an embodiment of the packet transmission schedule determination process of the present invention.

As shown in (a) of FIG. 7 , the decision unit has the largest Q based on the Q-Value of the 2×5 matrix at the decision time composed of the number of timeslots (T=5) and the number of transmission channels (N=2). -Value(5.7) is selected, and the transmission channel 1 of timeslot 1 including the corresponding Q-Value(5.7) can be determined as the transmission channel of timeslot 1 within the next determination time.

In addition, as shown in (b) of FIG. 7 , when the determination unit determines that the largest Q-Value (5.7) in the N x T matrix is the transmission channel, the determination unit determines the matrix (1,1) including the corresponding Q-Value (5.7). ), selects the largest Q-Value (4.3) again from the 2 x 4 matrix, and sets the transmission channel 2 of timeslot 4 including the corresponding Q-Value (4.3) to the transmission channel in timeslot 4 within the next decision time. can be decided with

Thereafter, as shown in (c) of FIG. 7 , the determination unit applies zero padding to the remaining timeslots 2, 3, 5 in which the Q-Value is not selected, and treats the transmission channel of the corresponding timeslot as 0. The size of the transmission channel set of the decision time and the size of the decision time can be matched.

Finally, as shown in FIG. 7(c), the decision unit can determine the packet transmission schedule in the form of a 1 x 5 matrix in which the transmission channel for each timeslot is determined within the next determination time (that is, the transmission channel set at the next determination time mentioned above). have.

That is, the active user terminal may transmit a packet through transport channel 1 in timeslot 1 included in the next determination time, and may transmit a packet through transport channel 2 in timeslot 4, and thereafter, in timeslots 2, 3 and 5 It may mean that the packet is not transmitted.

8 is a flowchart illustrating a method of determining a packet transmission schedule using the packet transmission determining apparatus according to an embodiment of the present invention.

Referring to FIG. 8 , a method for determining a packet transmission schedule using a packet transmission determining device includes a pre-processing step of generating status information (S100), a data management step of generating redefined status information (S200), present value and future It may include a value calculation step (S300) of calculating a value, a Q-Value calculation step (S400) of calculating a Q-Value, and a decision step (S500) of determining a packet transmission schedule.

In the pre-processing step ( S100 ), the pre-processing unit may generate state information for each determination time based on the packet transmission behavior information and the feedback information corresponding to the packet transmission.

In the data management step (S200), the data management unit may generate the redefined state information by using the state information and the accumulated state information.

In the value calculation step ( S300 ), the present value of packet transmission and the future value of packet transmission may be calculated using the state information redefined by the value calculation unit.

In the Q-Value calculation step ( S400 ), the Q-Value calculator may calculate a Q-Value for packet transmission by adding up the present value of packet transmission and future value of packet transmission.

In the decision step S500, the decision unit may determine the packet transmission schedule at the next decision time based on the Q-Value.

The method of determining the packet transmission schedule using the packet transmission determining device has been described in detail with reference to FIGS. 1 to 7 above, and thus will be briefly described as above.

9A is a diagram illustrating packet transmission fairness between active user terminals over time according to an embodiment of the present invention.

, 전송 채널

인 경우, 본 발명을 시간경과에 따라 실시함으로써 얻어지는 패킷전송 공정성을 확인할 수 있다.Referring to Figure 9a, the maximum number of allowed user terminals

, transmission channel

In the case of , the fairness of packet transmission obtained by implementing the present invention over time can be confirmed.

In the ideal case according to the above conditions, each active user terminal transmits a packet with a probability of 2/5, and thus, if all packets are successfully transmitted, a transmission rate of 2/5 can be obtained, so that all active user terminals can fairly transmit packets. can be transmitted.

If the present invention is implemented over time, it can be seen that the data rate of each active user is uneven and fairness is low until about timeslot 50, but through continuous implementation after timeslot 50, the data rate of all active users is the target data rate 2/ As it becomes constant like 5(0.4), it is possible to maintain the fairness of packet transmission.

9B is a diagram illustrating a data rate and a target data rate for each active user terminal according to an embodiment of the present invention.

, 전송 채널

인 경우의 활성 사용자 단말 별 전송률과 목표 전송률을 확인할 수 있다.9b (a) is the maximum number of allowed user terminals

, transmission channel

In the case of , it is possible to check the data rate and the target data rate for each active user terminal.

, 전송 채널

인 경우의 활성 사용자 단말 별 전송률과 목표 전송률을 확인할 수 있다.9b (b) is the maximum number of allowed user terminals

, transmission channel

In the case of , it is possible to check the data rate and the target data rate for each active user terminal.

Referring to (a) and (b) of FIG. 9B , it can be seen that the target data rate for each active user is different because the number of active users accessing in real time varies according to the lapse of time.

However, even in a random access communication environment in which the number of connected active users varies by implementing the present invention, it is possible to obtain a data rate close to the target data rate for each active user.

The above description is merely illustrative of the technical idea, and various modifications, changes and substitutions will be possible without departing from the essential characteristics by those skilled in the art of the present invention. Accordingly, the embodiments disclosed above and the accompanying drawings are for explanation rather than limiting the technical idea, and the scope of the technical idea is not limited by these embodiments and the accompanying drawings. The scope of protection should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights.

1: Random Access Communication System
10: user terminal
20: repeater
30: packet transmission schedule
110: communication department
120: process
121: packet transmission decision device
122: preprocessor
123: data management unit
124: value calculation unit
125: present value calculator
126: Future value calculation unit
127: Q-Value calculator
128: decision part
130: storage

Claims (18)

a pre-processing unit for generating status information at each decision time based on the packet transmission behavior information and the packet transmission feedback information;
a data management unit for generating redefined state information by using the state information and the accumulated state information;
a value calculating unit for calculating a present value of the packet transmission and a future value of the packet transmission by using the redefined state information;
a Q-Value calculator for calculating a Q-Value for the packet transmission by adding up the present value and the future value; and
a determination unit that determines a packet transmission schedule at a next determination time based on the Q-Value;
Packet transmission decision device comprising a. According to claim 1,
The value calculation unit,
A packet transmission determining device for calculating the present value and the future value by inputting the redefined state information into a pre-prepared Deep Q-Network. 3. The method of claim 2,
The value calculation unit,
a present value calculator comprising a plurality of the Deep Q-Networks corresponding to the maximum number of allowed user terminals;
Packet transmission decision device comprising a. 4. The method of claim 3,
The present value calculation unit,
The apparatus for determining packet transmission, characterized in that the present value is calculated by using only the number of the Deep Q-Networks corresponding to the size of the determination time among the plurality of Deep Q-Networks. 5. The method of claim 4,
The present value calculation unit,
The apparatus for determining packet transmission, characterized in that normalizing the present values calculated by the plurality of deep Q-Networks. According to claim 1,
The value calculation unit,
a future value calculator for calculating the future value, which is a value of packet transmission after the determined time at which the present value is determined, by using the redefined state information;
Packet transmission decision device comprising a. According to claim 1,
The determining unit is
A packet transmission determining apparatus for determining a transmission channel corresponding to a timeslot within the determination time for which the highest Q-Value is calculated among the Q-Values as a packet transmission schedule at the next determination time. 8. The method of claim 7,
The determining unit is
and applying zero-padding to the timeslot not determined by the packet transmission schedule to match the size of the packet transmission schedule and the size of the determination time. According to claim 1,
The data management unit,
The apparatus for determining packet transmission, characterized in that time series learning and aggregation of the state information and the redefined state information according to the elapse of the determination time by using a Long Short-Term Memory (LSTM) model. a pre-processing step of generating, by the pre-processing unit, status information at each decision time based on the behavior information of packet transmission and the feedback information received through packet transmission;
a data management step of generating, by a data management unit, the redefined state information using the state information and the accumulated state information;
a value calculation step of calculating, by a value calculation unit, a present value of the packet transmission and a future value of the packet transmission by using the redefined state information;
a Q-Value calculation step in which a Q-Value calculation unit calculates a Q-Value for the packet transmission by summing the present value and the future value; and
a decision step in which a decision unit determines a packet transmission schedule at a next decision time based on the Q-Value;
Packet transmission schedule determination method comprising a. 11. The method of claim 10,
The value calculation step is
A method for determining a packet transmission schedule for calculating the present value and the future value by inputting the redefined state information into a pre-prepared Deep Q-Network. 12. The method of claim 11,
The value calculation step is
a present value calculation step of configuring a present value calculation unit with a plurality of the Deep Q-Networks corresponding to the maximum allowed number of user terminals;
Packet transmission schedule determination method comprising a. 13. The method of claim 12,
The present value calculation step is
Method for determining the packet transmission schedule, characterized in that the present value is calculated by using only the number of the Deep Q-Networks corresponding to the size of the determination time among the plurality of Deep Q-Networks. 14. The method of claim 13,
The present value calculation step is
A method for determining a packet transmission schedule, characterized in that normalizing the present values calculated by the plurality of deep Q-Networks. 11. The method of claim 10,
The value calculation step is
a future value calculation step of calculating the future value, which is a value of packet transmission after the determination time at which the present value is determined, using the redefined state information;
Packet transmission schedule determination method comprising a. 11. The method of claim 10,
The decision step is
A method for determining a packet transmission schedule for determining a transmission channel corresponding to a timeslot within the determination time for which the highest Q-Value is calculated among the Q-Values as the packet transmission schedule at the next determination time. 17. The method of claim 16,
The decision step is
and applying zero-padding to the timeslot that is not determined by the packet transmission schedule to match the size of the packet transmission schedule with the size of the determination time. 11. The method of claim 10,
The data management step is
A method for determining a packet transmission schedule, characterized in that time series learning and aggregation of the state information and the redefined state information according to the elapse of the determination time using a Long Short-Term Memory (LSTM) model.

Images