KR102508071B1 - Beam forming training method and snr prediction method for beam forming training in wlan system - Google Patents

Abstract

Disclosed are a method for the beam forming training based on reinforced learning in a wireless LAN system and a device thereof. The disclosed method for the beam forming training based on reinforced learning comprises: a step of receiving SISO feedback information from a terminal; and a step of determining a current transmission sector group used for transmitting an action frame by using the pre-trained reinforced learning model. A status group of the reinforced learning model includes: the SISO feedback information, and a previous transmission sector group used for transmitting the action frame in a previous MIMO step. The action group of the reinforced learning model includes: a participation information of the terminal for a current MIMO step, and an update information for the current transmission sector group. Therefore, the time required for the beam forming training can be reduced.

Description

Reinforcement learning-based beamforming training method and apparatus

The present invention relates to a beamforming training method and apparatus in a wireless LAN system, and more particularly, to a reinforcement learning-based beamforming training method and apparatus.

According to the IEEE 802.11 standard, MU-MIMO beamforming training is performed in SISO and MIMO stages.

In the SISO step, the access point transmits a short sector sweep frame for each transmission sector and receives SISO feedback information from the terminal. The SISO feedback information includes an SNR value measured by the terminal with respect to the short sector sweep frame transmitted for each transmission sector.

In addition, the MIMO step includes a BF setup substep, a BF selection substep, a BF training substep, and a BF feedback substep. In each sub-step, the access point transmits an action frame for beamforming training to the terminal.

As related prior literature, there are Republic of Korea Patent Registration Nos. 10-2288198, 10-2154481, 10-2153923, and Republic of Korea Patent Publication No. 2022-0036494.

An object of the present invention is to provide a beamforming training method for allocating transmission sectors to phased array antennas so that action frames for beamforming training can be efficiently transmitted.

According to one embodiment of the present invention for achieving the above object, receiving SISO feedback information from the terminal; and determining a current transmission sector set used for transmission of an action frame by using a pre-learned reinforcement learning model, wherein the state set of the reinforcement learning model includes the SISO feedback information and an action frame in a previous MIMO step. A reinforcement learning-based beam including a previous transmission sector set used for transmission of , and an action set of the reinforcement learning model including information on participation of the terminal in a current MIMO step and update information on the current transmission sector set. A forming training method is provided.

In addition, according to another embodiment of the present invention for achieving the above object, a phased array antenna for receiving SISO feedback information from a terminal and transmitting an action frame; Memory; and at least one processor electrically connected to the memory, wherein the processor determines a current transmission sector set used for transmission of the action frame by using a pre-learned reinforcement learning model, and determines a state of the reinforcement learning model The set includes the SISO feedback information and a previous transmit sector set used for transmission of an action frame in a previous MIMO step, and the action set of the reinforcement learning model includes the UE's participation information for the current MIMO step and the current transmit sector set. An access point including update information about is provided.

According to an embodiment of the present invention, unnecessary transmission of action frames for beamforming training can be reduced and beamforming training time can be reduced by using a reinforcement learning model.

Also, according to an embodiment of the present invention, as many terminals as possible can participate in beamforming training.

1 is a diagram for explaining a general beamforming training method.
2 is a diagram for explaining a beamforming training method in a WLAN system according to an embodiment of the present invention.
3 is a diagram for explaining an SNR prediction method for beamforming training in a WLAN system according to an embodiment of the present invention.
4 is a diagram for explaining an artificial neural network according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a general beamforming training method in a WLAN system. In FIG. 1, there are 4 transmission sectors (TS1 to TS4), 2 phased array antennas of an access point, and a first phased array An example in which the first and second transmission sectors TS1 and TS2 are allocated to the antenna 110 and the third and fourth transmission sectors TS3 and TS4 are allocated to the second phased array antenna 120 is illustrated.

The access point transmits a short sector frame for each transmit sector in the SISO step. The first phased array antenna 110 transmits a short sector frame to the first and second transmission sectors TS1 and TS2, and the second phased array antenna 120 transmits a short sector frame to the third and fourth transmission sectors TS3 and TS4. Transmits a short sector frame with A terminal connected to the access point receives the short sector frame, measures the SNR, and transmits SISO feedback information including the measured SNR value to the access point. That is, one wireless terminal transmits SISO feedback information including four SNR values measured for the first to fourth transmission sectors (TS1 to TS4) to the access point. For example, the wireless terminal receives the short sector sweep frame transmitted to the first transmission sector TS1, generates an SNR value for the first transmission sector TS1, and transmits the short sector sweep frame to the second transmission sector TS2. is received to generate an SNR value for the second transmission sector TS2.

The access point performs beamforming training in the MIMO step using the measured SNR value. The access point performs beamforming training by transmitting action frames in each of the BF setup sub-step, BF selection sub-step, BF training sub-step, and BF feedback sub-step.

In the MIMO step, the access point transmits an action frame so that all terminals accessing the access point can receive it. can transmit The transmission sector combination may be determined by selecting one transmission sector assigned to each phased array antenna. In the example of FIG. 1, four transmission sector combinations may be determined, such as (TS1, TS3), (TS1, TS4), (TS2, TS3) (TS2, TS4), and the access point may select one of these transmission sector combinations By selecting one, an action frame may be transmitted to a plurality of transmission sectors included in the selected transmission sector combination.

Transmitting the action frame with all transmission sector combinations increases the beamforming training time and is inefficient. Therefore, it is important to determine an efficient transmission sector combination that can minimize the number of transmissions of action frames while all terminals receive the action frames. need. Accordingly, the present invention proposes a method of determining an optimal transmission sector combination using reinforcement learning and performing beamforming training therethrough.

A beamforming training method according to an embodiment of the present invention is performed in an access point of a wireless LAN system.

2 is a diagram for explaining a beamforming training method in a WLAN system according to an embodiment of the present invention, and FIG. 3 is a diagram showing a reinforcement learning model according to an embodiment of the present invention.

Referring to FIG. 2, the access point according to an embodiment of the present invention receives SISO feedback information from the terminal (S210), and uses a pre-learned reinforcement learning model to determine a current transmission sector set used for transmission of an action frame. It is determined (S220). Here, the transmission sector set means a set including a transmission sector combination used for transmission of an action frame among transmission sectors allocated to each phased array antenna.

As an example, the reinforcement learning model may be the actor-critic model of FIG. 3 . In the reinforcement learning model, an agent (RL Agent) corresponds to an access point, and an environment (Environment) corresponds to beamforming training. And agents can be composed of actors and critiques. The agent, that is, the access point, observes the state of the current beamforming environment and performs an action that maximizes the reward in the observed state. Here, the action corresponds to determining the current transmit sector set, and the actor and critique may be updated through a Temporal Difference (TD) error.

The state set of the reinforcement learning model includes SISO feedback information and a previous transmission sector set used for transmission of an action frame in a previous MIMO step, and the action set of the reinforcement learning model includes UE participation information for a current MIMO step and a current set of transmission sectors. It includes update information about the transmit sector set. Here, the previous transmission sector set refers to a transmission sector set used for transmission of an action frame before determining the current transmission sector set in step S220. The participation information is information determined according to the SNR value included in the SISO feedback information, and a terminal that transmits an SNR value equal to or greater than a preset threshold participates in the MIMO step.

Hereinafter, a method of determining a current transmission sector set for each of the BF setup sub-step, BF selection sub-step, BF training sub-step, and BF feedback sub-step constituting the MIMO step will be described in detail.

BF setup substep, BF select substep

In the BF setup sub-step and the BF selection sub-step, the access point determines the current transmit sector set using the reinforcement learning model described above.

When the number of terminals participating in beamforming training is N, and the previous transmission sector set and SISO feedback information included in the state set are expressed as T _N and F _N , respectively, the previous transmission sector set is expressed as in [Equation 1] It can be.

는 i번째 액션 프레임을 전송할 때 이용된 송신 섹터 조합을 나타낸다. 빔포밍 훈련에 참여하는 단말의 개수가 N개이기 때문에, 액션 프레임은 최대 N회 전송되며, 송신 섹터 조합의 최대 개수는 N개이다.here,

represents a transmission sector combination used when transmitting the i-th action frame. Since the number of terminals participating in beamforming training is N, action frames are transmitted up to N times, and the maximum number of transmit sector combinations is N.

는 j번째 위상 배열 안테나에 할당된 송신 섹터를 나타낸다.And when the number of phased array antennas is M, the transmission sector combination used for transmission of the i-th action frame can be expressed as in [Equation 2],

denotes a transmission sector allocated to the j-th phased array antenna.

는, k번째 단말로부터 수신된 SISO 피드백 정보를 나타낸다. SISO feedback information can be expressed as [Equation 3], where

Represents SISO feedback information received from the k-th terminal.

는 [수학식 4]와 같이 표현될 수 있으며,

은 송신 섹터(TS)의 총 개수를 L이라고 할 때, SISO 단계에서 l번째 송신 섹터로 전송된 짧은 섹터 스윕 프레임에 대한 SNR값을 나타낸다. 즉, 단말은 모든 송신 섹터 각각에 대한 SNR값을 포함하는 SISO 피드백 정보를 액세스 포인트로 전송한다.and

Can be expressed as [Equation 4],

denotes an SNR value for a short sector sweep frame transmitted to the l-th transmission sector in the SISO step, when the total number of transmission sectors (TS) is L. That is, the terminal transmits SISO feedback information including SNR values for all transmission sectors to the access point.

When the participation information and update information included in the action set are P _N , e, respectively, the participation information can be expressed as in [Equation 5], and when the k-th terminal participates in the current MIMO step, p _k is 1, otherwise p _k may be set to 0.

If there is one or more SNR values equal to or greater than the preset first threshold among the SNR values transmitted by the UE, it is determined that the corresponding UE participates in the current MIMO step.

As an example, the update information may include first to fourth information, and the access point determines a current transmission sector set according to the update information. First, first information (e=1) is information for adding at least one new transmission sector combination to a previous transmission sector set. Accordingly, in this case, the number of transmissions of action frames increases. At this time, the access point adds at least one transmit sector combination to which an action frame is additionally transmitted among transmit sector combinations other than transmit sector combinations included in the previous transmit sector set to the previous transmit sector set, thereby obtaining the current transmit sector set. Decide.

The access point determines a transmission sector combination to additionally transmit an action frame among transmission sectors including terminals that have not received an action frame in the BF setup sub-step and the BF selection sub-step of the previous beamforming training, and determines the transmission sector combination to the previous transmission sector set. Add. Whether or not the terminal has received the action frame can be confirmed through ACK information transmitted by the terminal. At this time, the access point determines a transmission sector combination to additionally transmit an action frame such that an SNR value greater than or equal to a preset second threshold is maximized among SNR values transmitted by terminals that have not received the action frame.

In other words, the access point transmits the previous transmit sector combination, in which the number of UEs having an SNR value of SISO feedback information equal to or greater than the second threshold is the maximum among UEs participating in the current MIMO step and not receiving an action frame in the previous MIMO step. In addition to the sector set, determine the current transmit sector set.

If there is no SNR value equal to or greater than the second threshold value, the access point determines a transmission sector combination to additionally transmit an action frame such that a transmission sector including a terminal having transmitted the maximum SNR value is included. That is, when the access point currently participates in the MIMO phase and does not have a terminal having a value higher than the second threshold value among the terminals that have not received an action frame in the previous MIMO phase, the transmission sector combination including the terminal corresponding to the maximum SNR value exists. is added to the previous transmitted sector set to determine the current transmitted sector set.

The second information (e=2) is information for removing at least one transmission sector combination from among transmission sector combinations included in a previous transmission sector set. Accordingly, in this case, the number of transmissions of action frames is reduced. When the action frame is transmitted to each of the transmission sector combinations included in the previous transmission sector set, the access point may remove the transmission sector combination having the minimum number of terminals that have received the action frame from the previous transmission sector set.

The third information (e=3) is information for changing at least one of the transmission sector combinations included in the previous transmission sector set to a new transmission sector combination. Accordingly, in this case, the number of transmit sector combinations included in the previous transmit sector set is equal to the number of transmit sector combinations included in the current transmit sector set. The access point uses a method of determining a transmission sector combination to be removed when the update information is the second information, and a method of determining a transmission sector combination to be added when the update information is the first information. to add a transmit sector combination.

Fourth information (e=4) is information using a previous transmission sector set as a current transmission sector set. That is, in this case, the access point determines the previous transmission sector set as the current transmission sector set and transmits the action frame.

)은 [수학식 6]과 같이 결정될 수 있다.As described above, the access point may determine one of the update information to maximize compensation, and determine a current transmit sector set according to the determined update information. And at this time, compensation (

) can be determined as in [Equation 6].

Here, N _S represents the number of terminals that have received at least one action frame in the BF setup sub-step, and N _t indicates the number of transmissions of action frames in the BF setup sub-step.

BF training sub-steps

In the BF training sub-step, the access point identifies a terminal capable of receiving an action frame from among terminals included in the transmit sector combination of the current transmit sector set, and determines the transmit sector combination.

The access point determines that the terminal included in the transmission sector combination of the previous transmission sector set can receive the action frame. And when at least one of the SNR values transmitted from the terminal included in the newly added transmission sector combination among the transmission sector combinations of the current transmission sector set is a preset second threshold value It is determined that the reception of the action frame is possible.

The access point may update the current transmission sector set so that a transmission sector combination including a terminal requiring beamforming training in a lower BF training step among terminals capable of receiving an action frame is included in the current transmission sector set.

BF feedback sub-step

In the BF feedback substep, the access point transmits an action frame to a transmission sector including a terminal that has transmitted the maximum SNR value, regardless of the determined current transmission sector set.

4 is a diagram for explaining a method for updating a reinforcement learning model according to an embodiment of the present invention.

An access point according to an embodiment of the present invention performs an update on a value function and a policy function deep learning network of a reinforcement learning model. That is, the access point updates the deep learning network by learning the reinforcement learning model using the collected data while determining the current transmission sector set using the reinforcement learning model.

In the reinforcement learning model according to an embodiment of the present invention, an actor of an agent performs an action, that is, determines a set of current transmission sectors by using a policy function. Then, the critique (critic) of the agent uses the value function to evaluate whether the action of the actor, that is, the determination of the current transmit sector set, is appropriate. The policy function and the value function may be implemented as a deep learning network, and the evaluation result of the value function is used to update parameters (weights) of the deep learning policy function network. In the value function deep learning network, parameters are updated according to the result of performing the action, that is, the reward.

Referring to FIG. 4 , the access point acquires output values of the policy function and the value function by using a deep learning network for the policy function and the value function (S410).

)는 특정 상태(S)에서 특정 행동(A)이 결정될 확률을 계산하기 위한 함수로서, 여기서

는 정책 함수 딥러닝 네트워크의 가중치를 나타낸다. 정책 함수 딥러닝 네트워크는 상태 정보를 입력받아, 입력된 상태 정보에 대해서 특정 행동을 결정할 확률에 대한, 평균(

)과 표준 편차(

)를 출력하도록 학습된 네트워크이다. 액세스 포인트는 정책 함수 딥러닝 네트워크의 출력값으로부터 정규 분포(

)를 생성하여, 특정 행동을 결정할 확률을 계산할 수 있다. 예컨대, 전술된 업데이트 정보 중 가장 높은 확률에 대응되는 정보에 따라서, 액세스 포인트는, 현재 송신 섹터 집합을 결정할 수 있다.policy function (

) is a function for calculating the probability that a specific action (A) is determined in a specific state (S), where

represents the weight of the policy function deep learning network. The policy function deep learning network receives state information and averages (

) and standard deviation (

) is a network trained to output The access point is normally distributed from the output of the policy function deep learning network (

), we can calculate the probability of deciding on a particular action. For example, according to information corresponding to the highest probability among the aforementioned update information, the access point may determine a current transmission sector set.

)는 입력된 상태에 대한 보상의 기대값을 계산하기 위한 함수이며, 여기서

는 가치 함수 딥러닝 네트워크의 가중치를 나타낸다. 가치 함수 딥러닝 네트워크는 상태 정보를 입력받아, 입력된 상태 정보에 대한 보상 기대값을 출력하도록 학습된 네트워크이다. value function (

) is a function for calculating the expected value of the reward for the input state, where

represents the weights of the value function deep learning network. A value function deep learning network is a network trained to receive state information and output an expected reward value for the input state information.

The access point applies the output values of the policy function and the value function deep learning network to the loss function, and updates the weights of the policy function and the value function deep learning network so that the loss value of the loss function is minimized (S420).

라고 할 때,

의 실제 값에 대한 추정치 TD 타겟(Temporal Difference target,

)은 [수학식 7]을 통해 계산될 수 있다.As an example, the access point may update the weights of the policy function and the value function deep learning network using temporal difference learning. The state at time t is S _t and the reward at that time is

When you say

An estimate of the actual value of TD target (Temporal Difference target,

) can be calculated through [Equation 7].

은 감가율(Discount Factor)을 나타낸다.here,

represents the discount factor.

)이 최소가 되도록 업데이트될 수 있다. The value function deep learning network is a TD error value calculated as in [Equation 8] (

) can be updated to be minimal.

And, the policy function deep learning network can be updated so that the loss value of the loss function expressed as in [Equation 9] is minimized.

5 is a diagram for explaining a reinforcement learning-based beamforming training apparatus according to an embodiment of the present invention. In FIG. 5, an access point is shown as an embodiment.

Referring to FIG. 5 , an access point according to an embodiment of the present invention includes a phased array antenna 510 , a memory 520 and at least one processor 530 .

The phased array antenna 510 receives SISO feedback information from the terminal and transmits an action frame.

The processor 530 is electrically connected to the memory 520 and determines a current transmission sector set used for transmission of an action frame by using a pre-learned reinforcement learning model. The reinforcement learning model is the above-mentioned reinforcement learning model, wherein the state set of the reinforcement learning model includes SISO feedback information and a previous transmission sector set used for transmission of action frames in the previous MIMO step, and the action set of the reinforcement learning model is the current MIMO action set. and information about the UE's participation in the step and update information about the current transmission sector set.

The technical contents described above may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiments or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. A hardware device may be configured to act as one or more software modules to perform the operations of the embodiments and vice versa.

As described above, the present invention has been described by specific details such as specific components and limited embodiments and drawings, but these are provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , Those skilled in the art in the field to which the present invention belongs can make various modifications and variations from these descriptions. Therefore, the spirit of the present invention should not be limited to the described embodiments, and it will be said that not only the claims to be described later, but also all modifications equivalent or equivalent to these claims belong to the scope of the present invention. .

Claims (12)

Receiving SISO feedback information from a terminal; and
Determining a current transmission sector set used for transmission of an action frame by using a pre-learned reinforcement learning model;
The state set of the reinforcement learning model is
It includes the SISO feedback information and a previous transmission sector set used for transmission of an action frame in a previous MIMO step,
The action set of the reinforcement learning model,
Includes participation information of the terminal for a current MIMO step and update information for the current transmission sector set,
The current and previous transmission sector set is
Among the transmission sectors assigned to each phased array antenna, a combination of transmission sectors used for transmission of the action frame is included,
The update information
Update information for the BF setup substep and BF selection substep,
first information for adding a new transmission sector combination to the previous transmission sector set;
second information for removing at least one transmission sector combination from among transmission sector combinations included in the previous transmission sector set;
third information for changing at least one of the transmission sector combinations included in the previous transmission sector set to a new transmission sector combination; and
At least one of fourth information using the previous transmission sector set as the current transmission sector set
Reinforcement learning-based beamforming training method.
According to claim 1,
The participation information
Information determined according to the SNR value included in the SISO feedback information
Reinforcement learning-based beamforming training method.
delete delete According to claim 1,
The step of determining the current transmit sector set is
Among the terminals participating in the current MIMO phase and not receiving the action frame in the previous MIMO phase, a transmission sector combination having the maximum number of terminals having an SNR value of the SISO feedback information equal to or greater than the threshold is assigned to the previous transmission sector set. In addition, determining the current transmission sector set
Reinforcement learning-based beamforming training method.
According to claim 5,
The step of determining the current transmit sector set is
Among the UEs participating in the current MIMO step and not receiving the action frame in the previous MIMO step, if there is no UE having a value greater than or equal to the threshold value, a transmission sector combination including a UE corresponding to the maximum SNR value is transmitted to the previous MIMO stage. In addition to the transmit sector set, determining the current transmit sector set
Reinforcement learning-based beamforming training method.
According to claim 1,
The step of determining the current transmit sector set is
In a situation where the action frame is transmitted for the previous transmission sector set, removing a transmission sector combination having the minimum number of terminals receiving the action frame from the previous transmission sector set
Reinforcement learning-based beamforming training method.
According to claim 1,
The step of determining the current transmit sector set is
In a situation in which the action frame is transmitted for the previous transmission sector set, determining a transmission sector combination having the minimum number of terminals receiving the action frame;
Among the UEs participating in the current MIMO step and not receiving the action frame in the previous MIMO step, the determined transmit sector set is a transmit sector combination in which the number of UEs having an SNR value of the SISO feedback information equal to or greater than the threshold value is the maximum. to change
Reinforcement learning-based beamforming training method.
According to claim 1,
Determining a transmission sector combination for a BF training sub-step by checking a terminal capable of receiving the action frame from among terminals included in the transmission sector combination of the current transmission sector set
Reinforcement learning-based beamforming training method further comprising a.
According to claim 1,
Updating the deep learning network for the value function and policy function of the reinforcement learning model.
Reinforcement learning-based beamforming training method further comprising a.
According to claim 10,
The step of performing an update on the deep learning network
obtaining output values of the value function and the policy function by using the deep learning network; and
Updating weights of the deep learning network by applying the output value to a loss function so that the loss value of the loss function is minimized.
Reinforcement learning-based beamforming training method comprising a.
a phased array antenna for receiving SISO feedback information from a terminal and transmitting an action frame;
Memory;
including at least one processor electrically connected to the memory;
The processor
Determine a current transmission sector set used for transmission of the action frame by using a pre-learned reinforcement learning model;
The state set of the reinforcement learning model is
It includes the SISO feedback information and a previous transmission sector set used for transmission of an action frame in a previous MIMO step,
The action set of the reinforcement learning model is
Includes participation information of the terminal for a current MIMO step and update information for the current transmission sector set,
The current and previous transmission sector set is
Among the transmission sectors assigned to each phased array antenna, a combination of transmission sectors used for transmission of the action frame is included,
The update information
Update information for the BF setup substep and BF selection substep,
first information for adding a new transmission sector combination to the previous transmission sector set;
second information for removing at least one transmission sector combination from among transmission sector combinations included in the previous transmission sector set;
third information for changing at least one of the transmission sector combinations included in the previous transmission sector set to a new transmission sector combination; and
At least one of fourth information using the previous transmission sector set as the current transmission sector set
access point.

Images