KR20190127480A - Method and apparatus for detremining transmisstion point and link adaptation scheme in communication system - Google Patents

Abstract

Disclosed is an operation method of a resource managing device connected with a plurality of transmission points in a communication system. According to the present invention, the operation method of a resource managing device comprises the following steps: receiving wireless channel state information and transmission point state information from a terminal; determining a transmission point providing a maximum data rate and a link adaptation scheme based on the wireless channel state information and the transmission point state information; and transmitting information on the transmission point providing the maximum data rate and information on the link adaptation scheme to the terminal.

Description

METHOD AND APPARATUS FOR DETREMINING TRANSMISSTION POINT AND LINK ADAPTATION SCHEME IN COMMUNICATION SYSTEM}

The present invention relates to a communication system, and more particularly to a method and apparatus for determining a transmission point and link adaptation scheme.

Long term evolution (LTE) systems have standardized technology to improve radio link quality and efficient reuse of frequency space by deploying small cells to meet bandwidth requirements for mobile communications. . However, unlike macro cells that are intentionally placed, small cells are difficult to predict position because they are randomly placed immediately. The traditional approach to managing the network through manual intervention and centralized control is time consuming and expensive, making it unsuitable for managing small cells with high batch density per area. The third generation project partnership (3GPP) has standardized a self organization network (SON) technology for small cell network management. In addition, 3GPP considers a transmission point network for transmitting high density data per unit area to satisfy a 1000 times improvement in data rate per unit area, which is one of the requirements of 5G systems.

SON, the standardization technology of 3GPP, has several functions designed independently. Accordingly, when a plurality of transmission points operate the SON function at the same time, it is possible to make mutually independent determinations by different SON functions. Therefore, a plurality of transmission points have a problem that performance degradation or link failure occurs due to setting parameters that are not allowed to each other.

3GPP has introduced a CoMP (Coordinated Multi Point) scheme for transmitting data to a terminal using a plurality of transmission points. However, at present, when there are a plurality of transmission points, a method of selecting a transmission point for performing data transmission to the terminal among the plurality of transmission points is lacking.

In the current mobile communication system, a link adaptation scheme is designed without considering a transmission point for transmitting a signal at a short distance, such as a small cell. In a frequency division duplex (FDD) system, a user equipment may report a channel quality indicator (CQI) to a macro base station by measuring a downlink radio channel. The macro base station may determine a modulation scheme and channel coding code rate suitable for a wireless channel situation based on the CQI received from the terminal. The macro base station may transmit a packet to the terminal according to a modulation channel coding scheme (MCS) modulation scheme. When an error is detected in a packet received from the macro base station, the terminal may perform a hybrid automatic repeat request (HARQ) operation. The macro base station may adapt the radio channel link in a manner to reduce packet error rate by performing packet retransmission according to HARQ. That is, the macro base station can adapt the radio channel link through MCS and HARQ. The conventional link adaptation scheme does not consider a drastic change in the radio channel environment due to dynamic resource allocation when there are transmission points in the local area of the macro base station. Therefore, the conventional link adaptation scheme has an inefficient problem when a plurality of transmission points quickly change the transmission scheme at a short distance of a macro base station.

In order to solve the above-mentioned problem, the present invention is to provide a network setting method in which a collision between parameters does not occur when each of a plurality of transmission points uses a SON function in a communication system. In addition, the present invention is to provide a method for selecting at least one transmission point for transmitting data to the terminal without collision between the parameters of the plurality of transmission points. In addition, the present invention is to provide a link adaptation scheme in consideration of a plurality of transmission points located in the near field of the macro base station.

In a communication system according to an embodiment of the present invention, a method of operating a resource manager connected to a plurality of transmission points includes: receiving wireless channel state information and transmission point state information from a terminal; Determining a transmission point and link adaptation scheme providing a maximum data rate based on the radio channel state information and the transmission point state information; And transmitting information on a transmission point providing the maximum data rate and information on a link adaptation scheme to the terminal. The transmission point and link adaptation scheme providing the maximum data rate is determined through a quality-learning scheme or a deep Q-network learning scheme.

The operation method of the resource manager may further include updating a quality (Q) value based on the radio channel state information and the transmission point state information.

The wireless channel state information includes channel state information (CSI) -reference signal (RS), interference measurement (CSI-IM), channel quality information (CQI), precoding matrix indicator (PMI), rank indication (RI), and interference state information. It may include at least one of. The transmission point state information may include at least one of a transmission rate of the transmission point and a handover rate of the transmission point. The Q value may indicate a maximum data rate determined based on the wireless channel state information and the respective transmission point state information.

The operation method of the resource manager may include an exploration that performs a resource management action such that the Q value is the highest value in a set of possible states and behaviors or an exploration that randomly behaves regardless of the Q value. Determining; And determining the transmission point according to the maximum Q value.

In the method of operating the resource manager, when the behavior mode is utilization, the channel state (CQI, RI, PMI) of the terminal and the load of the transmission point are left, and the Q value is updated to transmit the Q based on the Q value. The method may further include determining a handover transmission point that provides the largest Q value among the points.

The method of operation of the resource manager may include determining an action or exploration, and if the action is exploration, updating a Q value; And determining an arbitrary Modulation Channel Coding Scheme (MCS), Hybrid Automatic Repeat reQuest (HARQ), and an antenna transmission mode.

The operation method of the resource manager may further include determining an MCS, an HARQ, and an antenna transmission mode based on the maximum Q value when the action mode is utilization.

The operation method of the resource manager may include updating a weight value based on the radio channel state information and the transmission point state information such that a cost is minimized; And updating the Q value.

The operating method of the resource manager may include: updating the weight value to a value that minimizes cost; And if the action is exploration, determining any number of transmission points of the plurality of transmission points.

The operating method of the resource manager comprises: updating the weight value to a value that minimizes cost; And if the action is utilization, determining a transmission point that provides a maximum Q value among the plurality of transmission points.

The operating method of the resource manager comprises: updating the weight value to a value that minimizes cost; And if the action is exploration, determining any MCS, HARQ, and antenna transmission modes.

The operating method of the resource manager comprises: updating the weight value to a value that minimizes cost; And if the action is utilization, determining the MCS, HARQ, and antenna transmission mode providing the maximum Q value.

In addition, in the communication system according to an embodiment of the present invention, resource management connected to a plurality of transmission points includes: a processor; And a memory in which at least one command executed by the processor is stored, wherein the at least one command receives radio channel state information and transmission point state information from a terminal, and transmits the radio channel. Determine a transmission point and a link adaptation scheme providing a maximum data rate based on state information and the transmission point state information, and transmit information on the transmission point and the link adaptation scheme that provide the maximum data rate; To be sent to. The transmission point and link adaptation scheme providing the maximum data rate is determined through a quality-learning scheme or a deep Q-network learning scheme.

The at least one command may be further executed to update a quality (Q) value based on the wireless channel state information and the transmission point state information.

The wireless channel state information includes channel state information (CSI) -reference signal (RS), interference measurement (CSI-IM), channel quality information (CQI), precoding matrix indicator (PMI), rank indication (RI), and interference state information. It may include at least one of. The transmission point state information may include at least one of a transmission rate of the transmission point and a handover rate of the transmission point. The Q value may indicate a maximum data rate determined based on the wireless channel state information and the respective transmission point state information.

The at least one command determines whether an action is exploration or utilization, and if the action is utilization, selects the plurality of transmission points to a value whose Q value is the maximum value and performs a Q value update.

If the action is exploration, determine any number of transmission points of the plurality of transmission points. If the handover rate exceeds a predetermined threshold rate, it may be further executed to determine a transmission point that provides the largest transmission rate of the plurality of transmission points based on the Q value.

The at least one command determines if the action is exploration or utilization, if the action is exploration, determines any MCS, HARQ, and antenna transmission modes, and if the action is utilization, based on the Q value It may be further implemented to determine the MCS, HARQ, and antenna transmission modes that provide the maximum Q value.

The at least one command may be further executed to update a deep Q network weight value and update a Q value such that the cost is minimized based on the radio channel state information and the transmission point state information.

The at least one command determines whether the action is exploration or utilization, and if the action is exploration, determine any number of transmission points of the plurality of transmission points and minimize the cost of the deep Q network weight value. May be further updated to determine a transmission point providing the largest transmission rate among the plurality of transmission points based on the weight value when the handover rate exceeds a predetermined threshold rate.

The at least one command determines if the action is exploration or utilization, if the action is exploration, determines any MCS, HARQ, and antenna transmission modes, and updates the deep Q network weight value, The utilization may be further executed to determine the MCS, HARQ, and antenna transmission modes that provide the maximum Q value.

According to the present invention, it is possible to minimize the interference between the plurality of communication nodes through the link configuration operation according to the operation method between the plurality of communication nodes. In addition, according to the present invention, it is possible to minimize the interference between the plurality of communication nodes by transmitting and receiving data through a resource of a different resource allocation unit between the plurality of communication nodes. In addition, according to the present invention, it is possible to minimize the total interference between the plurality of communication nodes by averaging the interference power values by the signals transmitted and received to each other based on the interference information between the plurality of communication nodes.

1 is a conceptual diagram illustrating a communication system according to a first embodiment of the present invention.
2 is a block diagram showing a communication node according to the first embodiment of the present invention.
3 is a conceptual diagram illustrating an operation of a communication environment and a resource manager in a communication system according to an embodiment of the present invention.
4 is a conceptual diagram illustrating a deep Q-network learning operation in a communication system according to an embodiment of the present invention.
5 is a conceptual diagram illustrating a deep Q-network operation of updating weights in a communication system according to an embodiment of the present invention.
6 is a flowchart illustrating the operation of a resource manager in a communication system according to an embodiment of the present invention.
7 is a flowchart illustrating an operation of a Q-learning resource manager for determining a transmission point through Q-learning in a communication system according to an embodiment of the present invention.
8 is a flowchart illustrating an operation of a Q-learning resource manager for determining a link adaptation scheme through Q-learning in a communication system according to an embodiment of the present invention.
9 is a flowchart illustrating an operation of a deep Q-network learning resource manager for determining a transmission point through deep Q-network learning in a communication system according to an embodiment of the present invention.
10 is a flowchart illustrating an operation of a deep Q-network learning resource manager for determining a link adaptation scheme through deep Q-network learning in a communication system according to an exemplary embodiment of the present invention.

As the present invention allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and / or includes a combination of a plurality of related items or any item of a plurality of related items.

When a component is referred to as being "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that other components may be present in between. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art and shall not be construed in ideal or excessively formal meanings unless expressly defined in this application. Do not.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. In the following description of the present invention, the same reference numerals are used for the same elements in the drawings and redundant descriptions of the same elements will be omitted.

1 is a conceptual diagram illustrating a communication system according to a first embodiment of the present invention.

Referring to FIG. 1, the communication system 100 includes a plurality of communication nodes 110-1, 110-2, 110-3, 120-1, 120-2, 130-1, 130-2, 130-3, 130-4, 130-5, 130-6). Each of the plurality of communication nodes may support at least one communication protocol. For example, each of the plurality of communication nodes may include a code division multiple access (CDMA) based communication protocol, a wideband CDMA (WCDMA) based communication protocol, a time division multiple access (TDMA) based communication protocol, and a frequency division multiple (FDMA) based communication protocol. access based communication protocol, orthogonal frequency division multiplexing (OFDM) based communication protocol, orthogonal frequency division multiple access (OFDMA) based communication protocol, single carrier (SC) -FDMA based communication protocol, non-orthogonal multiple An access based communication protocol and a space division multiple access (SDMA) based communication protocol may be supported. The structure of each of the plurality of communication nodes is described with reference to FIG. 2 below.

2 is a block diagram showing a communication node in the communication system according to the first embodiment of the present invention.

Referring to FIG. 2, the communication node 200 may include at least one processor 210, a memory 220, and a transceiver 230 that communicates with a network. In addition, the communication node 200 may further include an input interface device 240, an output interface device 250, a storage device 260, and the like. Each component included in the communication node 200 may be connected by a bus 270 to communicate with each other.

However, each component included in the communication node 200 may be connected through a separate interface or a separate bus around the processor 210, instead of the common bus 270. For example, the processor 210 may be connected to at least one of the memory 220, the transceiver 230, the input interface device 240, the output interface device 250, and the storage device 260 through a dedicated interface. .

The processor 210 may execute a program command stored in at least one of the memory 220 and the storage device 260. The processor 210 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Each of the memory 220 and the storage device 260 may be configured as at least one of a volatile storage medium and a nonvolatile storage medium. For example, the memory 220 may be configured as at least one of a read only memory (ROM) and a random access memory (RAM).

3 is a conceptual diagram illustrating an operation of a communication environment and a resource manager in a communication system according to an embodiment of the present invention.

Referring to FIG. 3, the communication system 300 may include a communication environment 310 and a resource manager 320. The communication environment 310 may include at least one terminal (not shown) and a plurality of transmission points (not shown). The resource manager 320 may be a radio unit (not shown) connected to at least one terminal and a plurality of transmission points in the communication environment 310. Here, the communication system 300 may be the same as or similar to the communication system 100 of FIG. 1. In addition, the structure of the at least one terminal, the plurality of transmission points, and the resource manager 320 may be the same as or similar to that of the communication node 200 of FIG. 2.

In the communication system 300, the resource manager 320 determines a transmission point for transmitting data to the terminal by using an optimal decision making method in the communication environment 310 that changes dynamically over time. Link adaptation may be determined. For example, the resource manager 320 may utilize quality-learning and deep quality network.

The communication system 300 may select an optimal transmission point and perform link adaptation through trial and error experience in an unpredictable situation in which at least one transmission point is randomly disposed in the communication environment 310. For example, in the communication system 300, the resource manager 320 may be provided with a reward 313 such as a maximum data rate in a state 311 of each communication environment 310. Resource management action 312 may be performed. In this case, the resource manager 320 may set parameters and select an optimal transmission point and link adaptation scheme. That is, the communication system 300 may provide a model for optimizing the communication environment 310 by using the compensation 313 fed back according to the action 312 in a given state 311 through the resource manager 320. have.

The resource manager 320 in the communication system 300 may perform an operation for maximizing a reward 313 such as a data rate from the experience of trial and error. For example, the resource manager 320 defines a state 311 of a given communication environment 310 and performs an action 312 for resource management in each state 311 to obtain a reward 313. Can be performed. In this case, the resource manager 320 may perform an exploration of performing a random action 312 not related to the reward 313 to experience a new behavior that has not been experienced. In addition, the resource manager 320 may perform a compromise by utilizing an action 312 that maximizes the reward 313.

In the communication system 300, the resource manager 320 associates independent conditions that must satisfy SON functions when defining the reward 313 and the state 311, and thus may not generate impulses between parameters.

In the communication system 300, the resource manager 320 may provide a method of selecting a transmission point for inducing a maximum data rate through trial and error learning. In addition, the resource manager 320 in the communication system 300 may provide a link adaptation scheme for short-range transmission points for deriving a maximum data rate through trial and error learning.

4 is a conceptual diagram illustrating a deep Q-network learning operation in a communication system according to an embodiment of the present invention.

Referring to FIG. 4, the resource manager (not shown) uses the actions 430-1 through 430-n determined based on the state 410 of the communication environment through the deep Q-network 420 learning in the communication system. Q values 440-1 through 440-n may be derived. Here, the communication system may be the same as or similar to the communication system 300 of FIG. 3. In addition, the resource manager may operate similarly or similarly to the resource manager 320 of FIG. 3. In addition, the communication environment may be the same as or similar to the communication environment 310 of FIG. 3. Deep Q-network 420 learning is described with reference to FIG. 5 below.

5 is a conceptual diagram illustrating a deep Q-network operation of updating weights in a communication system according to an embodiment of the present invention.

를 연산하는 제2 연산부(540), 및 제2 연산부(540)로부터 수신되는 값에 기초하여 행동들을 출력하는 출력 레이어(550)를 포함할 수 있다. 자원 관리기는 딥 Q-네트워크 학습부를 통해 Q 값들을 출력(Y)할 수 있다. 자원 관리기는 도 4의 자원 관리기와 동일 또는 유사하게 동작할 수 있다.Referring to FIG. 5, the deep Q-network learner may include an input layer 520 that receives state 510 and a first calculator 530 that calculates a value through the input layer 520. Can be. In addition, the deep Q-network learner weights a value output from the first calculator 530.

It may include a second operation unit 540 for calculating a, and an output layer 550 for outputting actions based on the value received from the second operation unit 540. The resource manager may output (Y) the Q values through the deep Q-network learner. The resource manager may operate the same as or similar to the resource manager of FIG. 4.

6 is a flowchart illustrating the operation of a resource manager in a communication system according to an embodiment of the present invention.

Referring to FIG. 6, the resource manager may receive radio channel state information from the terminal (S601). The resource manager may receive transmission point state information from the terminal (S602). The resource manager may determine a transmission point and link adaptation scheme that provides the maximum data rate based on the wireless channel state information and the transmission point state information (S603). The resource manager may transmit information about a transmission point providing the maximum data rate to the terminal (S604). The resource manager may transmit information on a link adaptation scheme that provides the maximum data rate to the terminal (S605).

The resource manager may operate the same as or similar to the resource manager of FIG. 5. A detailed operation of the resource manager will be described with reference to FIGS. 7 to 10 below.

7 is a flowchart illustrating an operation of a Q-learning resource manager for determining a transmission point through Q-learning in a communication system according to an embodiment of the present invention.

In the communication system, the Q-learning resource manager may receive feedback information from each of the plurality of terminals (S701). For example, the feedback information may include radio channel state information measured by the terminal, channel interference information, load information of the transmission point, and handover ratio information of the transmission point.

Here, the wireless channel status information may include a common reference signal (CRS) and channel status information (CSI) -reference signal (RS). The channel interference information may include an interference measurement (CSI-IM).

The load information of the transmission point may include information about the transmission rate of the transmission point per terminal. Handover failure of a transmission point may include a handover or a ping pong handover that is faster or slower than a predetermined rate.

The CSI-RM may indicate a predefined value RS according to the strength of the received signal. Here, rs may be represented by 1, 2, ..., S. The CSI-IM may indicate a predefined value im according to the strength of the interference signal. Here, im = 1, 2, ..., I may be represented.

The load information of the transmission point may indicate a predefined value l. Here, l = 1, 2, ..., L can be represented. The handover rate information of the transmission point may indicate a predefined value h. Here, l = 1, 2, ..., can be represented by H. For example, h may be expressed as a handover failure rate or success rate. For example, when the handover failure rate or success rate of the transmission point is 0.1 or less, h = 1. In addition, when the handover failure rate or success rate of the transmission point is 0.01 or more and 0.1 or less, it may be indicated by h = 2. In addition, when the handover failure rate or success rate of the transmission point is 10 ^-H or more and 10 ^-H-1 or less, h = H may be indicated.

Here, S, I, L, and H are natural numbers.

The action that the Q-learning resource manager selects a transmission point may be represented by a. In addition, the number of transmission points that the Q-learning resource manager can select may be represented by P. In this case, the selection for the transmission point may be indicated by 1 and the non-selection by 0. For example, if P = 17, you would display a = (0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0). Can be. That is, when P = 17, a may be represented by a set of 17 zeros or ones.

The Q-learning resource manager may update the Q value (S702). The Q-learning resource manager may set the Q value to zero in the initial operation. In addition, the resource manager may also set S, I, L, and H corresponding to the feedback information to zero.

In a communication system, a resource manager may determine a Q value by defining a state of a communication environment, a compensation caused by state-based behavior, a random value ε, a learning rate α, and a discount factor γ. The maximized Q value may explain that the policy used to determine status, behavior, and reward in learning is optimal. The Q-learning resource manager can determine the Q value through two internal and external loop operations. Here, the outer circular loop may mean an operation of a sequence of input states. In addition, the outer circular loop may mean an operation of a sequence of time. For example, the Q-learning resource manager may perform an internal circular loop operation for rs, im, l, h which are in a wireless communication state. In addition, the resource manager may perform an external cyclic loop operation according to the time t = 1, 2, ..., T. The Q-learning resource manager may set a Q value of 0 for all states and actions in a circular loop initialization process, and then perform a circular loop operation.

The Q-learning resource manager can update the new state and Q values that change according to the selected behavior. For example, the Q-learning resource manager may update the Q value Q _t (s, a) for each state s and behavior a at time t as shown in Equation 1 below.

는 Q-학습 자원 관리기가 시간 t-1에서 상태 s에서 행동 a를 수행하여 얻은 보상 값을 의미한다.

는 Q-학습 자원 관리기가 상태 v에서 행동

를 수행한 최대 Q 값을 의미한다.here,

Denotes the reward value obtained by the Q-learning resource manager performing action a in state s at time t-1.

Q-Learning Resource Manager acts in state v

It means the maximum Q value.

에 할인 변수

를 곱한 값과 보상 값

을 더한

에 학습률

를 곱한 값, 그리고 시간 t-1에서의 Q 값인

에

를 곱한 값을 더하여 Q 값 Q_t(s, a)을 업데이트할 수 있다. 여기서, 학습률

는 경험에서 얻은 새로운 값과 과거 학습한 값에 대한 비율을 의미한다. 즉, Q 값은 결정적(deterministic)이 아닌 확률적(stochastic)으로 업데이트 될 수 있다.That is, the Q-learning resource manager has a maximum Q value

On discount variables

Multiplied by

plus

Learning rate

Multiplied by and Q value at time t-1

on

You can update the Q value Q _t (s, a) by adding the product times. Where learning rate

Is the ratio of the new value gained from experience to the value learned in the past. That is, the Q value may be updated stochastic rather than deterministic.

방식을 사용할 수 있다. In addition, the Q-learning resource manager is a method of exploring and utilizing iteratively generates a random value ε rather than an ε-greedy method, and reduces it according to the number of iterations.

Can be used.

After the Q-learning resource manager generates a random value (S702). The Q-learning resource manager compares the selected random value with a predetermined constant ε and performs a random action S705 when the random value is smaller than ε. On the other hand, if the random value is larger than ε, the behavior of providing the largest Q value in the current state is performed (S704). The above operation may be referred to as an ε-greedy method.

The Q-learning resource manager may provide handover information in which the transmission point is changed when the transmission point maximizing the Q value is changed (S706). For example, the Q-learning resource manager may determine the data rate of each transmission point based on load information of the transmission point. In addition, the Q-learning resource manager may determine the total data rate through the entire transmission points based on the load information of the transmission points.

The Q-learning resource manager selects a transmission point from the updated Q value Q _t (s, a), the Q value before update Q _t-1 (s, a), and the state s (rs, im, l, h). The data rate of each transmission point may be calculated based on the data rate r _t-1 (s, a) when performing action a, state s, and action a. For example, the Q-learning resource manager may calculate a data rate d _i of the i th transmission point through Equation 2 below.

The Q-learning resource manager may select a transmission point with a low load when the data rates are the same for different transmission points.

The Q-learning resource manager may transmit information about the transmission point to the terminal (S706). For example, the Q-learning resource manager may transmit information about a transmission point that provides the maximum transmission rate or information about less than three transmission points that are arbitrarily selected to the terminal. The Q-learning resource manager may return to step S701 and repeat the previous step after the operation ends.

In the communication system, the Q-learning resource manager may configure the Q learning, and then perform an action of inputting the state of the communication environment to provide the maximum data rate, which is an optimal reward. The Q-learning resource manager may operate the same as or similar to the resource manager of FIG. 6.

8 is a flowchart illustrating an operation of a Q-learning resource manager for determining a link adaptation scheme through Q-learning in a communication system according to an embodiment of the present invention.

The Q-learning resource manager may operate the same as or similar to the Q-learning resource manager of FIG. 7. In the following description, operations overlapping with those of the Q-learning resource manager of FIG. 7 will be omitted.

Referring to FIG. 8, the Q-learning resource manager may receive feedback information from the terminal (S801). The feedback information may include channel quality information (CQI), precoding matrix index (PMI), rank indication (RI), and interference information. The CQI may indicate a predefined value CQI. Here, CQI may be represented as 1, 2, ..., CQ. The RI may indicate a predefined value RI according to the radio channel state. Here, RI = 1, 2, ..., R can be represented. PMI may indicate a predefined value PMI. Here, PMI may be represented as 1, 2, ..., PM. The interference information may be CSM-IM. The CSM-IM may indicate a predefined value IM according to the strength of the interfering signal. Here, IM = 1, 2, ..., I can be represented. Here, CQ, R, PM and I are natural numbers.

The Q-learning resource manager may update the Q value based on the feedback information (S802). The Q-learning resource manager may set the Q value to zero in the initial operation. In addition, the Q-learning resource manager may also set CQI, RI, PMI, and IM corresponding to the feedback information to zero. If it is not the initial operation, the Q-learning resource manager may update the Q value based on Equation 1.

The Q-learning resource manager can perform internal and external circular loop operations. For example, the Q-learning resource manager may perform an internal cyclic loop operation for CQI, RI, PMI, and IM in a wireless communication state. In addition, the Q-learning resource manager may perform an external cyclic loop operation according to time t = 1, 2, ..., T.

The Q-learning resource manager selects a random value (S802). The Q-learning resource manager compares the selected random value with ε, and performs a random action S805 when the random value is smaller than ε. On the other hand, if the random value is greater than ε, the behavior of providing the largest Q value in the current state is performed (S804).

The Q-learning resource manager determines the MCS, HARQ, and antenna transmission method that maximizes the Q value as the link adaptation method.

For example, the Q-learning resource manager may determine the data rate of each transmission point based on load information of the transmission point. In addition, the Q-learning resource manager may determine the total data rate through the entire transmission points based on the load information of the transmission points.

The Q-learning resource manager may transmit information on a link adaptation scheme to the terminal (S807). For example, the resource manager may transmit information on an MCS, a HARQ, and an antenna transmission scheme, which is a link adaptation scheme providing a maximum data rate, to the terminal. Alternatively, the Q-learning resource manager may transmit information on a randomly selected MCS, HARQ, and antenna transmission scheme to the terminal. The Q-learning resource manager may return to step S801 and repeat the previous step after the operation ends.

In addition, the Q-learning resource manager in the communication system may configure a deep Q-network, and then perform an action of inputting a state of the communication environment to provide a maximum data rate that is an optimal compensation. In this case, the Q-learning resource manager may be referred to as a deep Q-network learning manager. The operation of the deep Q-network learning resource manager will be described with reference to FIGS. 9 and 10 below.

9 is a flowchart illustrating an operation of a deep Q-network learning resource manager for determining a transmission point through deep Q-network learning in a communication system according to an embodiment of the present invention.

The deep Q-network learning resource manager may operate the same as or similar to the Q-learning resource manager of FIG. 8. In the following description, operations overlapping with those of the deep Q-network learning resource manager of FIG. 8 will be omitted.

Referring to FIG. 9, in a communication system, the deep Q-network learning resource manager may receive feedback information from each of a plurality of terminals (S901). The operation of receiving the feedback information by the deep Q-network learning resource manager may be the same as or similar to the operation S701 of receiving the feedback information by the Q-learning resource manager of FIG. 7.

The deep Q-network learning resource manager may update the weight according to a predetermined cycle (S902). For example, the deep Q-network learning resource manager may define the state, behavior, reward, random value ε, and discount variable γ of the communication environment in the communication system. The deep Q-network learning resource manager may determine the weight constituting the network for deep Q-network learning through two circular loop operations inside and outside. The outer circular loop may mean an operation of a sequence of input states. An inner circular loop may mean an operation of a sequence of time. The deep Q-network learning resource manager may initialize the weight of the network to a random value at the beginning of the cyclic loop and then perform a cyclic loop operation.

The deep Q-network learning resource manager may perform a calculation to update a state changing according to a selected action in a given state to a new state and update weights as shown in Equation 3 below.

는 할인 변수를 의미한다.

는 가설(예측) 모델의 가중치를 의미한다.

는 실제 모델 네트워크의 가중치를 의미한다.here,

Means a discount variable.

Is the weight of the hypothesis model.

Denotes the weight of the actual model network.

의 곱으로 표현된 가설 모델 네트워크(hypothesis network) 가중치

s와 실제 모델 네트워크 가중치

의 차이의 제곱이 최소화되는 값으로 가중치를 업데이트할 수 있다. 즉, 딥 Q-네트워크 학습 자원 관리기는 코스트(cost) 또는 손실(loss)를 최소화하는 값으로 가중치를 업데이트할 수 있다.Deep Q-network learning resource manager weighted with input s

Hypothesis network weight expressed as the product of

s and physical model network weights

The weight may be updated to a value that minimizes the square of the difference of. That is, the deep Q-network learning resource manager may update the weight to a value that minimizes cost or loss.

는 순환 루프 동작때 마다 업데이트할 수 있다. 반면, 딥 Q-네트워크 학습 자원 관리기는 실제 모델 네트워크의 가중치

는 순환 루프 동작때 마다 매번 업데이트하지 않고, 미리 정해진 순환 루프 주기에 따라 업데이트할 수 있다. 즉, 가설 모델 네트워크에서의 가중치

와 실제 네트워크에서의 가중치

를 분리하지 않고 계산하는 경우, 오류가 발생할 수 있기 때문에, 딥 Q-네트워크 학습 자원 관리기는 가설 모델 네트워크에서의 가중치

와 실제 네트워크에서의 가중치

를 독립적으로 계산할 수 있다. 다시 말해, 딥 Q-네트워크 학습 자원 관리기는 가설 모델 네트워크에서의 가중치

와 실제 네트워크에서의 가중치

를 분리하여 계산할 수 있다.Deep Q-network learning resource manager weighted hypothetical model network

Can be updated for each loop operation. On the other hand, deep Q-network learning resource manager is the weight of the physical model network

May not be updated every time in a cyclic loop operation, but may be updated according to a predetermined cyclic loop period. That is, the weights in the hypothetical model network

And weights in real networks

If you calculate without splitting, the deep Q-network learning resource manager is weighted in the hypothetical model network, because errors can occur.

And weights in real networks

Can be calculated independently. In other words, the deep Q-network learning resource manager is weighted in the hypothetical model network.

And weights in real networks

Can be calculated separately.

와 실제 네트워크에서의 가중치

의 차이의 제곱이 최소화되는 가중치를 구하기 위해, 순환 루프때 마다

를 계산하고, 미리 정해진 순환 루프의 주기에 따라

를

로 업데이트할 수 있다.That is, deep Q-network learning resource manager is weighted in the hypothetical model network

And weights in real networks

In order to find the weight that minimizes the square of the difference between

Is calculated, and according to a predetermined cycle of the loop

To

Can be updated with

가 랜덤 값 ε 보다 작은 경우, 임의의 행동을 수행할 수 있다. 반면, 딥 Q-네트워크 학습 자원 관리기는 가중치

가 랜덤 값 ε 보다 큰 경우, 현재 상태에서 가장 큰 보상 값을 주는 행동을 수집할 수 있다. 여기서, 행동 수집은 행동을 직접 수행하지 않고 버퍼링(buffering)을 하여 임의의 몇 개만을 선택하여 행동을 수행하는 경험 재현(experience replay)을 수행하는 동작을 의미할 수 있다. 딥 Q-네트워크 학습 자원 관리기는 경험 재현 방식에서 많은 샘플들이 상관관계(correlation)를 갖고 있어 오류 계산을 유도할 수 있으므로 상관관계가 없어지도록 랜덤하게 샘플을 선택할 수 있다. The deep Q-network learning resource manager may generate a random value (S903). Deep Q-Network Resource Manager generates a random value ε

If is smaller than the random value ε, any action may be performed. In contrast, deep Q-network learning resource manager weighted

If is greater than the random value ε, the behavior giving the largest compensation value in the current state can be collected. Here, the behavior collection may refer to an operation of performing an experience replay of performing an action by selecting only a few randomly by performing buffering without directly performing an action. The deep Q-network learning resource manager can randomly select samples so that they do not correlate because many samples have correlations that can lead to error calculations.

The deep Q-network learning resource manager selects a random value (S903). The deep Q-network learning resource manager compares the selected random value with ε and performs random transmission point selection (S706) when the random value is smaller than ε. On the other hand, if the random value is larger than ε, it is possible to select a transmission point providing the largest Q value in the current state (S705). The deep Q-network learning resource manager may return to step S901 and repeat the previous step after the operation ends.

10 is a flowchart illustrating an operation of a deep Q-network learning resource manager for determining a link adaptation scheme through deep Q-network learning in a communication system according to an exemplary embodiment of the present invention.

The deep Q-network learning resource manager may operate the same as or similar to the deep Q-network learning resource manager of FIG. 9. In the following description, operations overlapping with those of the deep Q-network learning resource manager of FIG. 9 will be omitted.

The deep Q-network learning resource manager may receive feedback information from the terminal (S1001). The operation of the deep Q-network learning resource manager receiving the feedback information from the terminal may be the same as or similar to the operation S801 of receiving the feedback information from the terminal of the Q-learning resource manager of FIG. 8.

The deep Q-network learning resource manager may update the weight based on the feedback information (S1002). The deep Q-network learning resource manager may set the weight to zero in the initial operation. In addition, the deep Q-network learning resource manager may also set CQI, RI, PMI, and IM corresponding to the feedback information to zero. If the deep Q-network learning resource manager is not the initial operation, it may update the weight based on Equation 3. The weight update operation of the deep Q-network learning resource manager may be the same as or similar to the weight update operation S902 of the deep Q-network learning resource manager of FIG. 9.

The deep Q-network learning resource manager can perform internal and external circular loop operations. For example, the deep Q-network learning resource manager may perform an internal cyclic loop operation for CQI, RI, PMI, and IM in a wireless communication state. In addition, the deep Q-network learning resource manager may perform an external cyclic loop operation according to time t = 1, 2, ..., T.

The deep Q network learning resource manager selects a random value (S1003). The deep Q-network learning resource manager compares the selected random value with [epsilon] and performs random transmission point selection (S1006) when the random value is smaller than [epsilon]. On the other hand, if the random value is larger than ε, the transmission point providing the largest Q value in the current state is selected (S1005).

The deep Q network learning resource manager may determine an MCS, an HARQ, and an antenna transmission scheme that is resource management (action) maximizing the Q value (S1005).

For example, the deep Q-network learning resource manager may determine the data rate of each transmission point based on load information of the transmission point. In addition, the deep Q-network learning resource manager may determine the total data rate through the entire transmission points based on load information of the transmission points.

The deep Q-network learning resource manager may transmit information on a link adaptation method to the terminal (S1007). For example, the deep Q-network learning resource manager may transmit information regarding a link adaptation scheme, MCS, HARQ, and antenna transmission scheme, which provides a maximum data rate, to the terminal. Alternatively, the deep Q-network learning resource manager may transmit information on a randomly selected MCS, HARQ, and antenna transmission scheme to the terminal. The deep Q-network learning resource manager may return to step S1001 and repeat the previous step after the operation ends.

In addition, the deep Q-network learning resource manager in the communication system may configure the deep Q-network, and then perform an action of inputting a state of the communication environment to provide a maximum data rate, which is an optimal reward.

Conventional resource management methods perform state and resource management (behavior) by using only utilization behavior methods that provide maximum rewards at the moment, but the method of performing exploration behaviors with the use of the present invention provides more status and It provides an opportunity to perform resource management to obtain higher data rates.

The methods according to the invention can be implemented in the form of program instructions that can be executed by various computer means and recorded on a computer readable medium. Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like. The program instructions recorded on the computer readable medium may be those specially designed and constructed for the present invention, or may be known and available to those skilled in computer software.

Examples of computer readable media include hardware devices that are specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code, such as produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate with at least one software module to perform the operations of the present invention, and vice versa.

Although described with reference to the embodiments above, those skilled in the art will understand that the present invention can be variously modified and changed without departing from the spirit and scope of the invention as set forth in the claims below. Could be.

Claims (20)

A method of operating a resource manager connected to a plurality of transmission points in a communication system,
Receiving wireless channel state information and transmission point state information from a terminal;
Determining a transmission point and link adaptation scheme providing a maximum data rate based on the radio channel state information and the transmission point state information; And
And transmitting information on a transmission point that provides the maximum data rate and information on a link adaptation scheme to the terminal.
The transmission point and link adaptation scheme providing the maximum data rate is determined through a quality-learning scheme or a deep Q-network learning scheme. The method according to claim 1,
The operation method of the resource manager,
Updating a quality (Q) value based on the radio channel state information and the transmission point state information; And
Generating a random random value; the method of operation of a resource manager. The method according to claim 2,
The wireless channel state information includes channel state information (CSI) -reference signal (RS), interference measurement (CSI-IM), channel quality information (CQI), precoding matrix index (PMI), rank indication (RI), and interference state information. At least one of
The transmission point state information includes at least one of a transmission rate of a transmission point and a handover rate of the transmission point,
And wherein the Q value indicates a maximum data rate determined based on the radio channel state information and the respective transmission point state information. The method according to claim 3,
The operation method of the resource manager,
Selecting a mode of exploration or utilization; And
Determining a number of transmission points of the plurality of transmission points to maximize the Q value. The method according to claim 4,
The operation method of the resource manager,
When the Q value is a maximum value, determining the transmission point that provides the largest transmission rate among the plurality of transmission points based on the Q value when the handover ratio exceeds a predetermined threshold ratio. A method of operating a resource manager, including. The method according to claim 3,
The operation method of the resource manager,
Determining the Q value as a process of utilization behavior; And
Determining the MCS, HARQ, and antenna transmission mode when determining the Q value as a process of exploration. The method according to claim 6,
The operation method of the resource manager,
Determining the MCS, HARQ, and antenna transmission modes based on the maximum value of the Q value. The method according to claim 3,
The operation method of the resource manager,
Updating a weight value based on the radio channel state information and the transmission point state information; And
Generating a Q value; the method of operating a resource manager. The method according to claim 8,
The operation method of the resource manager,
Selecting an action in a manner that explores or utilizes the Q value; And
In the case of the exploration method, determining any number of transmission points among the plurality of transmission points. The method according to claim 9,
The operation method of the resource manager,
Determining a transmission point that provides the largest transmission rate among the plurality of transmission points when the Q value is maximum and the handover rate exceeds a predetermined threshold rate. Way. The method according to claim 8,
The operation method of the resource manager,
Determining behavior by way of utilization or exploration; And in the case of the exploratory behavior, determining any MCS, HARQ, and antenna transmission modes. The method according to claim 11,
The operation method of the resource manager,
When performing the behavior in a utilizing manner, further comprising the step of determining the MCS, HARQ, and antenna transmission mode according to the maximum Q value. A resource manager connected to a plurality of transmission points in a communication system,
A processor; And
At least one instruction executed by the processor includes a memory (memory),
The at least one command is
Receiving wireless channel state information and transmission point state information from a terminal, determining a transmission point and link adaptation scheme that provides a maximum data rate based on the wireless channel state information and the transmission point state information, and And transmits information about a transmission point that provides the maximum data rate and information about a link adaptation scheme to the terminal,
The transmission point and link adaptation scheme providing the maximum data rate is determined through a quality-learning scheme or a deep Q-network learning scheme. The method according to claim 13,
The at least one command is
And updating a quality (Q) value based on the radio channel state information and the transmission point state information. The method according to claim 14,
The wireless channel state information includes channel state information (CSI) -reference signal (RS), interference measurement (CSI-IM), channel quality information (CQI), precoding matrix index (PMI), rank indication (RI), and interference state information. At least one of
The transmission point state information includes at least one of a transmission rate of a transmission point and a handover rate of the transmission point,
Wherein the Q value indicates a maximum data rate determined based on the radio channel state information and the respective transmission point state information. The method according to claim 13,
The at least one command is
If the behavior is exploratory,
Determine any number of transmission points among the plurality of transmission points,
And further to update a quality value based on the radio channel state information and the transmission point state information, and to generate a random random value. The method according to claim 13,
The at least one command is
Determine behavior, perform exploratory behavior, determine any MCS, HARQ, and antenna transmission modes, and
And if the Q value calculation scheme is utilization, further determining to determine the MCS, HARQ, and antenna transmission modes that maximize the Q value. The method according to claim 13,
The at least one command is
And further to update a weight value based on the radio channel state information and the transmission point state information, and to generate a Q value. The method according to claim 18,
The at least one command determines the behavioral manner, and if the behavioral behavior is exploration, determines any number of transmission points among a plurality of transmission points,
And if the Q value behavior is utilization, further determine to determine a transmission point based on the maximum Q value. The method according to claim 18,
The at least one command is
Decides how to act or how to explore,
If exploring, determine any MCS, HARQ, and antenna transmission modes, and
And if utilized, further executing to determine an MCS, HARQ, and antenna transmission mode that provides a maximum Q value.

Images