KR101877243B1 - Ap apparatus clustering method using neural network based on reinforcement learning and cooperative communicatin apparatus using neural network based on reinforcement learning - Google Patents

Abstract

An AP device clustering method using a reinforcement learning-based neural network includes: a step of confirming the distribution of terminals in a cell of a first AP device by using channel state information; a step of determining at least one candidate AP device able to be serviced for a specific area of the cell along with the first AP device based on the distribution of the terminals; a step of determining at least one second AP device among candidate AP devices by using a reinforcement learning-based neural network considering a position of the first AP device, a position of the candidate AP device, the distribution of the terminals, and the channel state information of the terminals as input; and a step of clustering the first AP device and second AP device. As such, the present invention is capable of providing a high-quality service to terminals located on the border of a cell.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an AP apparatus clustering method using a neural network based on reinforcement learning and a cooperative communication apparatus using a neural network based on reinforcement learning. [0002]

The techniques described below relate to cooperative communications of a mobile communication AP device.

Various techniques are being studied for increasing data requirements as the number of mobile communication devices increases. The AP may cooperate with an adjacent AP device to provide a service to a terminal located in a boundary area of a cell served by the AP device such as a base station. That is, a plurality of AP devices cooperate to provide a communication service for one terminal. For example, there is a technology such as Coordinated Multi-Point (CoMP) proposed by LTE-Advanced.

For collaborative communication, a plurality of AP devices that provide services in cooperation must be selected. The process of selecting a plurality of AP devices is called clustering. Clustering techniques include dynamic clustering and static clustering. The dynamic clustering is a real-time clustering according to the location of the UE, and the static clustering performs clustering using a predetermined pattern.

Korean Patent Publication No. 10-2012-0060627

Dynamic clustering reflects the location of the terminal in real time, which results in a large overhead of the system. Correct clustering is difficult to provide stable service when the terminal moves out of the expected position or traffic surges. The technique described below is intended to provide clustering between AP devices using a reinforcement learning based neural network. The technique described below is intended to provide clustering between AP devices using a deep Q-network (DQN).

A method of clustering an AP apparatus using a reinforcement learning based neural network includes the steps of: checking a distribution of a terminal in a cell of a first AP apparatus using channel state information; The method comprising: determining at least one candidate AP device capable of serving a specific area; reinforcement learning for inputting a location of the first AP device, a location of the candidate AP device, a distribution of the terminal, Based neural network to determine at least one second AP device of the at least one candidate AP device and clustering the first AP device and the at least one second AP device.

The collaborative communication apparatus using the reinforcement learning based neural network includes a storage device for storing a neural network parameter and a location of a neighboring AP device, an antenna for receiving channel state information from a terminal in a cell, and a terminal in a cell identified using the channel state information Determining at least one candidate AP apparatus capable of serving a specific region of the cell among the neighbor AP apparatuses based on the distribution of the location of the AP apparatus, Based neural network to determine at least one target AP device of the at least one candidate AP device.

The technology described below provides optimal quality clustering through reinforcement learning using neural networks based on reinforcement learning, thereby providing high quality services to terminals located in a boundary region of a cell.

Figure 1 is an example of a communication environment for collaborative communication.
Figure 2 is an example of clustering for collaborative communications.
Figure 3 is an example of a sigmoid function.
Figure 4 is an example of a flowchart for Q learning.
Figure 5 is an example of clustering through reinforcement learning.
6 shows an example of DQN.
FIG. 7 shows an example of the post-learning process of the DQN.
8 is an example of clustering using DQN.
9 is another example of clustering using DQN.
10 is another example of clustering using DQN.

The following description is intended to illustrate and describe specific embodiments in the drawings, since various changes may be made and the embodiments may have various embodiments. However, it should be understood that the following description does not limit the specific embodiments, but includes all changes, equivalents, and alternatives falling within the spirit and scope of the following description.

The terms first, second, A, B, etc., may be used to describe various components, but the components are not limited by the terms, but may be used to distinguish one component from another . For example, without departing from the scope of the following description, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

As used herein, the singular " include "should be understood to include a plurality of representations unless the context clearly dictates otherwise, and the terms" comprises & , Parts or combinations thereof, and does not preclude the presence or addition of one or more other features, integers, steps, components, components, or combinations thereof.

Before describing the drawings in detail, it is to be clarified that the division of constituent parts in this specification is merely a division by main functions of each constituent part. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more functions according to functions that are more subdivided. In addition, each of the constituent units described below may additionally perform some or all of the functions of other constituent units in addition to the main functions of the constituent units themselves, and that some of the main functions, And may be carried out in a dedicated manner.

Also, in performing a method or an operation method, each of the processes constituting the method may take place differently from the stated order unless clearly specified in the context. That is, each process may occur in the same order as described, may be performed substantially concurrently, or may be performed in the opposite order.

The following description relates to a cooperative communication technique of a plurality of AP apparatuses. The techniques described below relate to clustering between multiple AP devices for cooperative communications. The AP device may be a mobile communication AP (base station), a small cell AP, a WiFi AP, and the like. Cooperative communication may be performed between APs of the same type or may be performed in a heterogeneous network such as a macro cell and a small cell. Further, the AP apparatus may be a device using the same communication method or, in some cases, a device using a different communication method. For convenience of explanation, it is assumed that cooperative communication is performed between AP devices such as base stations of mobile communication.

Figure 1 is an example of a communication environment for collaborative communication. The AP apparatus is arranged in a certain area. 1 shows an example in which the entire area is divided into n × n types of grids. For convenience of explanation, one AP device is shown for each square. The UEs are randomly distributed in the entire area. The terminal may also be repositioned with respect to time. For example, the AP apparatus 10A can provide communication services for the terminals 5A and 5B. The terminal 5B located in the border area of the cell may be served by another AP apparatus 10B. Therefore, the AP apparatus 10A and the AP apparatus 10B can interfere with each other. In this case, if the AP device 10A and the AP device 10B perform cooperative communication, the communication service can be provided to the terminal 5B without a trunk line. Hereinafter, a process of determining an AP apparatus to be clustered for cooperation communication will be described.

A reinforcement learning based neural network is used to determine AP devices that perform collaborative communications. For example, DQN can be used. Hereinafter, the DQN will be mainly described. DQN is an algorithm that performs reinforcement learning on a wider state space by adding a value network to the Q-learning technique.

Conventional Q-learning techniques can perform vigorous learning in a moving environment within a limited number of states. However, as the state space increases, there is a problem in storing the Q value. The Q value is a measure of the value function of the state. For example, the location and distribution of each terminal can be infinitely changed, and there is a very large number of combinations, so it is not efficient to store the Q value for each situation.

DQN solves the above problem by estimating a function that determines the Q value rather than storing the individual Q value. If the conventional Q learning technique stores the state in each table and confirms the Q value through a lookup, the DQN inputs the current state to the value network and extracts the Q value as a result value. DQN can approximate a function that determines the Q value using more than three layers of value networks.

Q-learning is basically a reinforcement learning algorithm consisting of environment, agent, state, action, and reward. First, the agent takes action and the agent can move to the new state. The agent receives two rewards (immediate reward and future reward) from the environment for the actions taken. Immediate compensation is an immediate reimbursement for the actions taken by the agent, and future compensation is compensation for the future environment caused by the behavior. Ultimately, the agent's ultimate goal is to update the Q value to get the maximum of the two rewards. This can be expressed as Equation 1 below.

Where s is the state, a is the action, and r is the compensation. γ is a discount factor between 0 and 1, and if it is close to 0, the closer to 1, the more emphasis is on the importance of compensation for the future. In the present invention, it is currently set at 0.5 to equally compensate for future. α _t has a learning rate between 0 and 1 and determines the learning rate of the Q value. For example, if α _t = 0, no agent learning is performed. If α _t = 1, the agent learns by using the most recent information. We assume that α _t = 1 because agents need to learn from past Q values.

In the clustering process, the behavior is clustering, and the compensation corresponds to the throughput according to the clustering. An agent corresponds to a subject performing clustering, and may be an AP device. Or a separate control device located in the mobile communication core network may be an agent.

Referring to FIG. 1, a terminal is randomly distributed in an entire area, and can move constantly with time. It is assumed that the user's location information can be known using CSI (channel state information), which is a unit representing the channel state information of the user. The state of Q-learning is defined as shown in Equation (2) below.

Where C represents the base station identification number. C ∈ {1,2,3, ..., N}. The UE is an identification number of the UE. UE ∈ {user ₁ , user ₂ , user ₃ , ..., user _M }. CI represents CSI information. CI ∈ {CSI ₁ , CSI ₂ , CSI ₃ , ..., CSI _m }. For example, assuming that there are three AP devices and four terminals, and the number of terminals supported by three base stations is {1,2,2} respectively, the state can be generated as follows. _{Q t (s) = [(} 1,2,3), (user 1, {user 2, user 3}, {user 4, user 5}), (CSI 1, {CSI 2, CSI 3}, {CSI ₄ , CSI ₅ })].

The behavior of the agent depends on the environment of the AP device. For example, behavior may vary depending on whether the surrounding environment is a road on which a car moves or a pedestrian zone. In the case of an AP apparatus located near the road, the number of AP apparatuses K forming the cluster can be determined and a cluster can be formed according to the road shape (anticipated moving direction).

Figure 2 is an example of clustering for collaborative communications. 2 (a) shows an example of clustering of AP devices arranged in the road area. When K = 2 is set, two adjacent AP devices in the vicinity of the road can be selected to form a cluster.

Fig. 2 (b) is an example of clustering for a human walking zone. In the walking zone, clusters can be formed according to the following steps. First, (1) CSI information is used to find the area with the greatest number of users in the boundary area. (2) As shown in FIG. 2 (b), a neighboring AP device is selected based on the AP device in the corresponding area to form a cluster. (a) A criterion for selecting a neighboring AP device is a neighboring AP device that can maximize interference cancellation. (b) If the interference size is the same, all adjacent AP devices are clustered. 2 (b) shows an example of selecting a neighboring AP device capable of eliminating the interference of a terminal located in the border area as much as possible. 2 (b) shows an example in which the AP apparatus 20A selects the AP apparatus 20B located below.

2 (b) shows an AP apparatus as an agent apparatus on the right side. The AP apparatus includes a storage device 21 for storing DQN variables and other information, a control circuit 22 for determining DQN learning and clustering type, and an antenna 23 for communicating with the terminal. The antenna 23 may receive channel state information from a terminal in the cell. The control circuit 22 determines at least one candidate AP apparatus capable of serving a specific region of the neighboring AP apparatus based on the distribution of terminals in the cell confirmed using the channel state information, , The distribution of the terminal, and the channel state information of each terminal to the DQN to determine at least one target AP apparatus among at least one candidate AP apparatus. Thereafter, the AP device and the target AP device perform clustering and cooperative communication. As will be described later, the storage device 21 may store behaviors and rewards for later learning.

The compensation may use the performance or throughput of the terminal as a compensation value for the action taken by the agent. The compensation can be set as shown in Equation (3) and Equation (4) below.

Where S is used to calculate e _t which compensates the compensation with an improved sigmoid function. Figure 3 is an example of a sigmoid function. T _lb is the performance value of the lower 5% of the total performance, and T _avg is the average value of the overall performance. 5% is an example. As T _lb increases, the compensation increases and maintains a cluster shape that increases the overall performance of the terminal. On the contrary, if the T _lb value is small, the compensation is greatly reduced due to the feature of the sigmoid function shown in FIG. 3, and the existing cluster behavior is modified to form the other type cluster to support the user. The sigmoid function of FIG. 3 has a characteristic in which the derivative value is small in the domain near 0 and 1, and increases as the value approaches 0.5. When the 5% performance is similar to the average performance, the penalty is small. If the 5% performance becomes smaller than a certain level, a large penalty is added to guarantee the capacity of the border area terminals.

FIG. 4 is an example of a flowchart for the Q learning process 100. FIG. The agent confirms the current state s (C, UE, CSI) (110). The agent acquires the Q value using the DQN (120). The agent selects an action to determine the clustering type according to the Q value (130). The agent then observes compensation for the behavior (140). If the learning is not terminated, the agent stores 150 its action and the compensation 150 accordingly. This process is repeated until learning ends. Through this process, the agent prepares a DQN for determining the clustering. The agent can perform learning while clustering in a real environment. Agents can also use certain sample data to learn in advance. The agent may be any one AP device as described above. Or another control device that receives information from the AP device. For example, the agent may be a control device located in the core network of mobile communication.

Figure 5 is an example of a clustering process 200 through reinforcement learning. FIG. 5 presupposes a situation in which the learned DQN is prepared according to FIG. The agent confirms the current state s (C, UE, CSI) (210). The agent acquires the Q value using the learned DQN (220). The agent selects an action that determines the clustering type according to the Q value (230). The agent then observes the compensation according to the behavior (240). The agent determines whether the compensation according to the current action is greater than the compensation just before (250). The agent can determine that the compensation is large if the current compensation is greater than a predetermined threshold than the previous compensation. That is, the agent determines whether the performance of the terminal is constantly improved according to the clustering.

If the current compensation is consistently greater than the previous compensation, then the agent changes the cluster according to the behavior (260). If the current compensation is not greater than the previous compensation, the agent does not change the cluster. The agent verifies (270) that the communication is terminated, and repeats the entire process until the communication is terminated.

To create a value network effectively, it must reflect the characteristics of the state. The current clustering environment is a two-dimensional environment of AP devices and terminals. AP devices with many terminals in the boundary area can increase the capacity by removing interference through clustering. It is efficient to operate AP devices separately if most terminals are near AP devices and there is little movement. Therefore, using artificial neural network which can reflect 2 - dimensional structure as value network helps to improve performance.

CNN (Convolutional Neural Network) is an artificial neural network structure that can best understand the above two-dimensional structure. CNN consists of several convolutional layers and several fully connected layers. The convolution layer extracts the 2-D structure as observed through a convolution mask and shared weights. By superimposing the convolutional layers, more complex features can be found. Using these complex features, we can derive the Q value to the complete connection layer. One of the most frequently used techniques in CNN is max pooling, which is used as a mechanism to ensure the translational invariance by lowering the complexity by extracting only the largest value in the masked space.

6 shows an example of DQN. Figure 6 is an example of the value network described above. The first convolution layer receives the location of the current AP device, the distribution of the terminal, and the CSI of each terminal as input. This layer uses a 5 * 5 convolution mask to find low-level features. The low level features mean, for example, simple features such as terminal distribution and density between any two AP devices. The next two layers use a 3 * 3 convolution mask to find high-level features. The high level features are the characteristics that can be deduced from the low level features found earlier, which means the spatial distribution of the two pairs of AP devices with many terminals and the terminal movement patterns in time.

In the last layer, 2 * 2 maximum pooling is performed. Maximum pooling is the task of leaving only one maximum value within the n * n mask, which can be seen as a job of reducing data and reducing accuracy. After this layer, all output values are input to the complete connection layer. The complete connection layer can have 1000, then 100, and 10 dimensions of the first layer. This is to gradually reduce the number of output values of the neurons, leaving only important features. At the end, 10 output values are collected in one neuron to derive the Q value. The value network structure shown in Fig. 6 is an example. The actual DQN may use a value network of a different structure.

The value network is learned according to the procedure of training the basic DQN. First, the behavior is changed in the communication environment, that is, the clustering environment is changed, and the compensation according to the behavior is observed. The agent stores the observed behavior and the compensated pair in the storage device. Agents learn value networks every regular training session. The agent can perform learning in the training period and update the DQN network using the behavior and compensation stored in the storage device.

FIG. 7 shows an example of the post-learning process 300 of the DQN. The agent confirms the behavior and compensation information stored in the storage device (310). The agent invokes the variable of the DQN (320). The DQN variable is pre-stored in the storage device. The agent learns the DQN using the behavior and compensation stored in the storage device (330). The agent then uses the learned DQN to observe behavioral compensation (340). The agent uses the sample data (behavior and compensation) stored in the storage device to repeat the learning process until the learning is terminated (350). Finally, the agent specifies a variable of the newly learned DQN (360). The newly specified variable can be stored in the storage device.

Hereinafter, some examples of clustering using the above-described DQN will be described.

8 is an example of clustering using DQN. Fig. 8 shows the situation of a performance hall performing a performance. A lot of terminals are gathered at a performance hall during a performance time and try to communicate at one time. Since the performance is performed at an unspecified time, a very heavy burden is imposed on the AP apparatus around the stadium at an unspecified time. Under the existing static clustering scheme, it is difficult to guarantee the QoS considering only the environment in which the performance is progressing or not. If reinforcement learning clustering is performed using the DQN according to the above-described method, the clustering type of the AP apparatus can be changed according to the changing terminal density to increase the capacity. For example, it is possible to utilize the communication resources of the AP apparatus having a low terminal density by clustering the AP apparatus around the stadium and the AP apparatus having a low terminal density. Accordingly, the total capacity of the terminals in the stadium can be increased by sharing resources that the AP device with a low terminal density can not utilize to the AP devices around the stadium.

9 is another example of clustering using DQN. Figure 9 is an example of a downtown situation. In urban areas, terminal traffic rapidly increases at a certain time (commute time), and traffic is resolved after commute. In the case of existing static clustering, since the clustering can not be performed in response to the terminal traffic changing in each time slot, the network capacity is degraded. In order to maximize the compensation value determined according to the throughput, the reinforcement learning clustering using DQN can increase the network capacity by eliminating the interference problem between the AP devices by forming only clusters where the terminal traffic increases. In addition, it is possible to provide a stable network capacity by decreasing the number of handover times of the terminal by utilizing the behavior of forming a cluster according to the road shape in the morning and morning time, where terminals move at a high speed. In case of dynamic clustering, it is difficult to apply to real network model because the system overhead is greatly increased because the state change of many terminals must be reflected in real time.

10 is another example of clustering using DQN. Figure 10 is an example of a situation in which a disaster occurred. In the event of a disaster, nearby AP devices are destroyed and the number of rescue personnel increases, so that the amount of data traffic to be temporarily handled by a movable AP device increases rapidly. In the case of static clustering, it is difficult to provide stable network capacity in a disaster situation because a fixed cluster pattern is applied without recognizing the situation change. DQN, the system recognizes that the compensation value is greatly reduced. In order to increase the compensation, a cluster is formed between the AP devices that can be operated as shown in FIG. 10, Resources are shared and used. Therefore, it is possible to concentrate the network resources on the AP device that needs support, thereby providing the network capacity necessary for the structure. On the other hand, dynamic clustering requires high computational power, but it is impossible to provide stable network capacity in a situation where most AP devices are lost in a disaster situation.

The present embodiment and drawings attached hereto are only a part of the technical idea included in the above-described technology, and it is easy for a person skilled in the art to easily understand the technical idea included in the description of the above- It will be appreciated that variations that may be deduced and specific embodiments are included within the scope of the foregoing description.

5: Terminal
10A, 10B: AP device
20A, 20B: AP device
21: Storage device
22: control circuit
23: Antenna

Claims (15)

Confirming a distribution of a terminal in a cell of the first AP apparatus using channel state information;
Determining at least one candidate AP device capable of serving a specific area of the cell together with the first AP device based on the distribution of the terminal;
Wherein at least one candidate AP device among the at least one candidate AP device uses a reinforcement learning based neural network that receives as input the location of the first AP device, the location of the candidate AP device, the distribution of the terminal, 2 AP device; And
And clustering the first AP device and the at least one second AP device using the reinforcement learning based neural network. The method according to claim 1,
And a neural network based on the reinforcement learning for confirming the distribution of the terminal using CSI (channel state information) for the terminal located in the cell. The method according to claim 1,
Wherein the specific region is a region in which the largest number of terminals are located in the boundary region of the cell. The method according to claim 1,
Wherein the first AP device or the control device of the core network determines the second AP device using the neural network. The method according to claim 1,
Wherein the first AP apparatus and the second AP apparatus provide a service for a target terminal located in the specific area through cooperative communication. 6. The method of claim 5,
Further comprising the step of measuring the performance of the cooperative communication with the target terminal and updating the neural network using the clustering and performance of the first AP device and the second AP device, Clustering of AP devices. The method according to claim 1,
Wherein the candidate AP apparatus is determined in consideration of a movement path of an area where the first AP apparatus is located. The method according to claim 1,
Wherein the reinforcement learning based neural network uses a reinforcement learning based neural network for outputting state information including a specific AP apparatus and a specific location by using learning data including the location of the AP apparatus, the distribution of the terminal, and the channel state information Clustering of AP devices. The method according to claim 1,
Wherein the reinforcement learning-based neural network is a Deep Q-network (DQN). A storage device for storing variables of the neural network and the location of the adjacent AP device;
An antenna for receiving channel state information from a terminal in a cell; And
Determining at least one candidate AP apparatus capable of serving a specific region of the neighboring AP apparatus based on a distribution of terminals in a cell identified using the channel state information,
And a control circuit for determining at least one target AP device among the at least one candidate AP device by inputting the location of the candidate AP device, the distribution of the terminal, and the channel state information of each terminal to a reinforcement learning- Collaborative communication device using reinforcement learning based neural network. 11. The method of claim 10,
Wherein the specific region is a region in which the largest number of terminals are located in the boundary region of the cell. 11. The method of claim 10,
Wherein the cooperating communication device is an AP device or a control device of a core network. 11. The method of claim 10,
And the cooperating communication device provides a service for the specific area through cooperative communication with the target AP device. 14. The method of claim 13,
Based on the reinforcement learning based on the performance of the target AP apparatus and the cooperative communication. 11. The method of claim 10,
Wherein the reinforcement learning-based neural network is a Deep Q-network (DQN).

Images