JP2022018901A - Optimization method of wireless communication system, wireless communication system, and program for wireless communication system - Google Patents

Abstract

To provide an optimization method for a wireless communication system of performing optimization in wireless communication for each of a plurality of wireless communication terminals, and also performing optimization when the plurality of wireless communication terminals are viewed as a whole.SOLUTION: A wireless communication systems including a plurality of wireless communication terminals is optimized. For each of the wireless communication terminals, action A (t+1) is determined such that the highest reward R'(t) can be obtained on the basis of the state S (t) provided by an environment 14. Due to returning of the action A (t+1) to the environment 14, each individual reward R (t+1) obtained by the wireless communication terminal is calculated. On the basis of the reward for each of the plurality of wireless communication terminals belonging to a group, the utility representing the fairness of the plurality of wireless communication terminals belonging to the group is calculated. The reward R'(t+1) for each of the wireless communication terminals is calculated on the basis of each reward R (t+1) and the utility.SELECTED DRAWING: Figure 3

Description

The present invention relates to a method for optimizing a wireless communication system, a wireless communication system, and a program for a wireless communication system, and in particular, a method for optimizing a wireless communication system for optimizing a communication state by using learning of multi-step evaluation, wireless. Related to communication systems and programs for wireless communication systems.

More specifically, the present invention uses a computer such as machine learning or reinforcement learning that evaluates to achieve the following two items together in an environment where different wireless communication systems interfere with each other and coexist. It's about learning.
1. 1. Maximize the communication capacity within each wireless communication system.
2. 2. Achieve overall optimization between wireless communication systems that share the same frequency resources. That is, fairness is realized for evaluation items such as the throughput achievement rate defined in each wireless communication system.

Wireless LAN is a wireless communication system that can be used inexpensively in the license-free band. For this reason, its spread is rapidly increasing, and a large number of wireless LAN terminals are mixed in the same area. As a result, it is a problem that wireless LAN terminals interfere with each other. In response to these problems, many techniques have been proposed for minimizing the influence of interference between wireless LAN terminals and expanding the individual or overall system capacity.

For example, FIG. 1 shows an example in which wireless communication terminals 1 to N are wireless LAN base stations (APs: Access Points) that interfere with each other. The wireless communication terminals N + 1 to N + M shown in the lower part of FIG. 1 are user terminals such as smartphones that establish communication with the above AP. In this example, each of the wireless communication terminals 1 to N functioning as an AP acquires interference information in the vicinity thereof and information on success / failure of connection with the wireless communication terminals N + 1 to N + M, and sends the control server 10 as wireless environment information. Send.

The control server 10 calculates the allocation of frequency channels and transmission power values so that the throughput of the AP group including the wireless communication terminals 1 to N is maximized, and returns the result as control information to each AP.

On the other hand, there are wireless communication systems that use unlicensed bands other than wireless LAN, and they share the same frequency resources as wireless LAN for communication. In particular, in the 920MHz band, which is currently open to RFID and IoT in Japan, a plurality of wireless communication systems coexist. For example, in Japan, wireless communication systems such as LoRAWAN, Wi-SUN, and SIGFOX have started services within the same frequency band. In wireless LAN, IEEE 802.11ah is considered to be a wireless communication system that uses the 920 MHz band.

Overseas, if the carrier sense is stipulated in the standard, the time is divided. In addition, if the carrier sense is not specified, the frequency resource will be shared or used at the same time as other wireless communication systems.

However, in Japan, the 920 MHz band is not sufficiently allocated to accommodate a plurality of wireless communication systems. Therefore, it is difficult to always divide the frequency resources, and it is assumed that the same frequency resources are used at the same time.

These wireless communication systems that share the same frequency resources have very different standards and specifications, even if they are for the same IoT. Since the modulation / demodulation method and access control are different, the frequency utilization efficiency and communication distance are also different, and it is not rational to handle them on the same evaluation axis.

According to Non-Patent Document 1, LoRAWAN has an occupied bandwidth of 125 kHz, a communication distance of about 10 km, and a communication speed of several tens of kbs at the maximum. In addition, SIGFOX has a bandwidth of 100 Hz, a communication distance of several tens of kilometers, and a communication speed of 100 bps. Wi-SUN has a maximum occupied bandwidth of 600kHz, a communication distance of about 1km, and a communication speed of several hundreds of kbps. 11ah has an occupied bandwidth of 1MHz or more, a communication speed of about 1km, and a communication speed of several Mbps.

The above wireless communication systems have different standards and specifications, as well as completely different use cases and traffic. For example, wide-area, low-speed wireless communication systems such as SIGFOX are applied in sensor-based use cases where traffic occurs several times a day and transmission is performed at low speed. On the other hand, in a high-speed wireless communication system such as 11ah, it may be applied to use cases where traffic always occurs, such as video transmission from a surveillance camera.

As described above, a plurality of wireless communication systems using the same frequency resource differ greatly in communication standards and specifications, and also differ in required throughput and frequency. Therefore, in the calculation of optimization regarding the allocation of frequency resources and the like, it is necessary to perform the calculation based on the conditions of each of those systems.

On the other hand, when frequency resources are allocated to a plurality of wireless communication terminals having different conditions, there are cases where the optimum calculation can be performed individually but not the optimum as a whole. For example, in the calculation result in which the frequency resource is preferentially allocated only to the wireless communication terminal considered to have a high communication speed, the service in the wireless communication terminal having a small frequency resource allocation may be delayed. In this case, the evaluation of all wireless communication systems using frequency resources will result in leaving an outage, which is not optimal.

Therefore, when there are multiple wireless communication systems with different conditions, it is possible to optimize each terminal and also consider the wireless communication terminals of each condition and the wireless communication terminals of all the coexisting wireless communication systems. Control is required.

Latest trends and future prospects of LPWA, Chiba University, Shiro Sakata, June 2018 IEEE Std 802.11ah-2016, December 2016

When optimizing a wireless communication resource in which multiple wireless communication systems with different conditions coexist, such as the license-free band described above, one type of wireless communication system is used as in the conventional method described above. Optimizing individual wireless communication resources is not enough. Under these circumstances, each of the multiple wireless communication systems is evaluated and optimized not only for individual systems but also for all wireless communication terminals belonging to all wireless communication systems that use wireless communication resources. It is necessary to realize the realization.

In the present invention, optimization in wireless communication is performed for each of a plurality of wireless communication terminals, and optimization is also performed when a plurality of wireless communication terminals are viewed as a whole. Therefore, reinforcement learning is highly evaluated. Perform in stages.

The first invention is a method for optimizing a wireless communication system including a plurality of wireless communication terminals in order to achieve the above object, and is the best based on the state provided by the environment for each wireless communication terminal. A step of determining an action so as to obtain a reward of the above, a step of calculating an individual reward obtained by the wireless communication terminal by returning the action to the environment, and the individual for each of a plurality of wireless communication terminals. A step of calculating the utility representing the fairness of the plurality of wireless communication terminals based on the reward of, and a reward calculation step of calculating the reward for each wireless communication terminal based on the individual reward and the utility. , Is desirable to include.

The second invention is a wireless communication system including a plurality of wireless communication terminals, which is a control server that receives wireless environment information from the plurality of wireless communication terminals and provides control information to the plurality of wireless communication terminals. The control server is provided with a process of determining an action of each wireless communication terminal so as to obtain the highest reward based on the state provided by the environment, and the action is returned to the environment. , A process of calculating the individual reward obtained by the wireless communication terminal, and a process of calculating the utility representing the fairness of the plurality of wireless communication terminals based on the individual reward for each of the plurality of wireless communication terminals. It is desirable to execute a process of calculating the reward for each wireless communication terminal based on the individual reward and the utility.

The third invention is a program for a wireless communication system implemented in a control server that receives wireless environment information from a plurality of wireless communication terminals and provides control information to the plurality of wireless communication terminals, and the control thereof. The wireless communication terminal is subjected to a process of determining an action of each wireless communication terminal to the server so as to obtain the highest reward based on the state provided by the environment, and the action is returned to the environment. The process of calculating the individual rewards to be obtained, the process of calculating the utility indicating the fairness of the plurality of wireless communication terminals based on the individual rewards for each of the plurality of wireless communication terminals, and the process of calculating the utility of the individual wireless communication terminals. It is desirable to execute a process of calculating the reward for the device based on the individual reward and the utility.

According to the present invention, the reward of a wireless communication terminal is calculated based on the reward individually received by the terminal and the utility which is the result of evaluating a plurality of wireless communication terminals from the viewpoint of fairness. Then, the behavior of each wireless communication terminal is determined so that the reward is maximized. Therefore, according to the present invention, both the optimization of each wireless communication terminal and the optimization when a plurality of wireless communication terminals are evaluated as a whole can be realized in a well-balanced manner.

It is a figure for demonstrating the configuration example of a wireless communication system. It is a figure for demonstrating the model example of the conventional reinforcement learning. It is a figure for demonstrating an example of the model of reinforcement learning carried out in Embodiment 1 of this invention. It is a flowchart for demonstrating the example of the learning algorithm implemented in Embodiment 1 of this invention. It is a figure for demonstrating an example of the model of reinforcement learning carried out in Embodiment 2 of this invention. It is a flowchart for demonstrating the example of the learning algorithm implemented in Embodiment 2 of this invention.

Embodiment 1.
[Structure of Embodiment 1]
The wireless communication system of the first embodiment of the present invention can be realized by the configuration example shown in FIG. In FIG. 1, the wireless communication terminals 1 to N shown in the middle stage each function as an access point (AP). These can communicate with the wireless communication terminals N + 1 to N + M shown in the lower part of FIG. The wireless communication terminals N + 1 to N + M are composed of a smartphone, a sensor for IoT, a smart meter, and the like. As described above, the configuration shown in FIG. 1 includes a plurality of wireless communication systems that share the same frequency resource but have different standards and specifications.

The wireless communication system of this embodiment includes a control server 10. The control server 10 includes hardware such as a communication interface, a processor unit, and a memory. The control server 10 realizes the functions described later by having these hardware proceed with processing according to a program stored in the memory.

The control server 10 can provide control information to the wireless communication terminals 1 to N that function as APs. The control information includes, for example, information such as available frequency resources and transmission power. On the other hand, the wireless communication terminals 1 to N can transmit wireless environment information to the control server 10. The wireless environment information includes interference information in the vicinity of each of the wireless communication terminals 1 to N and information on success or failure of connection with the wireless communication terminals N + 1 to N + M.

Further, the control server 10 is provided with a learning function for optimizing various parameters included in the control information based on wireless environment information and the like, and a function for determining these various parameters based on the learning result. ing.

[Outline of reinforcement learning]

In this embodiment, reinforcement learning is used for optimizing various parameters included in the control information. FIG. 2 shows a model diagram of general reinforcement learning. In the model shown in FIG. 2, an agent 12 exists as a learning target. The agent 12 calculates and executes the action A (t + 1) from the current state S (t) and the reward R (t) in the unique environment 14 with the observation timing of the event as t. As a result, the state S (t + 1) is realized. From this state S (t + 1), the reward R (t + 1) for evaluating the action is obtained, and the next action is calculated.

In the following description, s and S represent states, a and A represent actions, and r and R represent rewards, respectively. Here, lowercase letters mean parameters for individual agents (optimization targets), and uppercase letters mean parameters for their set (plurality of agents). Further, the subscript t of each parameter indicates that the parameter is a value at the observation timing t, and St, At, and Rt are the same as S (t), A (t), and R (t), respectively. It shall be.

The reinforcement learning shown in FIG. 2 is advanced by repeating the following steps.
1. 1. The agent 12 receives the state S (t) and the reward R (t) from the environment 14, and returns the action A (t) determined based on the policy π to the environment 14.
2. 2. The environment 14 changes to the next state S (t + 1) based on the action A (t) received from the agent 12 and the current state S (t), and the state S (t + 1) and the reward R (t + 1) after the transition. ) Is provided to the agent 12. The reward R is a scalar amount indicating the quality of the action A immediately before that.

Assuming that the action of the agent for a certain state S is A, the sum of the rewards R that can be obtained from the present time to the infinite future, that is, the profit G is as follows.

However, γ is 0 ≦ γ ≦ 1, and is a parameter for adjusting how much the influence of future rewards is evaluated as profit.

In Q-learning by reinforcement learning, the value of action a is evaluated by the following function.

However, E is a function indicating an expected value. Further, Q ^π is a value function (hereinafter referred to as “Q function”) representing an expected value when an agent taking action a from the state s takes an action according to the policy π.

The reinforcement learning shown in FIG. 2 is advanced so as to maximize this Q function. This learning can be advanced, for example, by obtaining a Q function for estimating the profit G when the action a is performed in the state s by the algorithm of the following equation.

Here, p is a parameter called the learning rate, which is an algebra determined by the machine learning designer. Usually set to a small value less than 1. Further, maxQ indicates the maximum value of the Q function that is considered to be ideally acquired. The learning of the Q function is to estimate all the Q values obtained by the action taken at the next time t + 1 for each time t, and update the Q value using the largest one among them.

[Characteristics of Embodiment 1]
FIG. 3 shows a model of reinforcement learning implemented in the wireless communication system of the present embodiment. In the present embodiment, individual evaluations and evaluations of each condition are carried out for optimization of a plurality of wireless communication systems having different conditions. Multiple wireless communication systems can be grouped based on their respective conditions. In the model shown in FIG. 3, there are three groups, and an agent exists for each group.

Agents 12-1, 12-2, and 12-3 shown in FIG. 3 evaluate the behavior of each user i belonging to each group under the same environment 14, and also evaluate the entire group. For example, the agent 12-1 includes an agent i corresponding to each of the plurality of users i included in the group 1. The agent i evaluates the behavior of the user i and evaluates the entire group 1 in consideration of fairness.

Requirements such as the number of connections and bandwidth required for each agent i are different, and it cannot be said that the resources can be fully utilized without considering the allocation of resources accordingly. Therefore, when evaluating the entire group, simply allocating resources equally by the number of agents i does not guarantee fairness. Therefore, the validity of the allocation to each agent i achieved by the distribution of resources will be defined by the utility function.

When the resource allocated to the user i is xi, the utility function of the user i is expressed as R (xi). If the sum of utility functions for each user i can be maximized, it can be said that the validity of resource allocation for the entire system is maximized and the resources are allocated fairly.

Specifically, the following function is used as the utility function R (xi).

However, α is a parameter for determining the fairness of the utility function R. In the above utility function R, if α is ∞, the utility that maximizes the minimum value between users, that is, max-min fairness can be evaluated. In the present embodiment, by using such a setting, it is possible to realize resource allocation according to the wireless communication terminal having the minimum reward by the above utility function R.

For example, consider the case of optimizing the allocation of frequency resources to a wireless communication system. Here, in the group 1 wireless communication system, the 1/2/4 MHz band can be allocated, and the throughput achievement rate can be calculated from the required traffic and throughput of each wireless communication terminal. The throughput can be calculated from the allocated bandwidth, the number of wireless communication terminals coexisting within the allocated frequency resource, the distance between the transmitting and receiving terminals, and the like.

Similarly, in the group 2 wireless communication system, the 200/400 / 600kHz band can be allocated, and the throughput achievement rate can be calculated from the traffic and the throughput as in the group 1 wireless communication system. Further, the wireless communication system of Group 3 can also calculate the throughput achievement rate from the same calculation method.

The evaluation value of each wireless communication terminal at this time is x1, x2, x3, ... In the wireless communication system of group 1. In the group 2 wireless communication system, the evaluation values are y1, y2, y3, .... The evaluation values of the wireless communication system of Group 3 are z1, z2, z3, .... In this case, the overall evaluation of each of the groups 1 to 3 can be expressed as follows. In the evaluation function below, β and ε are parameters for determining the fairness of the utility function, like α.

Based on the overall evaluation of the group, for example, the reward of a certain wireless communication terminal k can be calculated as follows.

Consider the following environment to show an example of a concrete algorithm.
First, it is assumed that there are n wireless communication terminals and k number of available frequency channels as an environment. At a certain time, each communication terminal shall select one from k channels and try to use that channel, or take one action from the (k + 1) options of not using the channel. .. At that time, each terminal receives an ACK when the selected channel does not overlap with the other terminal and the channel can be used for the action taken by itself, and it is the same as any one of the other terminals. If you select a channel, you will not receive an ACK. The success or failure of receiving this ACK is regarded as the reward of each terminal. The action of each terminal and the reward of the result for it will be regarded as a state at a certain time. As another reward, a utility function calculated from the total number of connections of each terminal (the number of wireless communication terminals that received ACK) at regular intervals is defined.

In addition to the success or failure of receiving the ACK, the throughput and communication capacity are calculated from the communication results so far, and the judgment result of whether or not the user is in an outage state that does not meet the required communication quality is specified as a reward. You may. Alternatively, the number of wireless communication terminals that have not reached the outage state in the group may be specified as a reward. Considering the outage state as a reward, it is possible to proceed with learning by using whether or not the user quality is maintained as an index. Therefore, according to the above method, effective learning according to the user's experience is possible.

FIG. 4 shows an outline of learning performed by the control server 10 in the present embodiment.
According to the algorithm shown in FIG. 4, first, the method of action selection of n users i is determined (step 100).

In step 100, one of the following three methods is randomly selected.
1. 1. Method of determining random behavior without using learning results (step 102)
2. 2. Method using learning using Main-net (steps 104, 106)
3. 3. Method using learning using Fair-net (steps 108, 110)
Here, the reason for randomly selecting a channel with a certain probability is to prevent learning from falling into a local solution and to promote learning efficiently.

In this embodiment, a known Deep Q Network (DQN) method is used for learning the Q function in order to deal with the case where the number of states that the agent can take is enormous. The above-mentioned Main-net is the name given to the reward r of each user i, that is, the DQN that searches for the measure π so as to maximize the expected value of the user i's channel availability at each time. The above-mentioned Fair-net is the name given to the DQN that searches for the measure π so as to maximize the above-mentioned utility function set in consideration of the utility of the entire group.

When all the actions of each terminal i are determined, the reward r and the state s of each terminal i are determined (step 112).

Next, the action a, the reward r, and the state s for each terminal i are added to the memory of the control server 10 for learning (step 114). The reward r is the result xi of the channel availability of the terminal i at each time and the calculation result of the utility function R. It should be noted that these data need only be stored for a certain period of time.

Next, the information of each terminal corresponding to the plurality of time slots is randomly extracted from the above memory (step 116).

Then, batch learning is executed using them as training data, and the parameters of Main-net and Fair-net are updated (step 118).

Based on the updated parameters, each terminal repeats channel selection (steps 104 to 110) or random channel selection (step 102) based on the learning result again, and learning proceeds following the same flow (steps 112 to 112). 118).

In the above description, the three methods by which the terminal determines the behavior are randomly determined, but the present invention is not limited thereto. For example, for the method in which each terminal selects an action based on the learning result, the learning result using Fair-net is used with a certain probability based on the learning result using Main-net. It may be that. Further, the probability of randomly determining the action may be set lower than the probability of determining the action using the learning result.

As described above, in the wireless communication system of the present embodiment, the behavior based on the individual learning results is set as the first step, and the overall evaluation based on the utility function is set as the second step, so that the behavior of each terminal is evaluated in multiple stages. Can be done. Therefore, according to the present embodiment, both the optimization of each of the plurality of wireless communication terminals belonging to the same group and the optimization for ensuring fairness within the same group can be realized. ..

Embodiment 2.
Next, a second embodiment of the present invention will be described with reference to FIGS. 5 and 6 together with FIG. The wireless communication system of the present embodiment can be realized by the configuration shown in FIG. 1 as in the case of the first embodiment. The system of the present embodiment is the same as that of the first embodiment except that the method of determining the behavior of the terminal and the method of learning the Q function are different.

[Characteristics of Embodiment 2]
FIG. 5 shows a model of reinforcement learning implemented in the wireless communication system of the present embodiment. In the model shown in FIG. 5, in addition to the processing performed by the model shown in FIG. 3, an overall evaluation targeting all three groups is performed. This overall evaluation is performed to optimize fairness for all wireless communication terminals operating under the same environment 14.

In the model shown in FIG. 5, the utility function R _all of the overall evaluation is calculated by the following algorithm using the evaluation results of the wireless communication systems of groups 1 to 3. Note that θ included in the following equation is a parameter for determining the fairness of the utility function, like α included in the utility function R for the group.

However, Σ in the above equation means to take the sum of the rewards Rx, Ry, and Rz of the three groups.

In the system of the present embodiment, the reward Rxk of a certain wireless communication terminal k can be calculated as follows by using the individual reward xk, the group reward R, and the total reward R _all .

If the reward Rxk of the wireless communication terminal k is calculated as described above, the behavior of each terminal can be determined in consideration of max-min fairness for all wireless communication terminals belonging to the same environment 14. ..

FIG. 6 shows an outline of learning performed by the control server 10 in the present embodiment. The flowchart shown in FIG. 6 is similar to the flowchart shown in FIG. 4, except that steps 120 and 122 are added.

As shown in FIG. 6, in the present embodiment, as a method of action selection, a method of using learning using Fair-net (whole) is adopted with a certain probability (steps 120 and 122). "Fair-net (whole)" is the name given to DQN that searches for a policy π to maximize the utility function R _all for all wireless communication terminals belonging to the same environment 14.

According to the above processing, optimization for individual wireless communication terminals, guarantee of fairness among terminals belonging to the same group, and guarantee of fairness among all terminals are realized in a well-balanced manner. Can be done.

10 Control server 12, 12-1, 12-2, 12-3 Agent 14 Environment S, s State R, r Reward A, a Action

Claims (8)

It is an optimization method for a wireless communication system that includes multiple wireless communication terminals.
For each wireless communication terminal, the steps to determine the behavior to get the highest reward based on the condition provided by the environment,
The step of calculating the individual reward obtained by the wireless communication terminal by returning the action to the environment, and
Based on the individual rewards for each of the plurality of wireless communication terminals, a step of calculating the utility representing the fairness of the plurality of wireless communication terminals, and
A reward calculation step that calculates a reward for an individual wireless communication terminal based on the individual reward and the utility.
How to optimize wireless communication systems, including. The plurality of wireless communication terminals include a plurality of wireless communication terminals belonging to a group having the same communication standard and at least one of the requirements.
The utility is calculated for the group and
The optimization method according to claim 1, wherein in the reward calculation step, the reward of the wireless communication terminal belonging to the group is calculated based on the individual reward and the utility for the group. The plurality of wireless communication terminals are classified into a plurality of groups, and the plurality of wireless communication terminals are classified into a plurality of groups.
Including the step of calculating the overall utility representing fairness for all of the plurality of wireless communication terminals based on the utility for each of the plurality of groups.
The optimization method according to claim 2, wherein in the reward calculation step, the reward of the wireless communication terminal belonging to the group is calculated based on the individual reward, the utility for the group, and the total utility. The optimization method according to any one of claims 1 to 3, wherein the utility is calculated based on the number of terminals that have received an ACK, which means success of communication, among the plurality of wireless communication terminals. The optimization method according to any one of claims 1 to 3, wherein the utility is a value evaluated by fairness calculated from throughput and traffic load of the plurality of wireless communication terminals. The optimization method according to any one of claims 1 to 3, wherein the utility is calculated based on the number of outage terminals calculated from the requirements for the plurality of wireless communication terminals. A wireless communication system that includes multiple wireless communication terminals.
A control server that receives wireless environment information from the plurality of wireless communication terminals and provides control information to the plurality of wireless communication terminals is provided.
The control server is
For each wireless communication terminal, the process of determining the behavior to get the highest reward based on the condition provided by the environment,
The process of calculating the individual reward obtained by the wireless communication terminal by returning the action to the environment, and
A process of calculating the utility representing the fairness of the plurality of wireless communication terminals based on the individual rewards for each of the plurality of wireless communication terminals.
A process of calculating a reward for an individual wireless communication terminal based on the individual reward and the utility, and
A wireless communication system that runs. A program for a wireless communication system implemented in a control server that receives wireless environment information from a plurality of wireless communication terminals and provides control information to the plurality of wireless communication terminals.
To the control server
For each wireless communication terminal, the process of determining the behavior to get the highest reward based on the condition provided by the environment,
The process of calculating the individual reward obtained by the wireless communication terminal by returning the action to the environment, and
A process of calculating the utility representing the fairness of the plurality of wireless communication terminals based on the individual rewards for each of the plurality of wireless communication terminals.
A process of calculating a reward for an individual wireless communication terminal based on the individual reward and the utility, and
A program for wireless communication systems to execute.

Images