JP7007669B2 - Communication system, traffic control device and traffic control method - Google Patents

Description

The present invention relates to a communication system, a traffic control device, and a traffic control method.

Millimeter-wave communication is expected as a next-generation wireless communication technology that can realize high-capacity and high-speed communication (see, for example, Non-Patent Document 1). One of the advantages of millimeter-wave communication is that the available frequency width is wide band, and high-speed communication exceeding 1 Gbit / s (Gigabit per second) is possible. On the other hand, millimeter waves have a drawback that they are greatly attenuated by moisture and oxygen, and when the line-of-sight communication path is shielded by a human body or the like, the communication quality sharply deteriorates (see, for example, Non-Patent Document 2). In order to deal with the problem of steep communication quality deterioration due to this shielding, a device for controlling the flow rate of the shielded communication path and the traffic path is required. Specifically, as shown in FIG. 9, in a wireless communication system in an environment in which an AP (Access Point) communicates with a plurality of STAs (Stations) by millimeter waves, the outlook for the AP and the STA. It is a situation where the human body can shield the communication path, and a control device for effectively using the radio band of the AP in such a situation is required. In the following, N STAs (N is an integer of 1 or more) are also referred to as STA-1 to STA-N.

As a method for solving a communication control problem in millimeter-wave communication, a traffic control device based on human body shielding prediction using an RGB-D camera has been proposed (see, for example, Non-Patent Document 3). In the prior art, the human body is detected using the image / moving image data obtained from the RGB-D camera, and the movement destination thereof is predicted. When the human body blocks the line-of-sight communication path between AP and STA by moving to the destination, the traffic between AP and STA is stopped immediately before the shielding occurs, and the traffic on the unshielded communication path is given priority for transmission. do. This control can increase the total throughput in the AP compared to the case without control. That is, traffic control for effectively using the radio band becomes possible. In addition, since the shielding is predicted and proactively controlled immediately before the shielding occurs, the total throughput can be increased as compared with the conventional reactive control method in which control is performed after the throughput decreases.

FIG. 10 is a functional block diagram of a traffic control device to which the technique of Non-Patent Document 3 is applied. In the figure, a traffic control device is mounted on a proxy server of a wireless communication system in which AP and STA-1 to STA-N communicate wirelessly. The traffic control device includes an image analysis unit, a shielding determination unit, and a communication control unit. When operating the traffic control device, a communication path is set in the shield determination unit as an initial setting. The image analysis unit estimates the position of the human body (obstacle) in millimeter-wave communication using the image obtained from the RGB-D camera. Next, the shielding determination unit determines whether or not the preset line-of-sight communication path is shielded by the human body from the estimated position of the human body and its moving speed, and if it is determined to be shielded, the timing. To estimate.

The communication control unit controls the flow rate of the traffic so as to stop the traffic at the time estimated that the shielding occurs, based on the shielding condition of the line-of-sight communication path estimated by the shielding determination unit. Specifically, the communication control unit stops the transmission of the packet addressed to the STA, which is received from the Internet and whose line-of-sight communication path is blocked. Further, the communication control unit resumes the transmission of the packet addressed to the STA at the time estimated that the shielding is released. This traffic control allows the AP to allocate resources to communication with another STA even when the throughput of communication with one STA decreases due to human body shielding. Therefore, the total throughput in the AP can be increased as compared with the case where the traffic control is not performed.

P. Wang, Y. Li, L. Song, and B. Vucetic, “Multi-gigabit thin wave wireless communications for 5G: From fixed access to cellular networks,” IEEE Communications Magazine, January 2015, vol.53, no .1, p.168-178 S. Collonge, G. Zaharia, and GE Zein, “Influence of the human activity on wide-band characteristics of the 60 GHz indoor radio channel,” IEEE Transactions on Wireless Communications, November 2004, vol.3, no.6 , p.2396-2406 T. Nishio, R. Arai, K. Yamamoto, and M. Morikura, “Proactive traffic control based on human blockage prediction using RGBD cameras for millimeter-wave communications,” Proc. 2015 IEEE Consumer Communications and Networking Conference (CCNC), Las Vegas, Nevada, USA, January 2015, p.152–153

The technique of Non-Patent Document 3 performs rule-based control such as blocking communication with an STA that uses the line-of-sight communication path when the line-of-sight communication path is likely to be blocked, and allocating resources to communication with another STA. ing. In this method, it is necessary to manually create rules according to the environment. For example, in an environment in which the line-of-sight communication path shielding does not affect the communication quality (an environment in which a communication path is created by reflection), it is not necessary to stop the communication even if the line-of-sight communication path is blocked. However, the millimeter-wave communication environment easily changes depending on the arrangement of millimeter-wave base stations and furniture, so it is necessary to reset the settings each time.

In addition, appropriate traffic control is used in environments where it is difficult to manually design appropriate rules, for example, when there are many pedestrians to shield and the arrival is uneven, or when applications such as video and voice calls are different. The policy may change. However, it is not easy to determine an appropriate control measure.

Furthermore, it is necessary to perform various processes such as human body recognition, movement prediction, and line-of-sight channel obstruction prediction from images. Their performance strongly affects the performance of communication control.

In view of the above circumstances, the present invention presents a communication system, a traffic control device, and a traffic control method capable of increasing the total throughput in an environment in which a line-of-sight communication path for wireless communication is temporarily obstructed by a moving obstacle. Is intended to provide.

One aspect of the present invention is to acquire a first communication device, one or more second communication devices that wirelessly communicate with the first communication device, and data transmitted from the first communication device to the second communication device. A communication system including a third communication device and a traffic control device, wherein the traffic control device includes image data of an image of a communication environment between the first communication device and the second communication device, and the first. 3 A value function for calculating the value of an action represented by a combination of throughputs of each of the second communication devices by using information on the amount of untransmitted data to the second communication device stored in the communication device. According to the throughput of each of the action determination unit that calculates the value of each of the plurality of types of actions and determines the action based on the calculated value, and the second communication device represented by the action determined by the action determination unit. Communication between the communication control unit that controls the third communication device so that the data addressed to the second communication device is transmitted to the first communication device, and the second communication device that is controlled by the communication control unit. The cumulative sum of the rewards calculated by the reward calculation unit for different time intervals is the maximum between the reward calculation unit that acquires the status and calculates the reward indicating the degree to which the acquired communication status is improved from the past communication status. With a learning unit that updates the value function as described above , the reward in the time interval is from the total throughput of the first communication device in the time interval and from the past time interval of the first communication device. The first value obtained by multiplying each total throughput up to the time interval by a coefficient corresponding to time and then subtracting the weighted average value, or the total throughput of the first communication device in the time interval is the first. A second value normalized by the average throughput of the communication device, or positive when the ratio of the total throughput of the first communication device in the time interval to the average throughput of the first communication device exceeds a predetermined value. When the ratio is equal to or less than the predetermined value, the absolute value is a negative constant value larger than the positive constant value, and the first communication device is the third communication device. It is a communication system that wirelessly transmits the data to the second communication device received from the second communication device.

One aspect of the present invention is to obtain image data of an image of a communication environment between a first communication device and one or more second communication devices and information on the amount of untransmitted data to the second communication device. The value of each of a plurality of types of actions is calculated by a value function that calculates the value of the action expressed as a combination of traffic of each of the second communication devices, and the action is determined based on the calculated value. Communication control that controls communication so that the data from the first communication device to the second communication device is delivered according to the traffic of each of the unit and the second communication device represented by the action determined by the action determination unit. A reward calculation unit that acquires the communication status of the second communication device due to control by the communication control unit and calculates a reward indicating the degree to which the acquired communication status is improved from the past communication status. And a learning unit that updates the value function so that the cumulative sum of the rewards calculated by the reward calculation unit for different time intervals is maximized, and the reward in the time interval is the said in the time interval. The first is obtained by subtracting the weighted average value obtained by multiplying the total throughput of the first communication device from the past time interval to the time interval by a coefficient according to the time and then averaging from the total throughput of the first communication device. Or the second value obtained by normalizing the total throughput of the first communication device in the time interval with the average throughput of the first communication device, or the said with respect to the average throughput of the first communication device. When the ratio of the total throughput of the first communication device in the time interval exceeds the predetermined value, it becomes a positive constant value, and when the ratio is equal to or less than the predetermined value, the absolute value is a negative constant value larger than the positive constant value. It is a traffic control device which is a third value to be a value .

One aspect of the present invention is the above-mentioned traffic control device, and the communication status of the second communication device depends on the throughput in the second communication device or the transmission of the data to the second communication device. Information that represents the time spent.

One aspect of the present invention is the above-mentioned traffic control device, in which the value function is approximated by a deep neural network.

One aspect of the present invention is the above-mentioned traffic control device, in which the image data used in the value function normalizes pixel values after reducing the resolution of each of a plurality of image data captured at different timings. It is the data that was done.

One aspect of the present invention is the above-mentioned traffic control device, in which information on the amount of data not transmitted to the second communication device used in the value function is not transmitted to each of the plurality of second communication devices. It is the information which arranged the vector which expressed the said data amount by One-hot expression.

One aspect of the present invention is the above-mentioned traffic control device, and the image data is depth image data.

One aspect of the present invention is a throughput control method in a throughput control device that controls wireless communication between a first communication device and one or more second communication devices, wherein the traffic control device is the first communication. A combination of the throughput of each of the second communication devices by using the image data of the communication environment between the device and the second communication device and the information of the data amount of the untransmitted data addressed to the second communication device. The action determination step in which the value of each of a plurality of types of actions is calculated by the value function for calculating the value of the action expressed as, and the action is determined based on the calculated value, and the action determined in the action determination step. According to the throughput of each of the second communication devices represented by, a communication control step for controlling communication so that the data from the first communication device to the second communication device is delivered, and control by the communication control step are performed. The communication status of the second communication device is acquired, and the reward calculation step for calculating the reward indicating the degree of improvement of the acquired communication status from the past communication status is calculated for different time intervals in the reward calculation step. A learning step of updating the value function so that the cumulative sum of the rewards is maximized is executed, and the reward in the time interval is obtained from the total throughput of the first communication device in the time interval. The first value obtained by subtracting the weighted average value obtained by multiplying the total throughput of the first communication device from the past time interval to the time interval by a coefficient corresponding to the time, or the said in the time interval. A second value obtained by normalizing the total throughput of the first communication device with the average throughput of the first communication device, or the total of the first communication device in the time interval with respect to the average throughput of the first communication device. When the ratio of the throughput of is more than the predetermined value, it becomes a positive constant value, and when the ratio is equal to or less than the predetermined value, the absolute value becomes a negative constant value larger than the positive constant value. This is a traffic control method.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to increase the total throughput in an environment where the line-of-sight communication path for wireless communication is temporarily obstructed by a moving obstacle.

It is a figure which shows the structural example of the wireless communication system by one Embodiment of this invention. It is a flow chart which shows the flow of the process of the traffic control apparatus by the same embodiment. It is a figure for demonstrating the episode by the same embodiment. It is a figure which shows the processing from the camera image to the input data by the same embodiment. It is a figure which shows the processing from the file remaining amount information to the input data by the same embodiment. It is a figure which shows the layer design of the behavior evaluation function by the same embodiment. It is a figure which shows the specification of the simulation evaluation of the traffic control apparatus by the same embodiment. It is a figure which shows the simulation evaluation result of the traffic control apparatus by the same embodiment. It is a figure which shows the configuration example of the wireless communication system to be controlled. It is a functional block diagram of the traffic control device by the prior art.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
The traffic control device of the present embodiment uses deep reinforcement learning in order to solve the conventional problems. The traffic control device of the present embodiment uses a camera image and a traffic buffer as "states", and acquires control appropriate for the "state" by trial and error. Reinforcement learning is a type of machine learning in which an agent who is the main actor acts on the environment through trial and error, and the behavior is rewarded by the environment to acquire better measures. The agent acts according to a value function that represents the reward expected from the "state" and updates this value function with the obtained reward. In deep reinforcement learning, a function approximation is performed using a neural network such as a convolutional neural network (CNN) for this value function. As a result, in addition to being applicable to a problem in which the number of states is enormous, the use of a convolutional layer is effective for a problem in which an image is input.

FIG. 1 is a diagram showing a communication system 1 according to an embodiment of the present invention. The communication system 1 includes an access point (AP) 2, a radio station (STA) 3, a proxy server 4, a traffic control device 5, and an image pickup device 6. Of the N STAs (N is an integer of 1 or more), the nth STA3 (n is an integer of 1 or more and N or less) is referred to as STA-n. Further, in the figure, the traffic control device 5 is mounted on the proxy server 4. The communication system 1 shown in the figure has a configuration in which the conventional traffic control device shown in FIG. 10 is replaced with the traffic control device 5.

AP2 wirelessly communicates with one or more STA3s. The AP2 wirelessly transmits a packet addressed to the STA 3 received by the proxy server 4 from a communication device connected via the Internet 7. Further, the AP2 wirelessly receives a packet addressed to a communication device connected via the Internet 7 from the STA 3 and transmits the packet to the proxy server 4. The proxy server 4 communicates via the Internet 7 on behalf of the STA 3. The image pickup apparatus 6 is, for example, an RGB-D camera. The RGB-D camera captures an RGB image (color image) and a depth image. The image pickup apparatus 6 captures an image of the environment including the wireless line-of-sight communication path between the AP2 and the plurality of STAs 3 and its surroundings at a predetermined cycle. The image pickup device 6 transmits the camera image, which is the data of the captured image, to the traffic control device 5.

The proxy server 4 includes a first communication unit 41, a storage unit 42, a second communication unit 43, and a traffic control device 5. The first communication unit 41 receives the packet of the file addressed to the STA3 received via the Internet 7, and writes the packet to the storage unit 42 separately for each STA3. The storage unit 42 has a plurality of file buffers. The file addressed to the STA3 is stored in the file buffer allocated to the STA3. Multiple file buffers can be assigned to one STA3. An upper limit may be set for the file buffer that can be allocated to one STA3. In this embodiment, three file buffers can be allocated to one STA3. The second communication unit 43 reads the file addressed to the STA 3 from the storage unit 42 and transmits it to the AP2 under the control of the traffic control device 5.

The traffic control device 5 includes a reinforcement learning unit 51, a reward calculation unit 52, and a communication control unit 53. The reinforcement learning unit 51 includes a processing unit 511, an action determination unit 512, and a learning unit 513. The action determination unit 512 and the learning unit 513 are processing units of the deep reinforcement learning algorithm. The processing unit 511 processes the camera image input from the image pickup apparatus 6 and the traffic buffer information into a data format suitable for processing, and outputs the data to the processing unit of the deep reinforcement learning algorithm. The action determination unit 512 outputs the traffic control signal as the "action" based on the "state" including the camera image processed in the data format and the traffic buffer information. The traffic buffer information is the amount of untransmitted data to each STA3 stored in the proxy server 4. In this embodiment, the remaining amount of the file is used as the traffic buffer information. The remaining amount of the file is the capacity of the untransmitted file addressed to each STA3 stored in the storage unit 42. The learning unit 513 learns a better control method based on the reward calculated by the reward calculation unit 52 for the output "behavior".

The reward calculation unit 52 outputs a reward designed according to the purpose from the throughput and traffic buffer information of each STA3, or a part thereof. The communication control unit 53 controls the second communication unit of the proxy server 4 so as to deliver the file addressed to the STA 3 while scheduling the traffic between the AP2 and each STA3. This is because, in millimeter-wave communication, a practical example of transmitting a large-capacity file is assumed by taking advantage of its high-speed communication.

The traffic control device 5 may have any one or more functional units of the first communication unit 41, the storage unit 42, and the second communication unit 43 of the proxy server 4. Further, the first communication unit 41 and the communication control unit 53 may be the same functional unit, the second communication unit 43 and the communication control unit 53 may be the same functional unit, and the first communication unit 41 and the second communication unit may be the same. 43 and the communication control unit 53 may be the same functional unit. Further, the traffic control device 5 may be an external device connected to the proxy server 4 by a communication network. Further, one or more arbitrary functional units of the first communication unit 41, the storage unit 42, the second communication unit 43, the reinforcement learning unit 51, the reward calculation unit 52, and the communication control unit 53 are proxied. It may be realized in cooperation with the server 4 and the traffic control device 5.

FIG. 2 is a flow chart showing a processing flow of the traffic control device 5.
When the traffic control device 5 is activated, the image pickup device 6 captures a communication environment at regular time intervals, generates a camera image, and transmits the camera image to the reinforcement learning unit 51 (step S1). On the other hand, the communication control unit 53 acquires the remaining amount of files in the file buffer of each STA 3 and transmits them to the reinforcement learning unit 51 (step S2). The processing unit 511 preprocesses the data received from each of the image pickup device 6 and the communication control unit 53 according to the design of the deep reinforcement learning, and then inputs the data to the action determination unit 512 (step S3).

Since a neural network is used as a value function in deep reinforcement learning, the processing unit 511 processes the camera image and file remaining amount information into input data suitable for the designed neural network. Examples of the neural network of this value function include a simple one having only a fully connected layer and a neural network including a convolutional layer often used in the field of image recognition. As an example, when the value function is a neural network with only fully connected layers, the processing unit 511 lowers the resolution of the depth image of the camera image and then converts it into one-dimensional data, and sets each depth value from 0 to 1. Normalize to. Further, the processing unit 511 discretizes the capacity of the file remaining in the file buffer of each STA 3 to generate the file remaining amount information expressed as One-Hot, and uses it as input data. The One-Hot expression is a vector expression in which only a certain element is 1 and the other elements are 0. Each element of the vector representing the file capacity corresponds to the range of the file capacity, 1 is set for the element corresponding to the file capacity remaining in the file buffer, and 0 is set for the other elements.

The action determination unit 512 uses the deep reinforcement learning algorithm to determine the communication traffic (“action” of reinforcement learning) of each STA3 based on the output result of the value function (step S4). Specifically, the action determination unit 512 has a state transition in which each of the possible "actions" has the highest value in the "state" of the camera image and the file remaining amount information of the file buffer. "Action" (traffic of each STA3) that causes the above is preferentially adopted. The action determination unit 512 transmits the traffic traffic control information of the determined communication of each STA 3 to the communication control unit 53. Upon receiving this, the communication control unit 53 controls the second communication unit 43 of the proxy server 4 to set the file held in the file buffer as a packet and transmit it to the AP2 according to the traffic control information (step S5). ).

After transmitting the packet, the communication control unit 53 acquires the remaining amount of files in the buffer destined for each STA3 and the throughput of each STA3 at that time, and transmits them to the reward calculation unit 52 (step S6). The reward calculation unit 52 calculates the reward using the received file remaining amount and the throughput information (step S7). Rewards are designed for the detailed purpose of traffic control. Examples of detailed purposes include maximizing the total throughput of AP2, minimizing the total file transmission time, and the like. When the purpose is to maximize the total throughput of AP2, the reward calculation unit 52 determines the action of the action determination unit 512, and each time the communication control unit 53 acts based on the decision, the AP2 at that time is used. Give total throughput as a reward. When the purpose is to minimize the total file transmission time, in the reward calculation unit 52, every time the action determination unit 512 determines an action and the communication control unit 53 acts based on the determination, the file is a proxy server 4 A negative constant is given as a reward from the time of arrival at to the completion of file transmission to STA3. That is, the cumulative sum of rewards is proportional to the total file transmission time.

For example, when the purpose is to maximize the total throughput of AP2, the reward rt in the time step _t is calculated by the following equation (1).

T _t is the total throughput in the time step t, and c (t) is a coefficient corresponding to the time parameter t. The term Σ is the value obtained by weighted averaging the total throughput so far by parameters such as time. For example, each c (i) may be determined in the second term of the equation (1) so as to obtain a weighted average throughput according to time. Further, assuming that c (i) = 1 (i is an integer of _t or less), the reward rt is calculated by the following equation (2).

Further, the reward may be a throughput normalized by the average throughput _Tt ￣ of the entire AP2 as shown in the equation (3), or may be a difference of the normalized throughput as shown in the equation (4).

Further, as in the following equation (5), a large negative reward may be given when the attenuation rate from the average throughput falls below a certain value α.

Further, the throughput in the equations (1) to (5) may be replaced with the physical transmission speed of millimeter wave communication.

The reward calculation unit 52 transmits the calculated reward to the reinforcement learning unit 51. The reinforcement learning unit 51 advances learning by updating the value function by the deep reinforcement learning algorithm based on the notified reward (step S8).

By repeating this series of operations, the reinforcement learning unit 51 determines the traffic of each STA3 while proceeding with learning so that the cumulative sum of the input rewards is maximized. Therefore, as the learning progresses, the traffic control method adapted to the environment in which the traffic control device 5 is installed is automatically acquired.

The traffic control device 5 performs the above processing based on the result of performing the plurality of episodes, and learns the behavior evaluation function. FIG. 3 is a diagram for explaining an episode. The episode represents a series of flows until all the files in the file buffer in the storage unit 42 are transmitted. The proxy server 4 transmits the files stored in the file buffer to each STA3 via the AP2 under the control of the communication control unit 53 of the traffic control device 5, and completes the transmission of all the files in the file buffer. At this point, one episode ends. No files are added in the middle of one episode. As the episode progresses, so does the learning of the traffic control device 5 of the present embodiment. The maximum number of learning may be determined in advance, and the learning may be terminated when the maximum number of episodes is reached.

An example of input data and layer design of a deep neural network (CNN) used as a value function will be described.
FIG. 4 is a diagram showing processing from a camera image to input data in step S3. The reinforcement learning unit 51 compresses the depth image data included in the past five camera images in one second into two-dimensional image data of 20 × 20 pixels, respectively. The reinforcement learning unit 51 uses a 5-channel two-dimensional image obtained by compressing each of the five depth image data as input data to the CNN.

FIG. 5 is a diagram showing processing from the file remaining amount information in step S3 to the input data. First, the remaining amount of each file is discretized in multiple stages. Here, the case where the maximum value of the file capacity is 2000 Mbit (megabit) and the file is discretized in 10 steps is taken as an example. In this case, each element of the One-Hot representation vector used as the file remaining amount information is set to [(0-200 Mbit), (200-400 Mbit), (400-600 Mbit), (600-800 Mbit), ..., (1800). -2000 bits)]. When the file remaining amount of STA-n (n is an integer of 1 or more and N or less) acquired from the storage unit 42 has a capacity of 700 Mbit, the file remaining amount information is vector [0,0,0,1,0,0,0. , 0,0,0]. Vectors representing the file remaining amount information generated for the reinforcement learning unit 51, STA-1, STA-2, ..., STA-N are arranged and combined to be input data.

FIG. 6 is a diagram showing a layer design of CNN. Note that "Affine, ab" represents an operation in which an a-dimensional vector is input to the fully connected layer and a b-dimensional vector is output. “K × l 2D Conversion, ab” represents an operation in which the input of a channel is convolved into b channel by a two-dimensional filter of k × l and output. Further, "k × l 2D Max Pooling" represents an operation in which the input is divided into grids having a size of k × l and the maximum value of each grid is output as a representative value. "ReLU" represents an operation to be input to the activation function ReLU (Rectified Linear Units). The activation function ReLU converts a negative value to 0.

In the input layer, a five-channel two-dimensional image (5 Channels 2D Image) is generated by the process shown in FIG. Further, in the input layer, the remaining amount of the file of each STA3 is converted into a vector of One-hot expression by the process shown in FIG. 4, and combined to generate a 60-dimensional vector.

The hidden layer includes layers 1a to 8a, layers 1b to 2b, and nine layers that input the outputs of the layers 8a and 2b.
In the 1a layer, a 5 channel 2D image is convoluted by a 5 × 5 2D filter and output as 20 channels. In the 2a layer, the output of the 1a layer of 20 channels is input to the activation function ReLU, and the negative value is removed. In the 3a layer, the output of the 2a layer of 20 channels is divided into 2 × 2 grids, and the maximum value of each grid is output. In the 4a layer, the output of the 3a layer of 20 channels is convoluted by a 5 × 5 two-dimensional filter to make 50 channels and output. In the 5a layer, the output of the 4a layer of 50 channels is input to the activation function ReLU, and the negative value is removed. In the 6a layer, the output of the 5a layer of 50 channels is divided into 2 × 2 grids, and the maximum value of each grid is output. In the 7a layer, the 1250-dimensional vector of the 6a layer is input to the fully connected layer, and the 500-dimensional vector is output. In the 8a layer, the output of the 7a layer is input to the activation function ReLU, and the negative value is removed.

On the other hand, in the 1b layer, the 60-dimensional vector obtained based on the remaining amount of the file of each STA3 is input to the fully connected layer, and the 100-dimensional vector is output. It is assumed that the multiplication of the number N of the STA3 and the number of elements of the vector of the One-Hot expression is 60. In the 2b layer, the output of the 1b layer is input to the activation function ReLU, and the negative value is removed.

In the 9th layer, a 600-dimensional vector including the output of the 8a layer and the output of the 2b layer is input to the fully connected layer, and the evaluation value of each action is obtained. The output layer outputs the evaluation value of each action. Each action may be a combination of turning on or off communication with each STA3, or may be a combination of traffic amounts of each of N STA3s. In the figure, the combination of turning on or off the communication with each of the two STA3s is excluded from the combination of turning off both of them. That is, there are three types of actions in which (STA-1, STA-2) is (ON, ON), (ON, OFF), and (OFF, ON). In order to obtain the evaluation values of each of these three types of actions, a three-dimensional vector is output from the nine layers.

Regarding the Conversion layer, it is expected that a filter for extracting features from an image will be learned near the input layer, and a filter for predicting values from features will be learned near the output layer. Be expected. ReLU is widely used as an activation function. It is known that ReLU has an empirically faster learning speed and higher performance than other activation functions (sigmoid function, etc.). Further, the Max Polling layer is used to shorten the learning time by reducing the number of parameters increased by passing through the Conversion layer. The Affine layer is used with the expectation that the value will be predicted from the features extracted by CNN. It is empirically known that the learning time can be expected to be shortened as compared with the layer design consisting only of CNN.

The learning unit 513 updates the CNN used as a value function. Specifically, the learning unit 513 updates the weights in the fully connected layer based on the reward calculated by the reward calculation unit 52. For example, when the action determination unit 512 obtains the result that the communication between AP2 and STA-1 is turned on and the communication between AP2 and STA-2 is turned off, the communication control unit 53 turns on only the communication between AP2 and STA-1. Control to. For example, the communication control unit 53 controls the second communication unit 43 of the proxy server 4 so as to output the file addressed to STA-1 to AP2 and not to output the file addressed to STA-2 to AP2. Alternatively, the control signal may be transmitted to the AP2 via the second communication unit 43 of the proxy server 4 so as to communicate with the STA-1 and not to communicate with the STA-2. However, even with such control, the state of the propagation path between AP2 and STA-1 is actually poor, such as shielding occurring between AP2 and STA-1 and reflection occurring in multipath. In that case, the communication speed becomes low. As an extreme example, if there is a metal wall between AP2 and STA-1 and no radio waves reach STA-1, the throughput will be 0 Mbit / s even when communication is ON. The learning unit 513 can perform learning to control ON / OFF of each STA3 so that such a situation does not occur.

According to the traffic control device 5 of the present embodiment, traffic control can be performed by deep reinforcement learning using a camera image as an input, and it is possible to automatically adapt to various communication environments and effectively use the radio band. In addition, even when the installation environment of the communication terminal or camera changes, it becomes possible to automatically control the traffic by adapting to the changed environment. In particular, it is useful in a situation where human body shielding may occur in a communication system in which a wireless LAN (Local Area Network) router equipped with a millimeter wave communication function and a plurality of millimeter wave communication terminals are connected. It is also possible to deal with changes in the installation environment of wireless LAN routers and millimeter-wave communication terminals.

A simulation evaluation using the measured data of the traffic control device 5 will be described. FIG. 7 is a diagram showing specifications of simulation evaluation. In this simulation evaluation, assuming a case where two STA3s are connected to one AP2, a round robin method is used in which the transmission destination is alternately switched between the case where the throughput control of the present embodiment is performed and the case where the file transmission is completed. The total throughput in AP when controlled was obtained. AP2 is a millimeter wave AP. For the line-of-sight communication of millimeter-wave communication used in the simulation, the throughput at the time of shielding and the camera image used the values measured from the actual machine experiment. As the camera image, the data of the image taken by the RGB-D camera was used. In addition, commercially available AP2 and STA3 were also used.

FIG. 8 is a diagram showing simulation evaluation results. The figure shows the transition of the total throughput with respect to the number of episodes when the traffic control of the present embodiment is performed and the control is performed by the round robin method. The total throughput of AP2 in the graph of the figure is displayed as the time average of the total throughput of AP2 in each episode. In this simulation, a file is initially given to the file buffer of the proxy server 4 in a random size, and the file is transmitted to each STA 3 through AP2. One episode ends when all the files in the file buffer have been sent. From the evaluation results shown in the figure, as the episode progresses and the learning of the traffic control device 5 progresses, the total when the traffic control of the present embodiment is performed is larger than the throughput when the control is performed by the round robin method. It can be seen that the throughput is exceeded.

According to the embodiment described above, the communication system transmits from the first communication device, one or more second communication devices that wirelessly communicate with the first communication device, and the first communication device to the second communication device. It has a third communication device for acquiring data and a traffic control device. For example, the first communication device is AP2, the second communication device is STA3, and the third communication device is proxy server 4.

The traffic control device has an action determination unit, a communication control unit, a reward calculation unit, and a learning unit. The action determination unit includes image data that captures the communication environment between the first communication device and the second communication device, and information on the amount of untransmitted data stored in the third communication device to the second communication device. Is used to calculate the value of each of a plurality of types of actions by a value function that calculates the value of the action represented by the combination of traffic of each of the second communication devices. The value function may be approximated by a deep neural network. In this case, the image data input to the deep neural network is data obtained by reducing the resolution of each of the plurality of image data captured at different timings and then normalizing the pixel values. Further, for the information of the amount of untransmitted data to the second communication device input to the deep neural network, a vector representing the amount of untransmitted data to each of the plurality of second communication devices by One-Hot expression is arranged. Information. The action decision unit decides the action based on the calculated value.

The communication control unit controls the third communication device so as to transmit data addressed to the second communication device to the first communication device according to the traffic of each of the second communication devices represented by the action determined by the action determination unit. The reward calculation unit acquires the communication status of the second communication device due to the control by the communication control unit, and calculates a reward indicating the degree to which the acquired communication status is improved from the past communication status. The communication status of the second communication device represents the throughput in the second communication device or the time required for transmitting data to the second communication device. The learning department updates the value function based on the calculated reward. The first communication device wirelessly transmits data to the second communication device received from the third communication device to the second communication device.

The function of the traffic control device 5 in the above-described embodiment may be realized by a computer. In that case, the traffic control device 5 is realized by recording a program for realizing this function on a computer-readable recording medium, causing the computer system to read the program recorded on the recording medium, and executing the program. May be good. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, and a storage device such as a hard disk built in a computer system. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or a client in that case. Further, the above-mentioned program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system.

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and the design and the like within a range not deviating from the gist of the present invention are also included.

It can be used for communication systems that perform wireless communication.

1 ... Communication system, 2 ... Access point, 3 ... Radio station, 4 ... Proxy server, 5 ... Traffic control device, 6 ... Imaging device, 7 ... Internet, 41 ... First communication unit, 42 ... Storage unit, 43 ... No. 2 Communication Department, 51 ... Reinforcement Learning Department, 52 ... Reward Calculation Department, 53 ... Communication Control Department, 511 ... Processing Department, 512 ... Action Decision Department, 513 ... Learning Department

Claims (8)

A first communication device, one or more second communication devices that wirelessly communicate with the first communication device, a third communication device that acquires data transmitted from the first communication device to the second communication device, and a traffic. A communication system having a control device,
The traffic control device is
Image data that captures the communication environment between the first communication device and the second communication device, and information on the amount of untransmitted data to the second communication device stored by the third communication device. The value of each of a plurality of types of actions is calculated by a value function that calculates the value of the action represented by the combination of traffic of each of the second communication devices, and the action is determined based on the calculated value. The decision department and
Communication control that controls the third communication device so as to transmit the data addressed to the second communication device to the first communication device according to the traffic of each of the second communication devices represented by the action determined by the action determination unit. Department and
A reward calculation unit that acquires the communication status of the second communication device due to control by the communication control unit and calculates a reward indicating the degree to which the acquired communication status is improved from the past communication status.
A learning unit that updates the value function so that the cumulative sum of the rewards calculated by the reward calculation unit for different time intervals is maximized .
Equipped with
The reward in the time interval is
A weighted average value obtained by multiplying the total throughput of the first communication device from the past time section of the first communication device to the time section by a coefficient corresponding to the time from the total throughput of the first communication device in the time interval. First value after subtracting
Alternatively, a second value obtained by normalizing the total throughput of the first communication device in the time interval with the average throughput of the first communication device.
Alternatively, when the ratio of the total throughput of the first communication device in the time interval to the average throughput of the first communication device exceeds a predetermined value, it becomes a positive constant value, and when the ratio is equal to or less than the predetermined value, it becomes a positive constant value. It is a third value whose absolute value is a negative constant value larger than the positive constant value.
The first communication device wirelessly transmits the data to the second communication device received from the third communication device to the second communication device.
Communications system. The second communication is performed by using the image data of the communication environment between the first communication device and one or more second communication devices and the data amount information of the untransmitted data addressed to the second communication device. An action decision unit that calculates the value of each of multiple types of actions by a value function that calculates the value of the action expressed as a combination of traffic of each device, and determines the action based on the calculated value.
A communication control unit that controls communication so that the data from the first communication device to the second communication device is delivered according to the traffic of each of the second communication devices represented by the action determined by the action determination unit.
A reward calculation unit that acquires the communication status of the second communication device due to control by the communication control unit and calculates a reward indicating the degree to which the acquired communication status is improved from the past communication status.
A learning unit that updates the value function so that the cumulative sum of the rewards calculated by the reward calculation unit for different time intervals is maximized .
Equipped with
The reward in the time interval is
A weighted average value obtained by multiplying the total throughput of the first communication device from the past time section of the first communication device to the time section by a coefficient corresponding to the time from the total throughput of the first communication device in the time interval. First value after subtracting
Alternatively, a second value obtained by normalizing the total throughput of the first communication device in the time interval with the average throughput of the first communication device.
Alternatively, when the ratio of the total throughput of the first communication device in the time interval to the average throughput of the first communication device exceeds a predetermined value, it becomes a positive constant value, and when the ratio is equal to or less than the predetermined value, it becomes a positive constant value. A third value whose absolute value is a negative constant value larger than the positive constant value.
Traffic control device. The communication status of the second communication device is information representing the throughput in the second communication device or the time required for transmitting the data to the second communication device.
The traffic control device according to claim 2. The value function is approximated by a deep neural network,
The traffic control device according to claim 2 or 3. The image data used in the value function is data obtained by normalizing pixel values after reducing the resolution of each of a plurality of image data taken at different timings.
The traffic control device according to claim 4. For the information on the amount of untransmitted data addressed to the second communication device used in the value function, a vector representing the amount of untransmitted data addressed to each of the plurality of the second communication devices by the One-Hot representation is arranged. Information,
The traffic control device according to claim 4. The image data is depth image data.
The traffic control device according to any one of claims 2 to 6. A traffic control method in a traffic control device that controls wireless communication between a first communication device and one or more second communication devices.
The traffic control device
Using image data that captures the communication environment between the first communication device and the second communication device and information on the amount of untransmitted data addressed to the second communication device, each of the second communication devices. An action decision step that calculates the value of each of multiple types of actions by a value function that calculates the value of the action expressed as a combination of traffic, and determines the action based on the calculated value.
A communication control step that controls communication so that the data from the first communication device to the second communication device is delivered according to the traffic of each of the second communication devices represented by the action determined in the action determination step. ,
A reward calculation step for acquiring the communication status of the second communication device due to the control performed by the communication control step and calculating a reward indicating the degree to which the acquired communication status is improved from the past communication status.
A learning step that updates the value function so that the cumulative sum of the rewards calculated for different time intervals in the reward calculation step is maximized .
And run
The reward in the time interval is
A weighted average value obtained by multiplying the total throughput of the first communication device from the past time section of the first communication device to the time section by a coefficient corresponding to the time from the total throughput of the first communication device in the time interval. First value after subtracting
Alternatively, a second value obtained by normalizing the total throughput of the first communication device in the time interval with the average throughput of the first communication device.
Alternatively, when the ratio of the total throughput of the first communication device in the time interval to the average throughput of the first communication device exceeds a predetermined value, it becomes a positive constant value, and when the ratio is equal to or less than the predetermined value, it becomes a positive constant value. A third value whose absolute value is a negative constant value larger than the positive constant value.
Traffic control method.

Images