JP7393765B2 - Wireless communication device, beam direction control device, beam direction control method and program - Google Patents

Description

The present invention relates to a wireless communication device, a beam direction control device, a beam direction control method, and a program.

In recent years, millimeter wave wireless communication technology that enables high-speed communication has attracted attention. In addition to wireless LAN (IEEE 802.11ad, 802.11ay)/wireless PAN (IEEE 802.15.3e), international standards have been established for wireless communication systems that use the millimeter wave band, such as fifth-generation mobile communication systems. There is.

A millimeter wave band with a frequency of several tens of GHz or more has a characteristic that propagation attenuation is large compared to a microwave band with a frequency of 6 GHz or less. In order to compensate for such propagation attenuation, the standardized wireless communication system described above incorporates a technique for improving wireless communication quality by forming a directional beam using an array antenna. The beam of the array antenna formed in a specific direction (hereinafter referred to as beam direction) has made millimeter wave band wireless communication systems applicable to high-speed wireless transmission outdoors.

There is a wireless communication system in which a macro cell (Macro-Cell) and a pico-cell (Pico-Cell) are connected by applying a millimeter wave wireless communication system outdoors (for example, see Non-Patent Document 1). Pico cells are generally installed outdoors near end users (on utility poles, overhead wires, etc.). In such outdoor environments, installed transmitters are frequently displaced due to external factors such as wind. When the transmitter is displaced, the beam directions that should be opposed during transmission and reception become unopposed, making it impossible to compensate for propagation attenuation, resulting in frequent deterioration of wireless communication quality.

An example in which beam directions become unopposed due to external factors such as wind will be described with reference to FIG. 8. The wireless communication device 91 is attached on an overhead wire 94 between a utility pole 92 and a utility pole 93. On the other hand, a wireless communication device 95 to which the wireless communication device 91 communicates is installed in a building 96 . Beam B91 of wireless communication device 91 is set to face beam B95 of wireless communication device 95. However, in FIG. 8, when the wireless communication device 91 is pushed by the force of the wind, the direction of the beam B91 deviates from the beam B95 of the opposing wireless communication device 95, resulting in a decrease in wireless communication quality. There is. In this way, in a wireless communication device using a miniaturized antenna, the beam direction swings in irregular motion due to external forces such as wind, resulting in beam misalignment with the opposing device.

In a typical millimeter wave wireless communication system, in order to improve wireless communication quality, it is necessary to align the beam directions of both communicating wireless communication devices. As a method for aligning the beam directions, search all the beam directions that can be changed by both wireless communication devices, identify the beam direction that provides the best wireless communication quality (for example, the direction where the received power is maximum), and then A possible method is to notify the communication partner of the identified beam direction. However, in an environment where there is frequent vibration due to wind, in order to maintain wireless communication quality, both wireless communication devices must frequently search for beam directions to obtain the best wireless communication quality. When beam direction searches as described above are performed frequently, a reduction in available wireless communication resources (time slots, etc.) becomes a major problem.

As mentioned above, in a millimeter wave wireless communication system, in order to improve the quality of wireless communication, it is necessary to make the beam directions of both sides facing each other. Increased need to search for direction. However, there is a problem in that during the period when the wireless communication device is performing this search, resources available for wireless communication decrease.

On the other hand, there is a technique for controlling, based on learning, a millimeter wave wireless communication system in which wireless communication quality may be affected by an external environmental factor such as shadowing by a human body (see Non-Patent Document 2). An overview of the configuration of a wireless communication system assumed in this technology and external factors that affect the wireless communication quality of the wireless communication system will be described using FIG. 9. As shown in FIG. 9, two millimeter wave access points (hereinafter referred to as AP) 98 are installed indoors. These two AP98s are referred to as AP98-1 and AP98-2. A terminal station (hereinafter referred to as STA) 99 is capable of wireless communication with AP 98-1 or AP 98-2 by changing the beam direction. The pedestrian 97 moves along a random route W between the AP 98-1 and the STA 99 or between the AP 98-2 and the STA 99. The pedestrian 97 can also change the direction and speed of movement. If either the beam of AP98-1 or STA99, or both beams are blocked by the pedestrian 97 (human body) at a certain moment when the pedestrian 97 crosses between AP98-1 and STA, a shadow occurs. wireless communication quality deteriorates significantly. Such a human body shielding event becomes an external cause that affects wireless communication quality.

In Non-Patent Document 2, in order to overcome the influence of external factors such as the above-mentioned human body shielding, a control method is disclosed in which the position of a pedestrian is grasped from a camera image, and based on the position, an AP with which communication is possible that is optimal for the STA is selected. is proposed. That is, position information of a pedestrian is acquired from a camera image using an event in which the directional beams of the communicating AP and STA are blocked by a human body as an external cause. Then, after predicting the occurrence of shielding based on the acquired position information, control is performed so that the STA is handed over to an AP that is not shielded by the human body. In the case of the wireless communication system shown in FIG. 9, an example of this operation is as follows. That is, if STA 99 predicts that a pedestrian 97 crossing between AP 98-1 and STA 99 will block the area based on the image while communicating with AP 98-1, it changes the beam direction in advance to point toward AP 98-2. (handover control). This makes it possible to maximize wireless communication quality observed over a long period of time (cumulative value of throughput, etc.) and avoid deterioration in wireless communication quality due to human body shielding.

S. Hur, et al., “Millimeter Wave Beamforming for Wireless Backhaul and Access in Small Cell Networks”, Fig. 1. Multi-tiered cell using wireless backhaul, IEEE Transactions on Communications, Vol. 61, No. 10, Oct. 2013 Koda et al., “Application of reinforcement learning to millimeter wave communication handover control using position information of the occluding person”, Institute of Electronics, Information and Communication Engineers, IEICE Technical Report SR2017-131, 2018, p.95-102

The control method of Non-Patent Document 2 described above only considers the phenomenon of human body shielding in an indoor environment as an external environmental factor that affects wireless communication quality. This is a control method for a relatively simple external cause such as the plane movement of a single pedestrian. In this way, in the technique of Non-Patent Document 2, since learning is performed for relatively simple external causes, the correspondence between environmental states and control methods can be successfully learned without using enormous processing resources.

However, in an outdoor environment as shown in FIG. 8, there are many external factors that affect wireless communication quality. For example, regarding wind conditions, it is necessary to consider conditions such as instantaneous wind speed, wind direction, atmospheric density, and air drag coefficient. Furthermore, regarding the installation conditions of the wireless communication device, it is necessary to consider conditions such as the length of the overhead wire, the material of the overhead wire, and the height from the ground. It is extremely difficult to comprehensively learn all about such a large number of external causes. In the control method based on learning shown in Non-Patent Document 2, when the external cause frequently changes, it is not possible to cope with the new external cause, and it becomes necessary to perform learning again. In other words, if there is an unlearned external cause, the beam directions can become unopposed. In other words, the control method itself based on this learning is considered to increase the number of failures in beam direction control in an environment where many external factors exist.

In view of the above circumstances, an object of the present invention is to provide a wireless communication device, a beam direction control device, a beam direction control method, and a program that can reduce failures in beam direction control even in an environment where complex external factors change.

One aspect of the present invention is a wireless communication device that can control a beam direction, and includes a wireless communication unit that forms a beam and performs wireless communication, and a sensor that acquires environmental status information that is information about the installation environment of the device. a wireless communication quality monitoring unit that acquires wireless communication quality information indicating the quality of wireless communication by the wireless communication unit; and a beam direction control unit that outputs a beam direction control instruction to the wireless communication unit. , the beam direction control unit controls the beam direction to improve the quality of wireless communication according to the environmental state information, using the environmental state information and the wireless communication quality information before and after the beam direction is controlled. learning a beam control policy indicating a method, determining the beam control policy according to the environmental state information based on the learning result, and instructing the radio communication unit to control the beam direction according to the determined beam control policy; a first learning unit that outputs, environmental status information generated according to an information generation policy indicating a calculation for generating environmental status information, and the first learning unit outputs according to the generated environmental status information. learning an information generation policy for generating environmental state information that degrades the quality of wireless communication using the wireless communication quality information before and after the beam direction was controlled based on the control instruction, and learning the learned information. a second learning unit that generates environmental state information based on a generation policy; and a second learning unit that generates environmental state information based on a generation policy; A wireless communication device includes a switching unit that switches input to a learning unit.

One aspect of the present invention provides environmental state information that is information regarding the installation environment of a wireless communication device that can control a beam direction, and wireless communication that indicates the quality of wireless communication before and after the beam direction of the wireless communication device is controlled. The quality information is used to learn a beam control policy that indicates a beam direction control method that improves the quality of wireless communication according to the environmental state information, and based on the learning result, the beam direction is adjusted according to the environmental state information. a first learning unit that determines a control strategy and outputs a beam direction control instruction to the wireless communication device according to the determined beam control strategy; and an information generation strategy that indicates a calculation for generating environmental state information. Using the generated environmental state information and the wireless communication quality information before and after the beam direction is controlled based on the control instruction outputted by the first learning unit according to the generated environmental state information. a second learning unit that learns an information generation strategy for generating environmental status information that degrades the quality of wireless communication, and generates environmental status information based on the learned information generation strategy; A beam direction control device comprising: a switching unit that switches which of the environmental state information acquired by a sensor and the environmental state information generated by the second learning unit is input to the first learning unit. be.

One aspect of the present invention is a beam direction control method executed by a wireless communication device capable of controlling a beam direction, which includes a communication step in which a wireless communication unit forms a beam and performs wireless communication, and a sensor performs the wireless communication. an environmental status information acquisition step of acquiring environmental status information which is information regarding the installation environment of the device; and wireless communication quality information where the wireless communication quality monitoring unit acquires wireless communication quality information indicating the quality of wireless communication by the wireless communication unit. and a beam direction control step in which the beam direction control unit outputs a beam direction control instruction to the wireless communication unit, and the beam direction control step includes the step of controlling the environmental state information and the beam direction. A beam control strategy indicating a beam direction control method that improves the quality of wireless communication according to the environmental state information is learned using the wireless communication quality information before and after the environment state information, and based on the learning result, the wireless communication quality information a first learning step of determining the beam control policy according to state information and outputting a beam direction control instruction according to the determined beam control policy to the wireless communication unit; The environmental state information generated according to the information generation policy indicating the calculation, and the beam direction before and after the beam direction is controlled based on the control instruction outputted in the first learning step according to the generated environmental state information. a second learning step of learning an information generation policy for generating environmental status information that degrades wireless communication quality using the wireless communication quality information, and generating environmental status information based on the learned information generation policy; , a switching step of switching which of the environmental state information acquired in the environmental state information acquisition step and the environmental state information generated in the second learning step is to be used in the first learning step; This is a beam direction control method.

One aspect of the present invention is a program for causing a computer to function as the beam direction control device described above.

According to the present invention, it is possible to reduce failures in beam direction control even in an environment with complex fluctuations due to external factors.

FIG. 1 is a block diagram showing the configuration of a wireless communication device according to an embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of a beam direction control section according to the same embodiment. It is a figure which shows the example of the beam control policy table by the same embodiment. It is a figure which shows the example of the information generation policy table by the same embodiment. FIG. 6 is a flow diagram illustrating an example of processing in a first learning mode of the beam direction control unit according to the embodiment. FIG. 7 is a flow diagram showing an example of processing in a second learning mode of the beam direction control unit according to the embodiment. FIG. 2 is a block diagram showing the hardware configuration of the wireless communication device according to the same embodiment. FIG. 6 is a diagram for explaining an example in which non-opposing beam directions occur. FIG. 2 is a diagram for explaining external factors that affect wireless communication quality.

Embodiments of the present invention will be described in detail below with reference to the drawings.

<Configuration and functions of wireless communication device>
FIG. 1 is a block diagram showing the configuration of a wireless communication device 1 according to an embodiment of the present invention. The wireless communication device 1 uses a directional beam to transmit and receive radio waves to and from another wireless communication device with which it communicates. The communication partner wireless communication device exists in a fixed direction from the wireless communication device 1. The wireless communication device 1 includes a wireless communication section 11 , a wireless communication quality monitoring section 12 , an environment sensor 13 , and a beam direction control section 15 .

The wireless communication unit 11 is comprised of devices such as an array antenna that can change the beam direction of a directional beam, a high frequency circuit for transmitting and receiving radio waves at a predetermined radio frequency, and a signal processing circuit. The wireless communication unit 11 allows the wireless communication device 1 to wirelessly communicate with other wireless communication devices. The wireless communication unit 11 appropriately adjusts the weight of the array antenna so that the radio wave directivity is formed in the beam direction instructed by the control instruction from the beam direction control unit 15. By adjusting the weights, it is possible to adjust the phase of a radio signal input/output to each antenna element using an analog method, or to adjust the amplitude and phase of a radio signal input/output to each antenna element using a digital method. Alternatively, by adjusting the weights, it is possible to combine the analog method described above and the digital method described above to adjust the amplitude or phase of the wireless signal input/output to each antenna element in multiple stages.

The wireless communication quality monitoring unit 12 acquires from the wireless communication unit 11 the beam direction of the directional beam that the own device uses for wireless communication and information regarding the wireless communication quality during the communication period using the beam direction. Information regarding wireless communication quality is, for example, received power, received power-to-noise ratio, and the like. In the following description, for convenience of explanation, received power will be taken up as an example representative of wireless communication quality, but other indicators may be used. The wireless communication quality monitoring unit 12 outputs beam direction information indicating the acquired beam direction and wireless communication quality information indicating the acquired information regarding the wireless communication quality to the beam direction control unit 15.

The environmental sensor 13 is composed of one or more sensing devices. The environmental sensor 13 detects or acquires environmental state information, which is information regarding the environment around the wireless communication device 1 . The environmental state information includes, for example, wind speed, wind direction, rotational speed and acceleration of the wireless communication device 1, height of the installation location of the wireless communication device 1, and the like. The environmental sensor 13 outputs the detected or acquired environmental state information to the beam direction control unit 15.

The beam direction control unit 15 controls the beam direction of a directional beam that the wireless communication device 1 uses for wireless communication. The beam direction control unit 15 monitors the quality of wireless communication by using beam direction information indicating the beam direction that the own device uses for wireless communication and wireless communication quality information obtained during the communication period in which the beam direction is used. Obtained from section 12. Furthermore, the beam direction control unit 15 acquires environmental state information during the above communication period from the environmental sensor 13. Based on the acquired information, the beam direction control unit 15 outputs a control instruction to the wireless communication unit 11 to cause the beam direction to face the communication partner in the next communication period. Setting the beam direction to face the communication partner means adjusting the beam direction of the directional beam so that the maximum received power can be obtained.

<Configuration and function of beam direction control unit>
The beam direction control unit 15 in this embodiment learns a beam control strategy using the environmental state information in a certain communication period and the beam direction used for wireless communication as input conditions, and performs the next communication period based on the learned beam control strategy. This section explains the configuration and functions for controlling the beam direction used. Note that the beam control policy refers to how the beam direction is controlled to improve communication quality.

FIG. 2 is a block diagram showing the detailed configuration of the beam direction control section 15. As shown in FIG. The beam direction control section 15 includes a mode setting section 151 , an environmental state information acquisition section 152 , a first learning section 153 , and a second learning section 154 .

The beam direction control unit 15 operates in two modes: a first learning mode and a second learning mode. The mode setting section 151 sets in which of these two modes the beam direction control section 15 operates. In the first learning mode, the first learning unit 153 learns a beam control strategy using environmental state information actually obtained by the environmental sensor 13. In the first learning mode, the mode setting section 151 controls the second learning section 154 so as not to operate. In the second learning mode, the second learning section 154 generates environmental state information that simulates the surrounding environment that is not actually obtained, and inputs it to the first learning section 153. This provides the first learning unit 153 with an opportunity to learn beam control strategies according to various environmental state information. In the following, the environmental state information generated by the second learning unit 154 will be referred to as pseudo environmental state information.

The mode setting unit 151 is, for example, a physical switch such as a dip switch attached to the wireless communication device 1. Alternatively, the mode setting unit 151 may be realized by software installed in the wireless communication device 1. In this case, the first learning unit 153 may change the mode in response to an instruction from an external control personal computer (PC) or remote control via a network. Alternatively, the mode setting unit 151 switches from the first learning mode to the second learning mode at a predetermined time, and switches from the second learning mode to the first learning mode at another time, according to a scheduler set in advance. The operating mode may be changed, such as switching to learning mode.

The environmental state information acquisition unit 152 inputs environmental state information from the environmental sensor 13 . The environmental state information acquisition unit 152 outputs the input environmental state information to the first learning unit 153 in the first learning mode, and outputs it to the second learning unit 154 in the second learning mode. Which of the environmental status information acquired by the environmental sensor 13 and the pseudo environmental status information generated by the second learning unit 154 is input to the first learning unit 153 by the mode setting unit 151 and the environmental status information acquisition unit 152? Realizes the function as a switching section that switches between

The first learning unit 153 uses the environmental state information and the wireless communication quality information before and after the beam direction is controlled to learn a beam control policy for improving the quality of wireless communication according to the environmental state information, A beam direction control instruction is output to the wireless communication unit 11 based on the learning result. The first learning section 153 includes a beam control policy storage section 1531 and a first cumulative reward storage section 1532. The beam control policy storage unit 1531 stores a beam control policy table. The beam control strategy table shows beam control strategies corresponding to the environmental state information. In this embodiment, the beam control strategy is expressed by the amount of correction from the current beam direction. The first cumulative reward storage unit 1532 stores the first cumulative reward. The first cumulative reward is the sum of the first rewards. The first reward is a value given according to the degree to which wireless communication quality has been improved by the beam control policy. In this embodiment, the larger the degree of improvement, the larger the first reward is given. The first reward may be a tiered value.

In the first learning mode, the first learning unit 153 inputs environmental state information from the environmental state information acquisition unit 152 and receives beam direction information and wireless communication quality information from the wireless communication quality monitoring unit 12. The first learning unit 153 reads out the beam control policy according to the environmental state information and the beam direction from the beam control policy table stored in the beam control policy storage unit 1531. The first learning unit 153 outputs a control instruction to the wireless communication unit 11 to control the beam direction according to the read beam control policy.

The first learning unit 153 inputs wireless communication quality information from the wireless communication quality monitoring unit 12 while wireless communication is being performed with a beam direction changed according to a control instruction based on a beam control policy. The first learning unit 153 gives a first reward to the beam control policy according to the change in communication quality indicated by the wireless communication quality information before and after the beam direction is changed. The first learning unit 153 outputs the provided first reward to the first cumulative reward storage unit 1532. The first cumulative reward storage unit 1532 updates the stored value of the first cumulative reward to a value obtained by adding the input first reward. The first learning unit 153 changes the beam control policy with a low first reward. Thereby, the first learning unit 153 changes the beam control policy so that the first cumulative reward over a certain period of time is maximized.

In the second learning mode, the first learning section 153 inputs pseudo environmental state information from the second learning section 154 instead of inputting the environmental state information from the environmental state information acquisition section 152. The first learning section 153 uses this pseudo environmental state information in place of the environmental state information input from the environmental state information acquisition section 152, and performs the same operation as in the first learning mode described above.

The second learning section 154 does not operate in the first learning mode, but operates in the second learning mode. The second learning unit 154 determines whether the beam direction is before or after being controlled based on the pseudo environment state information generated according to the information generation policy and the beam control policy determined by the first learning unit 153 according to the pseudo environment state information. The wireless communication quality information is used to learn an information generation policy for generating pseudo environmental state information that degrades wireless communication quality. The second learning unit 154 outputs pseudo environmental state information generated based on the information generation strategy of the learning result to the first learning unit 153.

The second learning unit 154 includes an information generation policy storage unit 1541 and a second cumulative reward storage unit 1542. The information generation policy storage unit 1541 stores an information generation policy table. The information generation policy table is a diagram showing the correspondence between environmental state information, pseudo environmental state information generated based on the environmental state information, and information generation policy of the pseudo environmental state information. The information generation policy is expressed, for example, by a calculation performed on the environmental state information. In this embodiment, it is assumed that an information generation policy is set for each combination of environmental state information and beam direction information. The second cumulative reward storage unit 1542 stores the second cumulative reward. The second cumulative reward is the sum of the second rewards. The second reward is a value given depending on how much the wireless communication quality has deteriorated due to the control instruction based on the beam control policy determined by the first learning unit 153 using the pseudo environment state information. In this embodiment, the larger the degree of decline, the larger the second reward is given. The second reward may be a tiered value.

In the second learning mode, the second learning unit 154 inputs the environmental state information output by the environmental sensor 13 from the environmental state information acquisition unit 152, and receives beam direction information and wireless communication quality information from the wireless communication quality monitoring unit 12. Enter. The second learning unit 154 generates an information generation policy according to the combination of the information regarding the environmental information indicated by the environmental state information and the beam direction indicated by the beam direction information in an information generation policy table stored in the information generation policy storage unit 1541. Read from. The second learning unit 154 generates pseudo environmental state information by performing the calculation indicated by the read information generation policy on the information regarding the environmental state indicated by the environmental state information. The second learning unit 154 associates the environmental state information input from the environmental state information acquisition unit 152 with the pseudo environmental state information generated based on the environmental state information and writes them into the information generation policy table.

The second learning unit 154 outputs the generated pseudo environment state information to the first learning unit 153. The first learning unit 153 outputs to the wireless communication unit 11 a beam direction control instruction based on the beam control policy determined based on the pseudo environment information input from the second learning unit 154. The second learning section 154 receives from the radio communication quality monitoring section 12 radio communication quality information indicating information regarding the radio communication quality during the communication period during which radio communication is performed using the beam direction changed according to this control instruction. The second learning unit 154 gives a second reward to the information generation policy according to the change in communication quality in the communication period before and after the control instruction. The second learning unit 154 updates the second cumulative reward stored in the second cumulative reward storage unit 1542 to a value obtained by adding the given second reward. The second learning unit 154 changes the information generation policy with a low second reward. Thereby, the second learning unit 154 learns the information generation policy so that the second cumulative reward in a certain period of time is maximized.

FIG. 3 is a diagram showing an example of a beam control policy table stored in the beam control policy storage unit 1531. The beam control policy table shown in FIG. 3 shows the beam control policy and previously acquired reward for each combination of beam direction and environmental state information. In FIG. 3, an example is shown in which the environmental state information is wind speed and the beam control policy is an angle correction amount for the current beam direction. The previously obtained reward indicates the first reward obtained when the beam direction was previously controlled by the corresponding beam control policy.

FIG. 4 is a diagram showing an example of an information generation policy table stored in the information generation policy storage unit 1541. The information generation policy table shown in FIG. 4 includes the beam direction, environmental state information, pseudo environmental state information generated based on the environmental state information, previously obtained reward, and information used to generate the pseudo environmental state information. This information is associated with a generation policy. In FIG. 4, a case where the environmental state information is wind speed is shown as an example. The previous reward indicates the second reward obtained when beam control was performed last time using the beam control policy determined by the first learning unit 153 according to the corresponding pseudo environment state information and beam direction.

Next, the operation of the beam direction control section 15 in each learning mode will be explained.

<First learning mode>
The environmental state information acquisition unit 152 acquires environmental state information from the environmental sensor 13. In the first learning mode, the environmental state information acquisition section 152 outputs the acquired environmental state information to the first learning section 153.

The first learning unit 153 receives environmental state information from the environmental state information acquisition unit 152 and further receives beam direction information and wireless communication quality information from the wireless communication quality monitoring unit 12. The first learning unit 153 determines the beam direction to be used in the next communication period as follows, according to beam direction information indicating the beam direction used in a certain communication period and environmental state information within the communication period. control.

The first learning unit 153 learns the beam direction to be used in the next communication period based on the combination of the input beam direction condition and the input environmental state information condition. The environmental state information input to the first learning unit 153 includes, for example, instantaneous wind speeds [10 _m /s, 8 m/s, 12 m _/ s, ...] is the format of time series data. Alternatively, the environmental state information may be a tuple composed of a plurality of elements, such as instantaneous wind speed and wind direction from time t ₀ to t _N , and the installation height of the wireless communication device 1.

As an example of controlling the beam direction, for example, the first learning unit 153 determines that the instantaneous wind speed from time t ₀ to t _N indicated by the environmental state information is [10 m/s, 8 m/s, 12 m/s, ...]. Based on the combination of this condition and the beam direction condition indicated by the input beam direction information, the angle correction amount corresponding to the environmental state information at each time is [5 degrees, 2 degrees, 11 degrees,...]. A control instruction such as "Yes" is output to the wireless communication section 11.

As described above, the first learning unit 153 outputs an instruction to control the angle correction amount in accordance with the input environmental state information and beam direction information. Therefore, the first learning unit 153 refers to the beam control policy table stored in the beam control policy storage unit 1531 and calculates the angle correction amount corresponding to the current beam direction and the environmental state information experienced in the past. Among them, the angle correction amount corresponding to the same environmental state information as the current one is acquired. The first learning unit 153 outputs a control instruction in which the obtained angle correction amount is set to the wireless communication unit 11.

Note that if there is no angle correction amount corresponding to the current beam direction and the environmental state information experienced in the past in the beam control policy table, the first learning unit 153 determines the angle that can be set in the wireless communication unit 11. A beam control strategy with an arbitrary angle correction amount within the range can be determined. The angle range that can be set depends on the design configuration of the array antenna possessed by the wireless communication device 1, and is determined in advance. The first learning unit 153 writes the beam direction and environmental state information and the beam control policy in which the instructed angle correction amount is set in a beam control policy table stored in the beam control policy storage unit 1531 in association with each other. The first learning unit 153 outputs a control instruction based on the determined beam control policy to the wireless communication unit 11.

Note that the environmental state information indicating wind speed as described above is only an example, and the first learning unit 153 also acquires environmental state information other than wind speed, configures environmental state information consisting of a plurality of elements, and then It is also possible to generate beam direction control instructions. Furthermore, information for changing the beam direction of the wireless communication unit 11 can be expressed in any format. For example, instead of the angle correction amount as described above, necessary weights may be output to each element of the array antenna so that the wireless communication unit 11 can form the directivity of radio waves in the required direction. In this case, the first learning unit 153 may acquire the weight corresponding to the environmental state information as the beam control policy without using the beam direction.

<An example of how to give the first reward>
The result of the wireless communication unit 11 changing the beam direction according to the control instruction output by the first learning unit 153 is reflected in the wireless communication quality such as received power acquired from the wireless communication quality monitoring unit 12 in the next communication period. Ru. The first learning unit 153 determines a first reward according to the change in wireless communication quality information accompanying the beam direction control described above, and records it in the first cumulative reward storage unit 1532. The reward is a numerical value that is given by any method depending on the degree of achievement of a predetermined control objective. The purpose of control of the first learning unit 153 is to "improve received power by controlling beam direction." An example of a method of providing the first reward according to this control purpose will be explained.

For example, in accordance with the control purpose of the first learning unit 153, the received power before beam direction control is compared with the received power after beam direction control, and if the increase is 3 dB or more, the reward is set to 100, and the reward is set to 100, If the increase is less than 3 dB, the reward is 1, and if the increase is less than 0 dB, the reward is 0. However, this first reward giving method is only an example, and other reward giving methods, such as a reward function generated by machine learning, may be used as long as it matches the control purpose.

The first learning unit 153 determines a first reward according to a predetermined reward giving method and a result of beam direction control based on a beam control policy. The first learning unit 153 outputs the first reward to the first cumulative reward storage unit 1532. The first cumulative reward storage unit 1532 updates the current value of the first cumulative reward information with the value obtained by adding the input first reward. In this way, the first learning unit 153 determines the first reward each time it obtains a control result, and the first cumulative reward storage unit 1532 calculates and stores the cumulative sum.

<Update of beam control policy table>
As described above, the first learning unit 153 can output an arbitrary angle correction amount when new environmental state information is input. However, since the received power cannot necessarily be maximized with the arbitrary angle correction amount, it is necessary to learn the angle correction amount that can maximize the received power through trial and error multiple times. Therefore, the first learning unit 153 writes the first reward given after the previous beam direction control was performed using the beam control policy corresponding to the environmental state information into the previously acquired reward corresponding to the environmental state information. Then, the beam control policy table stored in the beam control policy storage unit 1531 is updated. When the first reward set as the previously acquired reward is not the maximum value, the first learning unit 153 changes the beam control policy corresponding to the previously acquired reward. For example, the first learning unit 153 changes the angle correction amount corresponding to the previously acquired reward that is not the maximum value in the beam control policy table shown in FIG. Then, the first learning unit 153 writes the first reward given after beam direction control using the changed angle correction amount into the beam control strategy table. In this way, the first learning unit 153 updates the contents of the beam control policy table and stores it in the beam control policy storage unit 1531.

<Second learning mode>
In the second learning mode, the first learning section 153 and the second learning section 154 operate. In the first learning mode, the first learning section 153 can learn a beam control policy based on the environmental state information input from the environmental state information acquisition section 152. This environmental state information indicates information regarding the environmental state in which the wireless communication device 1 is placed, which is actually observed by the environmental sensor 13. However, beam control strategies cannot be learned for environmental conditions that are not actually observed and that the wireless communication device 1 has not experienced. In particular, in environments with complex external causes, there is a high possibility that all environmental conditions cannot be experienced through actual observation within a finite amount of time. It is assumed that many of the beam control methods determined by the first learning unit 153 for unexperienced environmental conditions are not optimized, and communication quality may deteriorate due to beam control failure.

Therefore, in the second learning mode, the environmental state information acquisition unit 152 generates pseudo environmental state information that is information about a pseudo environmental state that has not been experienced in actual observation, and uses the generated pseudo environmental state information. Input to the first learning section 153. The first learning unit 153 recognizes the pseudo environmental state information as environmental state information that has not been experienced in the past, and starts learning a new beam control strategy to cope with such environmental state information. That is, the environmental state information acquisition unit 152 has a function of promoting learning by the first learning unit 153.

There are many environmental conditions in the wireless communication device 1. In the first learning mode, learning can only be performed using previously experienced environmental states among existing environmental states. In the second learning mode, the first learning unit 153 is caused to experience in a simulated manner an unexperienced environmental state that is different from the previously experienced environmental states. Therefore, the first learning unit 153 can learn even in an inexperienced environmental state.

In order to realize the function of the second learning unit 154, in addition to promoting the learning of the first learning unit 153, an information generation policy for pseudo environment state information is required so that the self-function unit can also simulate a complex external environment. need to learn about. Usually, a complex external environment degrades the effect of beam direction control by the first learning section 153 (for example, maximizing received power), so the learning objective opposite to that of the first learning section 153 is set. It is thought that they have. Therefore, in the second learning mode, the second learning section 154 that simulates the influence of a complex external environment has a learning purpose of reducing received power, which is opposite to the learning purpose of the first learning section 153.

In the second learning mode, the first learning section 153 uses the pseudo environmental state information generated by the second learning section 154 instead of the environmental state information input from the environmental state information acquisition section 152. Similarly to the learning mode, the learning mode learns to increase the received power in the next communication period, and outputs a beam direction control instruction to the wireless communication unit 11. For the sake of brevity, unless otherwise specified, the operation of the first learning section 153 is the same as in the first learning mode described above, and the details thereof will be omitted below.

The second learning section 154 has the opposite purpose to the first learning section 153 in order to simulate an environmental state with a complex external cause as described above, and reduces the received power in the next communication period. learn strategies for generating pseudo-environmental state information.

The environmental state information acquisition unit 152 acquires environmental state information from the environmental sensor 13. In the second learning mode, the environmental state information acquisition unit 152 inputs the acquired environmental state information to the second learning unit 154. Furthermore, the beam direction information acquired by the wireless communication quality monitoring section 12 is also input to the second learning section 154.

The second learning section 154 performs calculations on the contents of the environmental state information input from the environmental state information acquisition section 152 to generate pseudo environmental state information. For example, the environmental state information input to the second learning unit 154 is the instantaneous wind speed [ _{10 m} /s, 8 m/s, 12 m/s ,...] is the format of time series data. Alternatively, the environmental state information may be a tuple composed of a plurality of elements, such as instantaneous wind speed and wind direction from time t ₀ to t _N , and the installation height of the wireless communication device.

The second learning unit 154 learns how to generate pseudo environmental state information indicating information regarding a new environmental state for the combination of the input beam direction condition and the input environmental state information condition. Learn information generation strategies. As an example of an information generation policy for generating pseudo environmental state information, the second learning unit 154 calculates, for example, that the instantaneous wind speed from time t ₀ to t _N acquired from the environmental state information acquisition unit 152 is [10 m/s, 8 m/s]. /s, 12m/s, . . .] and the beam direction indicated by the input beam direction information, an operation to double the wind speed is obtained. The second learning unit 154 uses the acquired information generation policy to generate pseudo environmental state information [20 m/s, 16 m/s, 24 ms, ...] at each time, and outputs it to the first learning unit 153. .

The second learning unit 154 associates the beam direction conditions and environmental state information conditions, the generated pseudo environmental state information, and the information generation strategy used to generate the pseudo environmental state information, as shown in FIG. Write to the information generation policy table. Note that if the information generation policy table stored in the information generation policy storage unit 1541 does not include pseudo environmental state information corresponding to the current beam direction and the environmental state information experienced in the past, the second learning unit 154 may perform an arbitrary It is possible to generate pseudo environmental state information using the information generation policy (for example, multiplying the value indicated by the environmental state information by a random positive number) and output it to the first learning unit 153.

Note that the above-mentioned environmental state information and the strategy for generating pseudo environmental state information by the second learning unit 154 are only examples, and the environmental state may be changed by acquiring other environmental state information or by using any other information generation strategy. Pseudo environmental state information may be generated from the information.

The second learning unit 154 outputs the generated pseudo environment state information to the first learning unit 153. The first learning section 153 uses the pseudo environmental state information input from the second learning section 154 instead of the environmental state information input from the environmental state information acquisition section 152 in the second learning mode. The same operation as in the first learning mode is performed.

Note that, when the combination of the beam direction and the pseudo environment state information input from the second learning unit 154 is not experienced, the first learning unit 153 determines the angle that can be set in the wireless communication unit 11 as described above. Determine a beam control strategy for an arbitrary angle correction amount within the range. The first learning unit 153 writes the beam direction and pseudo environment state information and the determined beam control policy in association with each other in the beam control policy table. In the subsequent first learning mode, when the first learning unit 153 inputs the same environmental state information as the pseudo environmental state information input in the past and the same beam direction as when the pseudo environmental state information was input, , the corresponding beam control strategies can be read out from the beam control strategy table and the beam direction can be controlled immediately. In other words, the first learning unit 153 can use the beam control policy as an initial value to learn a beam control policy for an unexperienced environmental state in advance, and when that environmental state occurs in the future, it can use the learned beam control policy as an initial value. Using beam control strategies, it is possible to respond to changing environments, and failures in beam direction control due to unlearning can be avoided.

Further, if the combination of the current beam direction and the pseudo environment state information input from the second learning unit 154 is already set in the beam control policy table, the first learning unit 153 selects the first learning mode. Similarly, a beam direction control instruction is output to the wireless communication unit 11 based on the corresponding beam control policy. However, in the second learning mode, the first learning unit 153 does not give the first reward to the beam control strategy and does not change the beam control strategy. This prevents correctly learned beam control strategies from being changed. Alternatively, the first learning unit 153 may notify the second learning unit 154 of information on the combination of the beam direction and environmental state information for which the beam control policy has been learned. When the second learning unit 154 determines that the combination of the current beam direction and the generated pseudo environment state information has been learned, the second learning unit 154 changes the information generation policy and generates different pseudo environment state information.

<Example of second reward granting method>
The second learning unit 154 can observe the result of beam direction control performed using the generated pseudo environment state information using the wireless communication quality information acquired from the wireless communication quality monitoring unit 12 in the next communication period. The second learning unit 154 determines a second reward according to changes in wireless communication quality information before and after performing beam direction control based on the pseudo environment state information generated as described above, and calculates a second cumulative reward. It is recorded in the remuneration storage unit 1542. The second reward given to the information generation policy after the generation of the pseudo environmental state information needs to be set for the opposite purpose to the first learning part. In other words, the purpose of control of the first learning section 153 is to improve the received power, while the purpose of the second learning section 154 is to decrease the received power. The second reward given by the second learning unit 154 needs to be determined in accordance with this purpose. Therefore, for example, if the received power after beam direction control based on the pseudo environment state information generated by the second learning section 154 is reduced by 3 dB or more compared to the received power before beam direction control, the second learning section 154 The reward is 100, and if the decrease is 0 dB or more and less than 3 dB, the second reward is 1, and if there is no decrease, the second reward is 0. Note that the above-mentioned method of awarding the second reward is only an example, and the reward may be determined using other methods, such as a reward function generated by machine learning, for example.

The second learning unit 154 outputs the second reward determined according to the beam direction control result based on the pseudo environment state information to the second cumulative reward storage unit 1542. The second cumulative reward storage unit 1542 adds the input second reward to the current value of the second cumulative reward information, and updates the second cumulative reward information with the added value. In this way, each time the second learning unit 154 generates pseudo environment state information, the second learning unit 154 obtains the control result of beam direction control based on the pseudo environment state information, determines the second reward, and calculates the second cumulative The reward storage unit 1542 calculates and stores the cumulative sum.

<Updating the storage contents of the information generation policy storage unit 1541>
As described above, when new environmental state information is input, the second learning unit 154 can output pseudo environmental state information generated using an arbitrary information generation strategy. However, the pseudo environmental state information generated by the arbitrary information generation strategy is not necessarily optimal for achieving the purpose. Therefore, it is necessary to learn information generation strategies to achieve the goal through multiple trials and errors. Therefore, the second learning unit 154 stores the second reward received after the previous beam direction control performed based on the generated pseudo environment state information in the information generation policy storage unit 1541 shown in FIG. Write to the information generation policy table. When the second reward received after the previous control is not the maximum value for certain pseudo environment state information, the second learning unit 154 determines an information generation policy or control range corresponding to the pseudo environment state information. change. The second learning unit 154 uses the changed information control policy or the second reward given after the beam direction control is based on the pseudo environment state information generated using the control range to update the information generation policy storage unit 1541. Update the stored information generation policy table.

An example in which the environmental state information is instantaneous wind speed as shown in FIG. 4 will be described. For example, in the first state information generation policy for pseudo environment state information, the second learning unit 154 multiplies the instantaneous wind speed at a certain time by 2 and outputs the result as pseudo environment state information. Then, when the second learning unit 154 is unable to obtain the maximum reward based on the generated pseudo environment state information, when generating the pseudo environment state information for the second time, the instantaneous wind speed is multiplied by 4. A possible method is to increase the value to be multiplied according to the number of repetitions. However, the present invention is not limited to the state information generation policy described above, and any calculation method or algorithm may be used to achieve the purpose of the second learning unit 154.

<Processing flow>
FIG. 5 is a flow diagram showing an example of the operation of the beam direction control section 15 in the first learning mode. The mode setting unit 151 starts the first learning mode (step S105). The first learning unit 153 inputs beam direction information and wireless communication quality information from the wireless communication quality monitoring unit 12, and inputs environmental state information from the environmental sensor 13 (step S110). The first learning unit 153 reads out the beam direction control policy corresponding to the input environmental state information and beam direction information from the beam control policy table (step S115). The first learning unit 153 determines an arbitrary beam direction control policy when there is no beam direction control policy corresponding to the input environmental state information and beam direction information. The first learning unit 153 writes the environmental state information and beam direction information and the determined beam direction control policy in a beam control policy table in association with each other. The first learning unit 153 outputs a beam direction control instruction based on the beam direction control policy to the wireless communication unit 11 (step S120). The wireless communication unit 11 changes the beam direction according to the beam direction control policy set in the control instruction, and performs wireless communication in the next communication period using the changed beam direction.

The first learning unit 153 inputs the beam direction information and wireless communication quality information after the control instruction from the wireless communication quality monitoring unit 12, and inputs the environmental state information after the control instruction from the environmental state information acquisition unit 152 (step S125). The first learning unit 153 determines the wireless communication quality in the communication period using the beam direction changed according to the control instruction output in the immediately preceding step S120, and the wireless communication quality in the communication period before the beam direction change immediately before that communication period. Compare with quality. The first learning unit 153 determines the first reward according to the comparison result (step S130). The first learning unit 153 writes the determined first reward into the beam control strategy table in association with the beam control strategy used when outputting the control instruction in the immediately previous step S120 (step S135). Furthermore, the acquired reward of the first learning unit 153 outputs the determined first reward to the first cumulative reward storage unit 1532. The first cumulative reward storage unit 1532 updates the stored value of the first cumulative reward to a value obtained by adding the input first reward (step S140).

The first learning unit 153 determines whether the determined first reward is the maximum value (step S145). When the first learning unit 153 determines that the determined first reward is the maximum value (step S145: YES), the process proceeds to step S155. If the first learning unit 153 determines that the determined first reward is not the maximum value (step S145: NO), it changes the previous beam control policy set in the beam control policy table (step S150). ).

If the first learning unit 153 determines that the first learning mode is not finished (step S155: NO), it repeats the process from step S115. Then, when the first learning section 153 determines that the first learning mode has ended (step S155: YES), the first learning section 153 ends the process of FIG.

Note that the processing from step S145 to step S150 may be performed at a timing independent of the processing from step S105 to step S140 and step S155. In this case, the first learning unit 153 detects a beam control policy whose previous reward is not the maximum value from the beam control policy table, and performs the process of step S150 on the detected beam control policy.

FIG. 6 is a flow diagram showing an example of the operation of the beam direction control section 15 in the second learning mode. The mode setting unit 151 starts the second learning mode (step S205). The first learning unit 153 and the second learning unit 154 receive beam direction information and wireless communication quality information from the wireless communication quality monitoring unit 12, and the second learning unit 154 receives the environmental state output from the environmental sensor 13. Information is input from the environmental state information acquisition unit 152 (step S210).

The second learning unit 154 reads out the information generation policy corresponding to the environmental state information and the beam direction information from the information generation policy table stored in the information generation policy storage unit 1541 (step S215). The second learning unit 154 generates pseudo environmental state information from the environmental state information according to the read information generation policy (step S220). Note that if the information generation policy corresponding to the environmental state information and beam direction information is not set in the information generation policy table, the second learning unit 154 determines an arbitrary information generation policy. The second learning unit 154 associates the environmental state information and beam direction information, the generated pseudo environmental state information, and the determined information generation policy and writes them into the information generation policy table. The second learning unit 154 outputs the generated pseudo environment state information to the first learning unit 153 (step S225). The first learning unit 153 uses the pseudo environmental state information instead of the environmental state information to perform the processes of steps S115 to S120 shown in FIG. 5 (step S230). The wireless communication unit 11 changes the beam direction according to the beam direction control policy set in the control instruction output by the first learning unit 153 in step S230, and performs wireless communication in the next communication period using the changed beam direction. conduct.

The first learning unit 153 and the second learning unit 154 input beam direction information and wireless communication quality information after a control instruction from the wireless communication quality monitoring unit 12, and the second learning unit 154 acquires environmental state information. Environmental state information after the control instruction is input from the unit 152 (step S235). The first learning unit 153 performs the processes from step S125 to step S155 shown in FIG. However, in step S125, no environmental state information is input to the first learning unit 153. Further, if the beam control policy corresponding to the beam direction and the pseudo environment state information has been learned, the first learning unit 153 does not perform the processing of steps S130 to S155.

The second learning unit 154 calculates the wireless communication quality in the communication period using the beam direction changed according to the control instruction output by the first learning unit 153 in step S230 immediately before, and the beam direction change immediately before the communication period. Compare the wireless communication quality with the previous communication period. The second learning unit 154 determines the second reward according to the comparison result (step S240). The second learning unit 154 writes the determined second reward into the information generation strategy table in association with the information generation strategy used when generating the pseudo environmental state information in the immediately preceding step S220 (step S245). Furthermore, the second learning unit 154 outputs the determined second reward to the second cumulative reward storage unit 1542. The second cumulative reward storage unit 1542 updates the stored second cumulative reward to a value obtained by adding the input second reward (step S250).

The second learning unit 154 determines whether the determined second reward is the maximum value (step S255). When the second learning unit 154 determines that the determined second reward is the maximum value (step S255: YES), the process proceeds to step S265. When the second learning unit 154 determines that the determined second reward is not the maximum value (step S255: NO), the second learning unit 154 changes the previous state information generation policy set in the information generation policy table (step S260).

If the second learning section 154 determines that the second learning mode has not ended (step S265: NO), it repeats the processing from step S215. Then, when the second learning section 154 determines that the second learning mode has ended (step S265: YES), the second learning section 154 ends the process of FIG. 6 .

Note that the processing from step S255 to step S260 may be performed at a timing independent of the processing from step S205 to step S250 and step S265. In this case, the second learning unit 154 detects an information generation strategy whose previous reward is not the maximum value from the information generation strategy table, and changes the detected information generation strategy.

According to the present embodiment, under conditions where external factors that affect wireless communication quality become more complex, an appropriate learning device is used to respond to changes in complex external factors (environmental conditions around the wireless communication device). Thus, a method of beam direction control can be realized. Therefore, even when a wireless communication device is placed in an environment where complex fluctuations due to external factors occur, the number of failures in beam direction control based on learning can be reduced.

Note that the wireless communication device may include a beam direction control device having the beam direction control section 15 inside or outside.

The functions of the beam direction control unit 15 of the wireless communication device 1 in the embodiment described above may be realized by a computer. In this case, the beam direction control unit 15 can realize this function by recording a program on a computer-readable recording medium, and having the computer system read and execute the program recorded on the recording medium. Good too. Note that the "computer system" herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, a "computer-readable recording medium" refers to a storage medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include a device that retains a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Further, the above-mentioned program may be one for realizing a part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

An example of the hardware configuration of the wireless communication device 1 will be described. FIG. 7 is a device configuration diagram showing an example of the hardware configuration of the wireless communication device 1. As shown in FIG. The wireless communication device 1 includes a processor 71, a storage section 72, a communication interface 73, a user interface 74, and a sensor 75.

The processor 71 is a central processing unit that performs calculations and control. Processor 71 is, for example, a CPU. The processor 71 reads a program from the storage unit 72 and executes it. The storage unit 72 further includes a work area when the processor 71 executes various programs. The communication interface 73 is communicably connected to other devices. The user interface 74 is an input device such as a dip switch or a button, or a display device such as a lamp or a display. A human operation is input through the user interface 74 . The sensor 75 detects or obtains environmental state information.

The functions of the wireless communication quality monitoring section 12 and the beam direction control section 15 are realized by the processor 71 reading a program from the storage section 72 and executing it. Note that all or part of these functions may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The wireless communication unit 11 is realized by the communication interface 73. Further, the communication interface 73 may realize communication with a PC or the like via a network. The environmental sensor 13 is realized by one or more sensors 75. Note that some functions of the wireless communication unit 11 and the environmental sensor 13 may be realized by the processor 71 reading out and executing a program from the storage unit 72.

According to the embodiment described above, the beam direction controllable wireless communication device includes a wireless communication section, a sensor, a wireless communication quality monitoring section, and a beam direction control section. The wireless communication unit performs wireless communication by forming a beam. The sensor acquires environmental status information that is information regarding the installation environment of its own device. For example, the sensor is an environmental sensor in an embodiment. The wireless communication quality monitoring unit acquires wireless communication quality information indicating the quality of wireless communication by the wireless communication unit. The beam direction control unit outputs a beam direction control instruction to the wireless communication unit. The beam direction control section includes a first learning section, a second learning section, and a switching section. The first learning section uses environmental state information and wireless communication quality information before and after the beam direction is controlled, and describes a beam direction control method that improves the quality of wireless communication according to environmental state information. Learn control strategies. The first learning unit determines a beam control strategy according to the environmental state information based on the learning result, and outputs a beam direction control instruction according to the determined beam control strategy to the wireless communication unit. The second learning unit uses the environmental status information generated according to the information generation policy indicating the calculation for generating the environmental status information and the control instruction outputted by the first learning unit according to the generated environmental status information. Based on this information, information generation strategies for generating environmental state information that degrades the quality of wireless communication are learned using wireless communication quality information before and after the beam direction was controlled. The second learning unit generates environmental state information based on the learned information generation policy. The switching unit switches which of the environmental state information acquired by the sensor and the environmental state information generated by the second learning unit is input to the first learning unit.

Note that the first learning unit compares the input environmental state information, the beam direction formed by the wireless communication unit, and the wireless communication quality information before and after the beam direction is changed. Changes in communication quality may be used to learn beam control strategies that improve the quality of wireless communication according to environmental state information and beam direction. The first learning unit determines a beam control policy according to the environmental state information and the beam direction formed by the wireless communication unit based on the learning result, and instructs to control the beam direction according to the determined beam control policy. is output to the wireless communication section.

Further, the first learning unit provides a first reward to the beam control policy according to a change in the quality of wireless communication before and after the beam direction is changed according to the control instruction output according to the beam control policy; The selected beam control strategy may be changed based on the reward.

Further, the second learning unit is configured to perform calculations on the environmental status information acquired by the sensor according to an information generation policy corresponding to the environmental status information and the beam direction formed by the wireless communication unit to generate an environmental state. Changes in the quality of wireless communication obtained by comparing the information and wireless communication quality information before and after the beam direction is controlled based on the control instruction output by the first learning unit according to the generated environmental state information may be used to learn an information generation policy for reducing the quality of wireless communication depending on the environmental state information and the beam direction. The second learning unit determines an information generation policy according to the environmental state information and the beam direction formed by the wireless communication unit based on the learning result, and applies the calculation indicated by the determined information generation policy to the environmental state information. and generates environmental state information to be input to the first learning section.

Note that the second learning unit determines the quality of wireless communication before and after the beam direction is changed based on the control instruction output by the first learning unit in accordance with the environmental state information generated by performing calculations indicated by the information generation policy. A second reward may be given to the information generation policy according to the change, and the selected information generation policy may be changed based on the second reward.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

DESCRIPTION OF SYMBOLS 1... Wireless communication device, 11... Wireless communication unit, 12... Wireless communication quality monitoring unit, 13... Environmental sensor, 15... Beam direction control unit, 71... Processor, 72... Storage unit, 73... Communication interface, 74... User interface , 75... sensor, 91... wireless communication device, 92... utility pole, 93... utility pole, 94... overhead wire, 95... wireless communication device, 96... building, 97... pedestrian, 98-1, 98-2... access point, 99 ...Terminal station, ...Route, 151...Mode setting section, 152...Environmental state information acquisition section, 153...First learning section, 154...Second learning section, 1531...Beam control policy storage section, 1532...First Cumulative reward storage unit, 1541... Information generation policy storage unit, 1542... Second cumulative reward storage unit

Claims (8)

A wireless communication device capable of controlling beam direction,
a wireless communication unit that performs wireless communication by forming a beam;
a sensor that acquires environmental status information that is information about the installation environment of the own device;
a wireless communication quality monitoring unit that acquires wireless communication quality information indicating the quality of wireless communication by the wireless communication unit;
a beam direction control unit that outputs a beam direction control instruction to the wireless communication unit;
Equipped with
The beam direction control section includes:
Using the environmental state information and the wireless communication quality information before and after the beam direction is controlled, a beam control strategy is learned that indicates a beam direction control method that improves the quality of wireless communication according to the environmental state information. , a first learning unit that determines the beam control strategy according to the environmental state information based on the learning result and outputs a beam direction control instruction according to the determined beam control strategy to the wireless communication unit;
The beam direction is determined based on the environmental state information generated according to the information generation policy indicating the calculation for generating the environmental state information and the control instruction outputted by the first learning unit in accordance with the generated environmental state information. The wireless communication quality information before and after the control is used to learn an information generation policy for generating environmental state information that degrades the quality of wireless communication, and the environmental state information is generated based on the learned information generation policy. a second learning section that generates;
a switching unit that switches which of the environmental state information acquired by the sensor and the environmental state information generated by the second learning unit is input to the first learning unit;
Equipped with
Wireless communication device. The first learning unit compares the input environmental state information, a beam direction formed by the wireless communication unit, and the wireless communication quality information before and after the beam direction is controlled . learning a beam control policy for improving the quality of wireless communication according to the input environmental state information and the beam direction formed by the wireless communication unit , and learning results. Based on the input environmental state information and the beam direction formed by the wireless communication unit, the beam control policy is determined based on the input environmental state information and the beam direction formed by the wireless communication unit, and a beam direction control instruction is issued in accordance with the determined beam control policy. output to the wireless communication unit;
The wireless communication device according to claim 1. The first learning unit provides a first reward to the beam control policy according to a change in the quality of wireless communication before and after the beam direction is changed according to the control instruction output according to the beam control policy, and changing the selected beam control strategy based on a first reward;
The wireless communication device according to claim 2. The second learning unit performs a calculation on the environmental status information acquired by the sensor and the information generation policy corresponding to the environmental status information and the beam direction formed by the wireless communication unit to generate the information. and the wireless communication quality information before and after the beam direction is controlled based on the control instruction outputted by the first learning unit according to the generated environmental status information. learning an information generation policy for reducing the quality of wireless communication according to the environmental state information acquired by the sensor and the beam direction formed by the wireless communication unit, using the change in the quality of wireless communication obtained by Based on the learning result , determine the information generation policy according to the environmental state information acquired by the sensor and the beam direction formed by the wireless communication unit, and decide on the environmental state information acquired by the sensor. generating environmental state information to be input to the first learning unit by performing calculations indicated by the information generation policy;
The wireless communication device according to claim 2 or claim 3. The second learning unit is configured to perform wireless communication before and after the beam direction is changed according to the control instruction outputted by the first learning unit in accordance with the environmental state information generated by performing calculations indicated by the information generation policy. assigning a second reward to the information generation policy according to a change in the quality of the information generation policy, and changing the selected information generation policy based on the second reward;
The wireless communication device according to claim 4. Using environmental state information, which is information about the installation environment of a wireless communication device whose beam direction can be controlled, and wireless communication quality information, which indicates the quality of wireless communication before and after the beam direction of the wireless communication device is controlled, Learning a beam control policy indicating a method of controlling a beam direction to improve the quality of wireless communication according to the environmental state information, and determining the beam control policy according to the environmental state information based on the learning result. a first learning unit that outputs a beam direction control instruction according to the beam control policy to the wireless communication device;
The beam direction is determined based on the environmental state information generated according to the information generation policy indicating the calculation for generating the environmental state information and the control instruction outputted by the first learning unit in accordance with the generated environmental state information. The wireless communication quality information before and after the control is used to learn an information generation policy for generating environmental state information that degrades the quality of wireless communication, and based on the learned information generation policy, environmental state information is generated. a second learning section that generates
a switching unit that switches which of the environmental state information acquired by a sensor of the wireless communication device and the environmental state information generated by the second learning unit is input to the first learning unit;
A beam direction control device comprising: A beam direction control method executed by a wireless communication device capable of controlling a beam direction, the method comprising:
a communication step in which the wireless communication unit performs wireless communication by forming a beam;
an environmental status information acquisition step in which the sensor acquires environmental status information that is information regarding the installation environment of the wireless communication device;
a wireless communication quality information acquisition step in which the wireless communication quality monitoring unit acquires wireless communication quality information indicating the quality of wireless communication by the wireless communication unit;
a beam direction control step in which the beam direction control section outputs a beam direction control instruction to the wireless communication section;
The beam direction control step includes:
Using the environmental state information and the wireless communication quality information before and after the beam direction is controlled, learn a beam control strategy that indicates a beam direction control method that improves the quality of wireless communication according to the environmental state information. a first learning step of determining the beam control strategy according to the environmental state information based on the learning result, and outputting a beam direction control instruction according to the determined beam control strategy to the wireless communication unit; ,
A beam is generated based on the environmental state information generated according to the information generation policy indicating the calculation for generating the environmental state information and the control instruction outputted in the first learning step according to the generated environmental state information. The wireless communication quality information before and after the direction is controlled is used to learn an information generation policy for generating environmental state information that degrades wireless communication quality, and the environmental state information is generated based on the learned information generation policy. a second learning step to generate;
a switching step of switching which of the environmental state information acquired in the environmental state information acquisition step and the environmental state information generated in the second learning step is used in the first learning step;
A beam direction control method comprising: computer,
A program for functioning as the beam direction control device according to claim 6.

Images