JP7720587B2 - Communication device, communication system, and communication method - Google Patents

Description

The present invention relates to packet scheduling in wireless communication systems.

Currently, wireless communication systems have evolved into heterogeneous networks based on multi-band, multi-access systems. In cellular communications, fifth-generation mobile communications (5G) has been put into practical use, utilizing a wide range of frequencies, from sub-1 GHz to millimeter wave bands, and we are entering a world where cells of various sizes, from small cells to macrocells, are provided in an overlapping manner.

Wireless LAN, another typical wireless access system, also uses radio frequencies in the 2.4, 5, and 60 GHz bands, with the use of the 6 GHz band also being considered. Wireless devices such as smartphones generally have interfaces that support both cellular and wireless LAN access, and each interface supports multiple bands. It is becoming common for devices to select a wireless base station to connect to from multiple frequencies and access methods, and it is also common for a single device to use multiple base stations in a manner such as dual connectivity.

In such a heterogeneous environment, controlling and optimizing which base station a terminal selects and which I/F across the entire system is effective in making efficient use of system resources.

In addition, as 5G advances, one of the goals is to realize communication functions for ultra-high reliability and ultra-low latency applications, such as uRLLC (Ultra-Reliable and Low Latency Communications), which have not been widely used in conventional wireless communications.

One conventional technique for achieving high reliability (low packet loss) and low latency is to send the same data redundantly over multiple wireless interfaces and multiple bands, and then combine the data on the receiving side (e.g., Non-Patent Document 1).

Cisco Parallel Redundancy Protocol Over Wireless https://www.cisco.com/c/ja_jp/td/docs/wireless/outdoor_industrial/iw3702/technote/b_prp_dg.html Yue Gao, Kry Yik Chau Lui, Pablo Hernandez-Leal, "Robust Risk-Sensitive Reinforcement Learning Agents for Trading Markets," RL4RealLife Workshop in Int. Conf. on Machine Learning (ICML), 2021.

The technology in Non-Patent Document 1 basically sets fixed redundant wireless I/Fs or bands according to the required QoS level, which can result in more wireless resources being used than necessary, resulting in poor wireless resource utilization efficiency. Furthermore, it is not possible to flexibly reflect the amount of resources required in response to changes in the environment.

The present invention was made in consideration of the above points, and aims to provide technology that can simultaneously achieve the desired communication quality and improve the utilization efficiency of wireless resources while adapting to changes in the environment.

According to the disclosed technology, there is provided a communication device that performs wireless communication using a plurality of wireless interfaces,
a wireless interface for transmitting packets to a device; and a reinforcement learning unit for determining, using risk-averse reinforcement learning, the number of packets to be transmitted to the device via the wireless interface;
a transmitting unit that transmits the number of packets determined by the reinforcement learning unit to the device;
A communication device comprising:
A communication device is provided in which the reinforcement learning unit learns actions relative to states through risk-averse reinforcement learning, with the satisfaction level based on the packet loss rate in each wireless interface being the state, and the combination of wireless interfaces used by each device and the number of packets transmitted in each wireless interface being the actions.

The disclosed technology provides a technique for achieving both desired communication quality and improved utilization efficiency of wireless resources while adapting to environmental changes.

FIG. 1 is a diagram illustrating an example of the configuration of a wireless communication system. FIG. 1 is a diagram illustrating the configuration of a wireless base station (or a wireless terminal). FIG. 1 is a diagram illustrating the configuration of a wireless base station (or a wireless terminal). 10 is a flowchart showing an outline of an operation. FIG. 1 is a diagram for explaining a system model. FIG. 1 is a diagram illustrating reinforcement learning. FIG. 1 illustrates Algorithm 1. FIG. 2 illustrates an example of a hardware configuration of the apparatus.

The following describes an embodiment of the present invention (the present embodiment) with reference to the drawings. The embodiment described below is merely an example, and the embodiments to which the present invention can be applied are not limited to the following embodiment.

(System configuration example)
Fig. 1 shows an example of the configuration of a wireless communication system according to this embodiment. As shown in Fig. 1, this system includes a wireless base station 100 and a plurality of wireless terminals 200. In the example of Fig. 1, the wireless base station 100 is connected to the Internet.

In this embodiment, a wireless base station 100 equipped with multiple wireless interfaces uses a reinforcement learning technique described below to determine the wireless interface through which packets to be transmitted to a device (wireless terminal) will be transmitted, as well as the number of packets to be transmitted over that wireless interface, and then transmits the packets. Note that determining the number of packets may also be referred to as packet scheduling. However, the technique according to this embodiment can also be applied to a wireless terminal 200. The wireless base station and wireless terminal may be collectively referred to as a communication device.

Furthermore, in the specific examples described below, two types of wireless interfaces are described: Sub-6 GHz and mmWave, but the wireless interface is not limited to these. Furthermore, "wireless interface" may also be interpreted as "frequency." In other words, in this embodiment, in a form in which multiple frequencies are aggregated and used, frequency selection and packet count determination can be achieved using the reinforcement learning method described below.

Figure 2 shows an example configuration of the wireless base station 100. The wireless terminal 200 may also have a configuration similar to that shown in Figure 2.

As shown in FIG. 2, the wireless base station 100 has a communication I/F unit 110, a control unit 120, a wireless communication unit 130, and an antenna 101.

The wireless communication unit 130 includes a scheduler unit 140, a receiver 131, a wireless communication signal generator 132, and an RF unit 135. The scheduler unit 140 includes a reinforcement learning unit 150, a communication quality measurement unit 141, a total wireless resource allocation calculator 142, and an individual wireless resource allocation calculator 143. The number of "individual wireless resource allocation calculators 143, receivers 131, wireless communication signal generators 132, RF units 135, and antennas 101" is equal to the number of wireless interfaces. However, any of the "individual wireless resource allocation calculators 143, receivers 131, wireless communication signal generators 132, RF units 135, and antennas 101" may be shared by multiple interfaces. Furthermore, the "individual wireless resource allocation calculators 143, receivers 131, wireless communication signal generators 132, RF units 135, and antennas 101" may be referred to as "wireless interfaces."

The reinforcement learning unit 150 includes a Q table management unit 151, a state calculation unit 152, a reward calculation unit 153, and a risk assessment unit 154. The operation of each unit is as follows:

The communication I/F unit 110 communicates with, for example, the Internet. The control unit 120 includes, for example, a CPU and memory, and controls the entire device. The wireless communication unit 130 performs operations related to wireless communication.

The scheduler unit 140 performs packet scheduling, etc. The receiver unit 131 receives signals from other communication devices (e.g., feedback from a wireless terminal) via an antenna and RF unit. The wireless communication signal generator unit 132 generates signals to be transmitted wirelessly from the data of the packets to be transmitted. The RF unit 135 performs processing such as placing the signals on a carrier wave. The scheduler unit 140 can also be realized by a computer and program, and the program can be recorded on a recording medium or provided via a network.

The communication quality measurement unit 141 measures communication quality (e.g., packet loss rate) based on, for example, the number of transmitted packets and feedback (e.g., ACK/NACK) from the communication partner. Note that in this embodiment, instantaneous CSI feedback (ACK/NACK, etc.) cannot be obtained from each device, but it is assumed that sporadic CSI feedback is available, and communication quality statistics (e.g., average value across all devices) can be obtained from the sporadic CSI feedback.

The total wireless resource allocation calculation unit 142 determines the amount to be allocated to each wireless interface relative to the total number of packets to be transmitted, based on the behavior determined by the reinforcement learning unit 150 for each frame. Furthermore, the individual wireless resource allocation calculation unit 143 determines the amount of wireless resources corresponding to the number of packets to be transmitted on the relevant wireless interface (wireless interface connected to the individual wireless resource allocation calculation unit 143), based on the behavior determined by the reinforcement learning unit 150 for each frame.

The wireless base station 100 (or wireless terminal 200) can also be represented by the configuration shown in Figure 3. As shown in Figure 3, the wireless base station 100 has a reinforcement learning unit 10, a transmitter 20, and a receiver 30. The reinforcement learning unit 10 performs the same processing as the reinforcement learning unit 150. The transmitter 20 performs processing related to transmission (e.g., calculating transmission resource allocation, transmitting packets), and the receiver 30 performs processing related to reception (e.g., receiving feedback, calculating communication quality).

(Regarding the reinforcement learning unit 150)
In this embodiment, the radio base station 100 (or the radio terminal 200) employs a configuration in which a plurality of radio interfaces (or a plurality of frequencies) are aggregated.

By equipping the scheduler unit 140, which allocates wireless resources to packets transmitted from each wireless interface, with a reinforcement learning unit 150, reinforcement learning is applied (1) to autonomously learn and perform optimal connections to achieve the desired communication quality. By using a risk-averse learning method (Non-Patent Document 2) (2) based on parallel updating of multiple Q tables (including single Q tables), it is possible to select actions that prioritize communication reliability.

In applying reinforcement learning (1) above, in this embodiment, the state s(t) is the satisfaction level based on packet loss rate information (detected from ACK feedback) for each wireless interface, and the action a(t) is the combination of wireless interfaces to be used for each device (wireless terminal if the source is a wireless base station) and packet scheduling (number of packets to be transmitted over each wireless interface). In this embodiment, the optimal action a(t) for each device is learned from the state s(t) using risk-averse average Q-learning.

In the case of uRLLC, which is assumed in this embodiment, instantaneous CSI feedback cannot be used to maintain low latency. In this embodiment, the radio interface selection and packet scheduling methods are designed to enable good risk-averse learning even when the instantaneous channel conditions are unknown.

Regarding the risk-averse learning method (2) above, the equations (Equations (11) and (12) described below) that show the concept of the evaluation function that responds to risk (magnitude of variance) in risk-averse learning include a term that quickly responds to the variance (risk) of past reward r, thereby reflecting the reduction in reward for high-risk behavior. The term that reacts to the variance of past reward r and reflects it in the evaluation is the second term (the term with Var) in Equation (12) described below (the equation obtained by Taylor expansion of Equation (11)).

As will be explained in more detail later, in this embodiment, the instantaneous reward reflects the average packet reception success rate across all devices, as well as the penalty due to a risk state (e.g., a state in which QoS targets such as reliability and delay are not achieved).

In the reinforcement learning unit 150 shown in Figure 2, the Q table management unit 151 holds, initializes, updates, etc. the Q table. The state calculation unit 152 calculates the state s(t). The reward calculation unit 153 calculates the reward r for s(t) and a(t). The risk assessment unit 154 calculates an evaluation function based on the Q table and selects an action. Note that the calculation of the evaluation function may be performed by the reward calculation unit 153.

Here, an overview of the operation of the radio base station 100 related to reinforcement learning will be explained with reference to the flowchart in Figure 4.

At S101, the state calculation unit 152 obtains the satisfaction level based on packet loss rate information (detected from ACK feedback) for each wireless interface and calculates the state s(t).

In S102, the risk assessment unit 154 determines action a using the ε-greedy method based on the multiple Q tables (or a single Q table) managed by the Q table management unit 151.

In S103, the reinforcement learning unit 150 notifies the total radio resource allocation calculation unit 142, the individual radio resource allocation calculation unit 143, etc. of the determined action a, and the radio base station 100 executes action a.

In S104, packet loss information is acquired by the communication quality measurement unit 141, and the packet loss information is passed to the reward calculation unit 153 in the reinforcement learning unit 150.

In S105, the reward calculation unit 153 calculates the reward. In S106, the Q table management unit 151 updates the multiple Q tables (or a single Q table).

Below, the operation of the radio base station 100 in this embodiment (particularly the operation of the reinforcement learning unit 150) will be explained in more detail using an example that uses a specific radio interface.

(System model)
In this embodiment, as shown in Fig. 5, downlink (DL) transmission in a wireless network configured with multiple APs accommodating multiple devices will be described as an example. Each AP is assumed to have a Sub-6 GHz and mmWave (millimeter wave) interface. Each AP corresponds to a wireless base station 100. A device corresponds to a wireless terminal 200. In the following description, the wireless base station 100 will be described as performing the reinforcement learning operation according to the present invention, but the wireless terminal 200 can also perform the same operation.

As shown in Figure 5, AP b transmits a desired packet to a set of devices K. The set of devices K also receives DL interference from all other APs b'≠b.

At the beginning of each scheduling frame t, AP b has L _k (t) packets to each device k ∈ K. Each packet l ∈ L _k (t) is d bits in size and is to be sent to device k ∈ K.

AP b transmits these packets via N subchannels on the Sub-6 GHz interface and M beams on the mmWave interface. Each Sub-6 GHz subchannel or each mmWave beam can be assigned to a unique device in each scheduling time frame. Multiple devices can be supported in each frame via different subchannels in Sub-6 GHz and different beams in mmWave.

In the Sub-6 GHz band, the signal-to-interference plus noise ratio (SINR) from AP b to device k on subchannel n is:

Here, the transmit power p _bkn ^sub on subchannel n from AP b to device k is assumed to be equal across subchannels. W _sub is the bandwidth per subchannel. The term h _bkn ^sub is the channel power between AP b and device k on subchannel n, and is given by h _bkn ^sub (t) = | ^∼ h _bkn ^sub (t) | ^2. Note that in the text of this specification, for convenience, a symbol that starts with a letter may be written before the letter. " ^∼ h" is an example. Here, ^∼ h _bkn ^sub (t) is the complex channel coefficient including small-scale and large-scale fading effects. _{σ n} ² represents the additive white Gaussian noise (AWGN) power. I _bkn ^sub is the interference power on subchannel n from APs b'≠b to device k.

For the mmWave interface, analog beamforming is assumed, and the transmission beam width and beam direction from AP b to device k on beam m are expressed as θ _bkm and β _bkm , respectively, and are adjusted according to the target device k and time frame t in each beam m.

For simplicity and without loss of generality, we assume that the receive beam gain G _k ^Rx at device k is fixed. To maximize the obtained rate, θ _bkm is set to the narrowest beamwidth, and β _bkm is given by the line-of-sight (LoS) direction from AP b to device k. Therefore, the SINR of beam m at device k served by AP b is given as follows:

where p _bkm ^mW and h _bkm ^mW are the transmission power and channel power, respectively, between AP b and device k on beam m, and W _mw is the bandwidth. The channel power h _bkm ^mW is a function of the transmit beam width and direction on beam m, as follows:

Here, PL _bkm denotes the path loss between AP b and device k on beam m, and G _b (θ _bkm , β _bkm ) is the main transmit beam gain between AP b and device k, which is modeled as follows:

where ε is the side lobe beam gain. In equation (2), I _bkm ^mW is the interference power from all APs b'≠b to device k accommodated by AP b, and is calculated based on their side lobe beam gains.

Therefore, the achievable rate for device k accommodated by AP b is:

where v = {Sub, mW} (Sub 6 GHz or mmWave). Given the low latency requirements of device applications, instantaneous CSI feedback from devices to APs is not assumed. Therefore, APs must make allocation decisions without knowing the achievable rate (Equation (5)).

The number of packets transmitted to device k on interface v in frame t is denoted as l _k ^v (t) ∈ {0, ..., L _k (t)}. Since L _k (t) is the total number of packets queued in frame t, l _k ^sub (t) + l _k ^mW (t) ≤ L _k (t). On each interface, the number of packets successfully received by device k, Ω _k ^v (t), can be calculated by AP b based on the ACK feedback from device k as follows:

where ω _kl ^v (t) denotes the feedback from device k for packet l on interface v at frame t, as follows:

Furthermore, the maximum number of packets of size d bits successfully received by device k on interface v within a frame of duration T _s is given by

is given as:

Here, since r _bk ^v (t) is unknown at the AP, l _k,max ^{v (t)} is also unknown at the AP. Therefore, if l _k ^v (t)≦l _k,max ^v (t), that is, if the number of transmitted packets in device k's assigned subchannel or beam is smaller than the number of packets that device k can receive, it is assumed that all these packets are received successfully and their ACKs are fed back to the AP. However, if l _k ^v (t)≧l _k,max ^v (t), then l _k ^v (t)-l _k,max ^v (t) packets are in the NACK state.

Based on the above, the PLR (packet loss rate) of device k at frame t is defined as the average packet loss up to frame t across both interfaces, as follows:

where:

Denote the Packet Successful Delivery Rate (PSR) across both interfaces at frame τ. The PLR of device k at frame t at each interface is updated as follows:

The method according to this embodiment will be described in detail below.

(About Markov Decision Processes (MDPs))
The goal here is to maximize the long-term PSR averaged over all devices (here, ρ _max ) while satisfying the individual PLR constraints of each device. This problem can be modeled as an MDP characterized by a state space, an action space, transition probabilities, and a reward function, as shown in FIG. 6. In FIG. 5, state s( _t) is the PLR satisfaction level (and ACK feedback state) for all devices. Action a( _t ) is the interface selection and packet scheduling for all devices. In this embodiment, the action is optimized by obtaining a reward r(t) based on state s(t) and action a(t) and maximizing the objective function.

Each AP (wireless base station) is an agent that makes interface selection and packet scheduling decisions. At each frame t, the AP knows the current state s _t , which consists of the current PLR satisfaction levels of devices associated with the AP and their feedback states in the previous frame t-1. Based on _{s t} , the AP takes action a _t . That is, the AP determines the number of packets on each interface of each device in the current frame t, obtains an immediate reward r _t from the environment, and transitions to a new state s _t+1 .

Since information such as real-time CSI and interface statistics is unknown, the AP does not have knowledge of the transition probability P(s _t+1 |s _t , a _t ). In this embodiment, this problem is solved using a reinforcement learning (RL) framework.

(Risk-Averse Reinforcement Learning)
To best satisfy stringent reliability requirements, the present embodiment uses a Risk-Sensitive Reinforcement Learning (RSRL) approach called Risk-Averse Average Q-learning (RAQL). Compared to traditional RL methods, which aim to maximize expected returns like QL, RSRL introduces the concept of risk, which is linked to the variance of rewards. RAQL achieves further variance reduction, thereby reducing risk.

Instead of taking the expected reward as the objective function as in traditional RL, we use the expected utility of the reward as the objective function:

In the above equation (11), the expectation is over the probabilistic policy π:S×A→[0,1] for choosing an action and the channel realization h over both interfaces. Taking a Taylor expansion, we obtain the following equation (12):

With β<0, the expected reward is maximized while the variance is minimized, making the objective function risk averse.

The symbols in the above formulas (11) and (12) have the following meanings:

J _π : Average utility function (discounted sum of immediate rewards r _t ) according to policy π in the Markov decision process
Π: Policy
E _π,h : Expected value under policy π and state h of the wireless channel (propagation path, etc.) r _t : Immediate reward value in process t β: Parameter Var[ ]: Variance of [ ] O(): Order of ( ) As will be described later, in this embodiment, multiple Q-tables are learned simultaneously by using equation (22) as the update rule. Then, the sample variance of these Q-tables is used as an approximation of the true variance. From this variance, a risk-averse ^Q-table is calculated and used for action selection.

RAQL-based interface selection and packet scheduling method
Next, the algorithm based on RAQL executed by the AP (wireless base station 100) in this embodiment will be described in detail. The state space and the action space are defined as follows.

State: s(t) is the current QoS satisfaction level of the PLR for all devices k∈K at frame t and the most recent ACK feedback for packets sent in frame t-1, as shown in equations (13) and (14) below. s(t) may not include ACK feedback.

where:

is.

Action: a(t) indicates the interface selection for each device over which packets should be transmitted. To avoid the explosion of the action space size and make the proposed method scalable, we aggregate the interface selection task and packet scheduling task into three actions a _k (t) for device k, as explained below. Although the AP does not have knowledge of the instantaneous CSI, it is reasonable to assume that the long-term CSI, such as the average path loss or average SINR, is known due to sporadic feedback.

Therefore, each AP can assign subchannels and beams based on the average CSI of each device. In this case, all subchannels are equivalent for each device, so the AP can randomly select each subchannel to be assigned to each device. The scheduling task of each AP then corresponds to determining the number of packets to be transmitted per subchannel for each device. The maximum number of packets transmitted from AP b and successfully received by device k during a frame length _Ts can be estimated using the following equation (15):

^r _bk ^v is the known average rate of device k at interface v. Each action a _k (t) is:

a _k (t) = 0: Only the Sub-6 GHz interface is used, and the number of transmitted packets is

is.

a _k (t) = 1: Only the mmWave interface is used, and the number of transmitted packets is

is.

a _k (t) = 2: Both the Sub-6 GHz and mmWave interfaces are used, but mmWave is given higher priority to take advantage of the high data rate and maximize the number of transmitted packets.

Finally, under the constraints of the number of subchannels and beams, the behavior a(t) for all devices is given by the following equation (20):

Reward: r(s(t), a(t)) represents the immediate reward achieved by performing action a(t) at frame t, given by the average PSR across devices. In particular, this reward function also considers the risk state defined in equation (14). Based on the ACK/NACK feedback for the AP to obtain Ω _k ^v (t) in equation (6), the reward is calculated by equation (21) below.

The meanings of the symbols in formula (21) are as follows:

r(s(t), a(t)): immediate reward value in process t Ω _k ^sub (τ): number of packets successfully transmitted via the Sub6GH I/F Ω _k ^mW (τ): number of packets successfully transmitted via the millimeter wave I/F l _k ^sub (τ): number of packets transmitted via the Sub6GH I/F l _k ^mW (τ): number of packets transmitted via the millimeter wave I/F u _k ^sub (t): variable that changes depending on whether the packet loss rate ρ at the Sub6GH I/F has reached the required quality ρ _max u _k ^mW (t): variable that changes depending on whether the packet loss rate ρ at the millimeter wave I/F has reached the required quality ρ _max As is clear from equation (14), u _k ^ν If (t)=0, ie, device k is in a risk state that does not satisfy the PLR in equation (14), the reward is penalized.

In this embodiment, the RAQL-based interface selection and packet scheduling method is executed by Algorithm 1 shown in Figure 7. In other words, the radio base station 100 executes this algorithm by, for example, running a program on the CPU. The meanings of each symbol are as follows:

ε: Exploration rate λ: Decay rate I: Number of Q-tables λ _p : Risk control parameter Q: Q-table V: Number of Q-table updates α: Learning rate In Algorithm 1, first, the AP initializes I Q-tables along with table V, which counts the number of selections of each action a under state s. The corresponding learning rate α is also initialized to 0, and the algorithm starts from a random state (lines 1-2).

At each frame t, a Q-table is randomly selected and used to calculate the risk aversion^Q-table using equation (24) described below (lines 3-5). Unlike traditional QL, in RAQL, the Q-function is updated using equation (22) below.

In equation (22), "x ₀ " is a constant, and is set, for example, as x ₀ = -1. α(s(t), a(t)) is the learning rate of the state-action pair (s(t), a(t)), γ is the decay rate, and u(x) is a monotonically increasing concave utility function, which is expressed as follows:

β is a parameter for imparting risk-averse characteristics, and here β<0. The risk-averse ^Q table is calculated by the following equation (24).

λ _p is the risk control parameter, ⁻ Q(s, a)=(1/I)Σ _i=1 ^I Q ⁱ (s, a) is the average Q table.

Next, given the current state and search rate ε, an action a(t) is selected using the ε-greedy strategy. The AP transmits a packet based on the selected action and receives an immediate reward (Equation (21)) (lines 6-9). The environment then transitions to a new state (lines 10-16). This process is repeated until the maximum number of frames T is reached.

(Example of hardware configuration)
Both the radio base station 100 and the radio terminal 200 can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on a cloud. Hereinafter, the radio base station 100 and the radio terminal 200 will be collectively referred to as communication devices.

In other words, a communication device can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the communication device. The program can be recorded on a computer-readable recording medium (such as portable memory) and then saved or distributed. The program can also be provided via a network such as the Internet or email.

Figure 8 is a diagram showing an example of the hardware configuration of the computer. The computer in Figure 8 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected by a bus BS. Note that the communication device may not have the display device 1006.

The program that realizes processing on the computer is provided by a recording medium 1001, such as a CD-ROM or memory card. When the recording medium 1001 storing the program is inserted into the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001; it can also be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

When an instruction to start a program is received, the memory device 1003 reads and stores the program from the auxiliary storage device 1002. The CPU 1004 implements functions related to the communication device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, touch panel, or the like, and is used to input various operational instructions. The output device 1008 outputs the results of calculations.

(Effects of the embodiment)
The technology according to the present embodiment can provide a technology for achieving both a desired communication quality and improved utilization efficiency of wireless resources while adapting to changes in the environment.

(Additional Note)
This specification discloses at least the following communication devices and communication methods.
(Section 1)
A communication device that performs wireless communication using a plurality of wireless interfaces,
a wireless interface for transmitting packets to a device; and a reinforcement learning unit for determining, using risk-averse reinforcement learning, the number of packets to be transmitted to the device via the wireless interface;
a transmitting unit that transmits the number of packets determined by the reinforcement learning unit to the device.
(Section 2)
The communication device described in paragraph 1, wherein the reinforcement learning unit learns actions for the states through risk-averse reinforcement learning, with the satisfaction level based on the packet loss rate on each wireless interface being the state, and the combination of wireless interfaces used by each device and the number of packets transmitted on each wireless interface being the actions.
(Section 3)
the reinforcement learning unit further includes a receiving unit that receives feedback from a plurality of devices that are packet destinations;
The communication device according to claim 2, wherein the reinforcement learning unit calculates the packet loss rate based on the feedback.
(Section 4)
The reinforcement learning unit calculates an immediate reward based on the average packet reception success rate for all devices and a penalty due to a risk state in which the QoS target value is not achieved, and calculates a policy that maximizes the average utility function using past immediate rewards so as to reflect a decrease in reward for high-risk behavior. The communication device described in any one of paragraphs 1 to 3.
(Section 5)
the communication device includes a first wireless interface and a second wireless interface that performs communication at a data rate higher than that of the first wireless interface;
The communication device described in any one of paragraphs 1 to 4, wherein the behavior selected by the reinforcement learning unit is one of three behaviors: using only the first wireless interface, using only the second wireless interface, and using the second wireless interface preferentially.
(Section 6)
6. A communication system including the communication device according to any one of claims 1 to 5.
(Section 7)
A communication method executed by a communication device that performs wireless communication using a plurality of wireless interfaces,
a reinforcement learning step of determining a wireless interface for transmitting packets to a certain device and the number of packets to be transmitted to the device via the wireless interface using risk-averse reinforcement learning;
a transmitting step of transmitting the number of packets determined by the reinforcement learning step to the device.

The present embodiment has been described above, but the present invention is not limited to this specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention as set forth in the claims.

100 Wireless base station 101 Antenna 110 Communication I/F unit 120 Control unit 130 Wireless communication unit 131 Receiving unit 132 Wireless communication signal generation unit 135 RF unit 140 Scheduler unit 141 Communication quality measurement unit 142 Total wireless resource allocation calculation unit 143 Individual wireless resource allocation calculation unit 150 Reinforcement learning unit 151 Q table management unit 152 State calculation unit 153 Reward calculation unit 154 Risk evaluation unit 200 Wireless terminal 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU
1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims (8)

A communication device that performs wireless communication using a plurality of wireless interfaces,
a wireless interface for transmitting packets to a device; and a reinforcement learning unit for determining, using risk-averse reinforcement learning, the number of packets to be transmitted to the device via the wireless interface;
a transmitting unit that transmits the number of packets determined by the reinforcement learning unit to the device;
A communication device comprising:
The reinforcement learning unit learns actions for the states through risk-averse reinforcement learning, where the state is a satisfaction level based on the packet loss rate in each wireless interface, and the actions are the combination of wireless interfaces used by each device and the number of packets transmitted in each wireless interface. the reinforcement learning unit further includes a receiving unit that receives feedback from a plurality of devices that are packet destinations;
The communication device according to claim 1 , wherein the reinforcement learning unit calculates the packet loss rate based on the feedback. A communication device that performs wireless communication using a plurality of wireless interfaces,
a wireless interface for transmitting packets to a device; and a reinforcement learning unit for determining, using risk-averse reinforcement learning, the number of packets to be transmitted to the device via the wireless interface;
a transmitting unit that transmits the number of packets determined by the reinforcement learning unit to the device;
A communication device comprising:
The reinforcement learning unit calculates an immediate reward based on the average packet reception success rate for all devices and a penalty due to a risk state in which the QoS target value is not achieved, and calculates a policy that maximizes the average utility function using past immediate rewards so as to reflect a decrease in reward for high-risk behavior. A communication device that performs wireless communication using a plurality of wireless interfaces,
a wireless interface for transmitting packets to a device; and a reinforcement learning unit for determining, using risk-averse reinforcement learning, the number of packets to be transmitted to the device via the wireless interface;
a transmitting unit that transmits the number of packets determined by the reinforcement learning unit to the device;
A communication device comprising:
the communication device includes a first wireless interface and a second wireless interface that performs communication at a data rate higher than that of the first wireless interface;
The behavior selected by the reinforcement learning unit is one of three behaviors: using only the first wireless interface, using only the second wireless interface, and using the second wireless interface preferentially. A communication system including a communication apparatus according to any one of claims 1 to 4 and said device. A communication method executed by a communication device that performs wireless communication using a plurality of wireless interfaces,
a reinforcement learning step of determining a wireless interface for transmitting packets to a certain device and the number of packets to be transmitted to the device via the wireless interface using risk-averse reinforcement learning;
a transmitting step of transmitting the number of packets determined by the reinforcement learning step to the device ,
In the reinforcement learning step, the communication device learns an action for the state by risk-averse reinforcement learning, where a satisfaction level based on a packet loss rate in each wireless interface is set as a state, and a combination of wireless interfaces used by each device and the number of packets transmitted in each wireless interface are set as actions.
Communication method . A communication method executed by a communication device that performs wireless communication using a plurality of wireless interfaces,
a reinforcement learning step of determining a wireless interface for transmitting packets to a certain device and the number of packets to be transmitted to the device via the wireless interface using risk-averse reinforcement learning;
a transmitting step of transmitting the number of packets determined by the reinforcement learning step to the device ,
In the reinforcement learning step, the communication device calculates an immediate reward based on an average packet reception success rate for all devices and a penalty due to a risk state in which the QoS target value is not achieved, and calculates a policy that maximizes an average utility function using past immediate rewards so as to reflect a decrease in reward for high-risk behavior.
Communication method . A communication method executed by a communication device that performs wireless communication using a plurality of wireless interfaces,
a reinforcement learning step of determining a wireless interface for transmitting packets to a certain device and the number of packets to be transmitted to the device via the wireless interface using risk-averse reinforcement learning;
a transmitting step of transmitting the number of packets determined by the reinforcement learning step to the device;
A communication method comprising:
the communication device includes a first wireless interface and a second wireless interface that performs communication at a data rate higher than that of the first wireless interface;
The behavior selected by the reinforcement learning step is one of three behaviors: using only the first wireless interface, using only the second wireless interface, and using the second wireless interface preferentially.
Communication method .