JP7510661B2 - Control quantity calculation device and control quantity calculation method - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies. Publication date: October 15, 2019 Publication: 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE) Proceedings, p. 129-130, Publisher: IEEE [Publications, etc.] Date held: October 15-18, 2019 (Announcement date: October 15, 2019) Name of meeting, venue: IEEE 8th Global Conference on Consumer Electronics (GCCE 2019) (Venue: Senri Life Science Center) [Publications, etc.] Date issued: January 7, 2020 Publication: The 34th International Conference on Information Networking (ICOIN 2020) Proceedings, p420-425, Publisher: IEEE [Publications, etc.] Date: January 7-10, 2020 (Announcement date: January 9, 2020) Meeting name, venue: The 34th International Conference on Information Networking (ICOIN 2020) (Venue: AC Hotel Barcelona Forum)

The present invention relates to a control amount calculation device and a control amount calculation method for calculating a control amount for controlling the amount of communication resources allocated to a slice.

The commercialization of the fifth generation (5G) mobile communications system will enable larger capacity, faster speeds, and multiple simultaneous connections in communications networks, facilitating the provision of a variety of services. In addition, one technology being considered for provision in the 5G mobile communications system is network slicing, which provides an optimal network environment for each of a variety of services. In this technology, a resource management mechanism that allocates communications resources (hereinafter simply referred to as resources) such as base stations, communication paths, switches, routers, and server CPUs to each slice set up in the network is important.

As resource management methods for allocating resources to each slice in a network, methods using genetic algorithms (see Non-Patent Document 1 below) and methods using deep reinforcement learning (see Non-Patent Document 2 below) are being considered.

B. Han,et al., “Slice as an Evolutionary Service: Genetic Optimization for Inter-Slice Resource Management in 5G Networks,” IEEE Access, vol. 6, pp. 33137-33147, 2018. R. Li et al., “Deep Reinforcement Learning for Resource Management in Network Slicing,” arXiv:1805.06591 [cs], May 2018.

In the methods described in Non-Patent Document 1 and Non-Patent Document 2 above, if the number of slices to be controlled changes, the model needs to be re-learned, making it difficult to respond to dynamic changes in the number of slices.

The present invention has been made in consideration of the above problems, and aims to provide a control amount calculation device and a control amount calculation method that can realize resource allocation to multiple slices even when the number of slices changes dynamically.

In order to solve the above problem, a control amount calculation device according to one embodiment of the present invention is a control amount calculation device that calculates a control amount for controlling the allocation amount of communication resources to multiple slices that are virtualized networks on a communication network, and includes multiple control amount derivation units that are provided corresponding to each of the multiple slices, acquire a state value related to the slice and a reward value related to the slice, and input the state value into a reinforcement learning model to determine and output an action that is a control amount for the slice, a learning data storage unit that stores learning data that is a combination of the state values and reward values acquired in the multiple control amount derivation units and the action determined corresponding to the state value, and a training unit that optimizes the reinforcement learning model shared by the multiple control amount derivation units by training using the learning data stored in the learning data storage unit.

Alternatively, a control amount calculation method according to another embodiment of the present invention is a control amount calculation method executed by a control amount calculation device that calculates a control amount for controlling the allocation amount of communication resources to a plurality of slices that are virtualized networks on a communication network, and includes a plurality of control amount derivation steps that are executed corresponding to each of the plurality of slices, acquire a state value for the slice and a reward value for the slice, and input the state value into a reinforcement learning model to determine and output an action that is a control amount for the slice, a learning data storage step that stores learning data that is a combination of the state values and reward values acquired in the plurality of control amount derivation steps and the action determined corresponding to the state value, and a training step that optimizes the reinforcement learning model shared by the plurality of control amount derivation steps by training using the learning data stored in the learning data storage step.

According to the above one or other of the above embodiments, by inputting state values for each of the multiple slices into a reinforcement learning model, an action for controlling the resource allocation amount for each slice is determined and output, and a combination of the state value and action used at that time and the reward value for that action is stored as learning data. At this time, the reinforcement learning model shared in the control of resource allocation of the multiple slices is optimized by training using the above learning data stored in advance. As a result, even if the number of slices in the network to be controlled changes dynamically, the reinforcement learning model used for resource allocation control for the multiple slices can be optimized using the control results for each slice as learning data. As a result, resource allocation to the multiple slices can be appropriately controlled.

Here, it is preferable that the state value for a slice includes at least the degree of satisfaction with the slice requirements and the utilization rate of the communication resources allocated to the slice. In this case, when controlling resource allocation to multiple slices, it is possible to control so as to satisfy the requirements of each slice, and to improve the efficiency of resource utilization.

It is also preferable that the reward value for a slice is a value that takes into account the satisfaction level regarding the requirements of the slice and the utilization rate of the communication resources allocated to the slice. In this case, when optimizing a reinforcement learning model for controlling resource allocation to multiple slices, it can be optimized to satisfy the requirements of each slice and to improve the utilization efficiency of resources.

It is also preferable to further include a control unit that controls the allocation amounts for the multiple slices based on the control amounts output by the multiple control amount derivation units. In this way, the resource allocation amounts for the multiple slices can be determined based on the control amounts output by the multiple control amount derivation units. As a result, for example, priority control based on a predetermined judgment criterion becomes possible.

It is also preferable that the control unit determines priorities for the multiple slices based on the immediately preceding resource allocation amounts for the multiple slices and the satisfaction levels of the requirements for the multiple slices, and executes control for the multiple slices according to the order indicated by the priorities. In this case, priority control of resource allocation based on priorities determined from the immediately preceding resource allocation amounts and the satisfaction levels of the slice requirements becomes possible. This enables smooth resource allocation to the multiple slices.

It is also preferable that the amount of control for a slice relates to the number of resource blocks in the radio access network. In this case, the allocation of resource blocks to multiple slices can be appropriately controlled.

According to the present invention, it is possible to allocate resources to multiple slices even when the number of slices changes dynamically.

1 is a diagram showing a schematic configuration of a control system according to a preferred embodiment of the present invention; 2 is a block diagram showing a hardware configuration of the control amount calculation device shown in FIG. 1; FIG. 2 is a block diagram showing a functional configuration of the control amount calculation device shown in FIG. 1 . FIG. 4 is a diagram showing an example of a network structure of a learning model used by the control amount derivation units 31 ₁ to 31 _N in FIG. 3. 10 is a diagram for explaining an overall picture of RB allocation to slices by the control amount calculation device 3 and the control device 5. FIG. 11 is a diagram showing a change over time in the amount of RBs allocated to each slice by the control amount calculation device 3 and the control device 5 allocating RBs to the slices. FIG. 10 is a flowchart showing a procedure of an RB allocation process in a control amount calculation method according to a preferred embodiment of the present invention. 5 is a flowchart showing a procedure of a learning process in a controlled variable calculation method according to a preferred embodiment of the present invention. 11 is a graph showing the results of evaluating the performance of RB allocation control by the control system 1 according to the present embodiment through simulation calculations. 11 is a graph showing the results of evaluating the performance of RB allocation control by the control system 1 according to the present embodiment through simulation calculations.

Below, a preferred embodiment of the control system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are given the same reference numerals, and duplicated explanations will be omitted.

The control system 1 shown in FIG. 1, which is a preferred embodiment of the present invention, is a computer system that controls resource allocation to slices for mobile communication systems such as the fifth generation mobile communication (5G). A slice is one of multiple virtualized networks into which the resources of the entire communication network are divided.

The control object of the control system 1 is a plurality of slices set in the mobile communication system, and the number of the plurality of slices can change dynamically. For example, in this embodiment, as the plurality of slices, slices S ₁ , S ₂ , ..., S _N (N is any natural number), and S _B are assumed to be (N+1) slices including slices S ₁ to S _N , S B, each of which has different slice requirements defined by throughput, delay, and reliability. Each of these slices S 1 to S N , S _B is configured by dividing resources of a radio access network (RAN) including a base station BS, and is shared by a plurality of user terminals UE (User Equipment).

The control system 1 includes a control amount calculation device 3 that calculates a control amount for controlling the allocation of resource blocks, which are resources for slices S ₁ , S ₂ , ..., S _N , and a control device (control unit) 5 that controls the allocation amount of resource blocks (hereinafter also referred to as RB) for slices S ₁ , S ₂ , ..., S _N , and S _B based on the control amount calculated by the control amount calculation device 3. Here, in this embodiment, the control amount calculation device 3 and the control device 5 are configured as separate devices, but may be integrated devices. The control device 5 is configured to be able to transmit and receive various data between the mobile communication system so as to be able to control the mobile communication system, and the control amount calculation device 3 is configured to be able to transmit and receive data between the control device 5, and receives various data such as state values or reward values, which will be described later, via the control device 5. The RBs, the allocation of which is controlled by the control system 1, are a type of resource in the RAN, and are blocks obtained by dividing a two-dimensional area consisting of a frequency axis and a time axis, in which a certain frequency band is divided by frequency and the time axis is also divided.

Figure 2 shows the hardware configuration of the computer 20 constituting the control amount calculation device 3. As shown in Figure 2, the computer 20 is physically a computer including a processor, a CPU (Central Processing Unit) 101, a recording medium, a RAM (Random Access Memory) 102 or a ROM (Read Only Memory) 103, a communication module 104, and an input/output module 106, each of which is electrically connected. The computer 20 may include a display, keyboard, mouse, touch panel display, etc. as the input/output module 106, or may include a data recording device such as a hard disk drive or semiconductor memory. The computer 20 may also be composed of multiple computers. The control device 5 has a similar hardware configuration.

3 is a block diagram showing a functional configuration of the control amount calculation device 3. The control amount calculation device 3 includes N control amount derivation units 31 ₁ to 31 _N , a training unit 32, and a learning data storage unit 33. The control amount derivation units 31 ₁ to 31 _N are provided in a number corresponding to the number of slices S ₁ to S _N of the control object. Each functional unit of the control amount calculation device 3 shown in FIG. 3 is realized by reading a program onto hardware such as the CPU 101 and the RAM 102, operating the communication module 104 and the input/output module 106 under the control of the CPU 101, and reading and writing data in the RAM 102. The CPU 101 of the control amount calculation device 3 executes this computer program to make the control amount calculation device 3 function as each functional unit of FIG. 3, and sequentially executes processing corresponding to a control amount calculation method described later. All of the various data required for executing this computer program and all of the various data generated by executing this computer program are stored in built-in memories such as the ROM 103 and the RAM 102, or in storage media such as a hard disk drive.

The control amount derivation units 31 ₁ to 31 _N each determine a control amount (RB amount) for controlling the allocation amount of RB for the corresponding slices S ₁ to S _N , and output the control amount to the control device 5. That is, the control amount derivation units 31 ₁ to 31 _N acquire state values for the corresponding slices S ₁ to S _N from the mobile communication system via the control device 5, input the state values to a common learning model of deep reinforcement learning, calculate an action value using the learning model, and select an action that maximizes the action value. In addition, the control amount derivation units 31 ₁ to 31 _N acquire reward values for the slices S ₁ to S _N for the selected action via the control device 5. Furthermore, the control amount derivation units 31 ₁ to 31 _N calculate a control amount (RB amount) based on the selected action and output it to the control device 5. In addition, the control amount derivation units 31 ₁ to 31 _N store a combination of a state value, an action, and a reward value for the action as learning data in the learning data storage unit 33 for each control amount calculation step.

In detail, the control amount derivation units 31 ₁ to 31 _N acquire, as state values, a throughput value and a delay amount indicating the requirements of slices S ₁ to S _N , an NS requirement satisfaction (NSRS) value indicating the user's satisfaction with the slice requirements, an RB usage ratio (RBUR) value indicating the usage rate of slices S ₁ to S _N , the number of RBs actually allocated to slices S ₁ to S _N , and the number of arriving packets, the number of transmitted packets, and the number of packets in the buffer (waiting to be transmitted) on slices S ₁ to S _N in the base station BS to be controlled. Among the above state values, the throughput value, the delay amount, and the NSRS value are values indicating the state regarding the requirements of slices S ₁ to S _N , the RBUR value and the number of RBs are values indicating the state regarding the usage rate of slices S ₁ to S _N , and the number of arriving packets, the number of transmitted packets, and the number of packets in the buffer are state values acquired auxiliary to eliminate ambiguity of these state values.

The NSRS value among the above state values is generated by the base station BS or the like by collecting and aggregating data from the user terminal UE, and is expressed by the following formula (1):

Here, Su indicates the satisfaction of the slice requirements for each user with "0" or "1", and ues indicates the number of users accommodated in the slice. This NSRS value indicates the average satisfaction of users, and the closer it is to 1, the more the slice requirements are satisfied.

Moreover, the RBUR value among the above state values is generated by the base station BS or the like, and is expressed by the following formula (2):

Here, URB indicates the number of RBs actually used, and MRB indicates the number of RBs actually allocated to the corresponding slice. The RBUR value indicates the usage rate of the corresponding slice, and the closer it is to 1, the less excess RBs are allocated.

The control amount derivation units 31 ₁ to 31 _N use the Ape-X technique, which applies distributed learning to DQN, as a learning model to which state values are input. That is, each of the control amount derivation units 31 ₁ to 31 _N applies state values related to the slices S ₁ to S _N of the control target to the learning model to determine an action, obtains a reward value for the action, and collects and accumulates the state value, action, and reward value as experience (learning data). In response to this, the training unit 32, which will be described later, optimizes the parameters of the learning model by training (learning) using this learning data.

FIG. 4 shows an example of a network structure of a learning model used by the control amount derivation units 31 ₁ to 31 _N. As shown in FIG. 4, the learning model includes an input layer N _IN to which a state value is input, a fully connected layer N ₁ to which the state value input from the input layer N _IN is propagated in order, a batch normalization layer N ₂ , a fully connected layer N ₃ , a batch normalization layer N ₄ , a fully connected layer N ₅ , and a batch normalization layer N ₆ , which are branched and connected from the batch normalization layer N ₆ , a fully connected layer N ₇ , a batch normalization layer N ₈ , and a fully connected layer N ₉ , a fully connected layer N ₁₀ , a batch normalization layer N ₁₁ , and a fully connected layer N ₁₂ , and an output layer N _OUT connected to the fully connected layers N ₉ and N _12. In such a learning model, by inputting a state value to the input layer N _IN , an action value, which is a reward value expected as a result of an action, is output from the output layer N _OUT .

Furthermore, the control amount derivation units 31 ₁ to 31 _N select an action that maximizes the action value based on the action value obtained as a result of inputting the state value into the learning model (greedy method). In detail, the control amount derivation units 31 ₁ to 31 _N obtain an action value Q(S _t , a _t , θ) for the state value s _t and action a _t at time t using the learning model shown in FIG. 4 (θ is a parameter such as a weighting parameter of the neural network), and determine an action a (a is an integer of 1 or more) that maximizes the action value Q(S _t , a _t , θ). Then, based on the determined action a, the control amount derivation units 31 ₁ to 31 _N calculate the relative amount of RB (IDRB) at time t to be assigned to the corresponding slice, using the following formula (3);

In the above formula (3),

is a floor function. Furthermore, the control amount derivation units 31 ₁ to 31 _N add a relative amount (IDRB) to the allocated RB amount (ARB) at time t with respect to the allocated RB amount at time t−1 one step before, to obtain the following formula (4);
_ARBt = ARBt _-1 + _IDRBt ... (4)
The calculated amount of allocated RBs is output to the control device 5.

The training unit 32 updates the parameter θ of the learning model shared by the control amount derivation units 31 ₁ to 31 _N to an optimal value using the learning data stored in the learning data storage unit 33 (training). That is, the training unit 32 acquires the reward value at time t+1 one step later and the state value at time t+1, which are included in one set of learning data, and calculates the target value y _t to be output by the learning model at time t based on these values. This reward value is a value taking into account the NSRS value and the RBUR value, for example, a value obtained by multiplying the NSRS value and the RBUR value. Then, the training unit 32 updates the parameter θ so that the action value Q(S _t , a _t , θ) determined based on the state value s _t at time t approaches the target value y _t .

Here, the training unit 32 randomly extracts learning data to be used for training based on the priority from among the learning data stored in the learning data storage unit 33. In addition, the training unit 32 updates the priority of the learning data stored in the learning data storage unit 33 while performing training so that the older the experience, the lower the priority of the learning data. Furthermore, the training unit 32 periodically copies the parameter θ updated by training to the control amount derivation units 31 ₁ to 31 _N , thereby updating the parameter θ in each of the control amount derivation units 31 ₁ to 31 _N. Here, each of the control amount derivation units 31 ₁ to 31 _N may obtain the parameter θ updated by the training unit 32 and update the internal parameter θ.

Next, the functions of the control device 5 will be explained.

The control device 5 has a function of periodically acquiring state values and reward values from the mobile communication system and transferring them to the control amount calculation device 3, and a function of controlling to determine the amount of RBs (MRBs) to be allocated to each slice S ₁ , S ₂ , ..., S _N , S _B at time t based on the amount of allocated RBs (ARBs) at time t output from a plurality of control amount derivation units _{31 1} , 31 2 , ..., 31 N of the control amount calculation device 3. That is, for slices S ₁ , S ₂ , ..., S _N other than slice S _B , the control device 5 executes control to determine the MRB at time t for slices S _{1 , S 2 ,} ..., S _N to a value equal to the ARB, in ascending order of priority, using a value obtained by multiplying the MRB and NSRS value at the immediately preceding time t- ₁ as a priority. At this time, if the ARB is larger than the number of remaining RBs, the number of remaining RBs is determined as the MRB. By such priority control, slices requiring many RBs are given priority in allocation so that the slices requiring fewer RBs do not occupy the RBs. Furthermore, for slice S 1 -S ₂ , the control device 5 determines the remaining RB amount as the MRB, which is determined from the total value of the MRBs allocated to slices S ₁ , S ₂ , ..., S _N.

Fig. 5 is a diagram for explaining an overall picture of RB allocation to slices by the control amount calculation device 3 and the control device 5, and Fig. 6 is a diagram showing a time change of RB amounts allocated to slices S ₁ , S ₂ , and S _N by RB allocation to slices by the control amount calculation device 3 and the control device 5. In this way, the training unit 32 of the control amount calculation device 3 learns a learning model based on learning data accumulated in the learning data storage unit 33 as experience of each control amount derivation unit 31 ₁ , 31 ₂ , ..., 31 _N. Then, the parameters of the learning model optimized by learning are copied to each control amount derivation unit 31 ₁ , 31 ₂ , ..., 31 _N in the control amount calculation device 3, and each control amount derivation unit 31 ₁ , 31 ₂ , ..., 31 _N inputs a state value to a common learning model, thereby selecting an action targeted at each slice S ₁ , S ₂ , ..., S _N. In the control device 5, the RB amount to be allocated to each slice S ₁ , S ₂ , ..., S _N , and S _B is determined based on the action selected for each slice S ₁ , S ₂ , ..., S _N , and the resource of the RB amount is allocated to each slice S ₁ , S ₂ , ..., S _N , and S _B. According to the example shown in Fig. 6, as time passes, slice S ₁ is allocated RB amounts of "0", "3", and "0" in order, slice S ₂ is allocated RB amounts of "3", "2", and "6" in order, and slice S _N is allocated RB amounts of "2", "1", and "0" in order.

Next, the steps of the control amount calculation method executed by the above-mentioned control system 1 will be described. FIG. 7 is a flowchart showing the steps of the RB allocation process in the control amount calculation method, and FIG. 8 is a flowchart showing the steps of the learning process in the control amount calculation method. The RB allocation process shown in FIG. 7 is repeatedly executed at times t, t+1, ... of multiple steps, and the learning process shown in FIG. 7 is repeatedly executed at regular intervals or when the number of experiences accumulated by the RB allocation process reaches a certain number.

First, referring to Fig. 7, when the RB allocation process is started, the control amount derivation unit ₃₁₁ of the control amount calculation device 3 acquires a state value for the slice _S1 (step S101). Next, the control amount derivation unit ₃₁₁ inputs the acquired state value into a learning model to calculate an action value, and selects an action that maximizes the action value (step S102). After that, the control amount derivation unit ₃₁₁ determines an allocated RB amount (ARB) for the slice _S1 based on the selected action, and outputs the RB amount to the control device 5 (step S103). Furthermore, the control amount derivation unit ₃₁₁ acquires a reward value for the slice _S1 for the previous RB allocation process, and a combination of the state value, action, and reward value is stored in the learning data storage unit 33 as learning data (step S104). The series of processes from steps S101 to S104 are repeated by the control amount derivation units 31 ₂ to 31 _N for the remaining slices S ₂ to S _N (step S105).

Next, the control device 5 determines the priorities of slices S ₁ to S _N based on the RB amount (MRB) and NSRS value at the immediately preceding time t-1 (step S106). After that, the control device 5 determines the RB amount (MRB) for slices S ₁ to S _N based on the allocated RB amount (ARB) for slices S ₁ to S _N in the determined order of priority, and controls RB allocation to slices S ₁ to S _N in the order of priority (step S107). Finally, the control device 5 executes control to allocate the remaining RB to slice S _B (step S108).

Next, referring to Fig. 8, when the learning process is started, the training unit 32 of the controlled variable calculation device 3 randomly extracts learning data to be used for training from the learning data stored in the learning data storage unit 33 while weighting based on the priority (step S201). Next, the training unit 32 updates the parameter θ of the learning model of the neural network by training using the extracted learning data (step S202). After that, the training unit 32 copies the updated parameter θ into each of the controlled variable derivation units 31 ₁ , 31 ₂ , ... 31 _N , thereby updating the learning model in each of the controlled variable derivation units 31 ₁ , 31 ₂ , ... 31 _N (step S203). Finally, the training unit 32 updates the priority of the learning data stored in the learning data storage unit 33 so that the older the learning data, the lower the priority (step S204).

The effects of the control system 1 of the above embodiment will be described.

According to this embodiment, by inputting a state value into the reinforcement learning model for each of the slices S ₁ to S _N , an action for controlling the resource allocation amount for each of the slices S ₁ to S _N is determined and output, and a combination of the state value and the action used at that time and the reward value for the action is stored as learning data. At this time, the reinforcement learning model shared in the control of the resource allocation of the slices S ₁ to S _N is optimized by training using the above-mentioned learning data stored in advance. As a result, even if the number of slices in the network to be controlled changes dynamically, the reinforcement learning model used for the resource allocation control for the slices S ₁ to S _N can be optimized by using the control results for each slice as learning data. As a result, it is possible to appropriately control the resource allocation to the slices S ₁ to S _N that does not depend on the number of slices.

Here, the state values for the slices S ₁ to S _N include an NSRS value and an RBUR value. In this case, when controlling resource allocation for a plurality of slices S ₁ to S _N , it is possible to control so as to satisfy the requirements of each slice S ₁ to S _N , and to improve the utilization efficiency of the resource RB.

In addition, the reward value for slices S ₁ to S _N is a value that takes into account the NSRS value and the RBUR value. In this case, when optimizing a learning model for controlling resource allocation for a plurality of slices S ₁ to S _N , it is possible to optimize the slices S ₁ to S _N so as to satisfy the requirements of each slice S 1 to S N and to improve the utilization efficiency of the resource RB.

The functions of the control device 5 also include a function of controlling the allocation RB amount (MRB) for the multiple slices S ₁ to S _N based on the RB amount (ARB) output by the multiple control amount derivation units 31 ₁ , ... ₃₁ N. With this function, the source allocation amount for the multiple slices S ₁ to S _N can be determined based on the control amount output by the multiple control amount derivation units 31 ₁ , ... 31 _N. As a result, for example, priority control based on a predetermined judgment criterion becomes possible.

Specifically, in this embodiment, the priority of resource allocation to the slices S ₁ to S _N by the control device 5 is determined based on the immediately preceding allocated RB amount (MRB) and the NSRS value. In this case, by preventing slices requiring many RBs from occupying RBs, smooth resource allocation to the slices S ₁ to S _N is possible.

Here, the results of evaluating the performance of the RB allocation control by the control system 1 according to the present embodiment by simulation calculation are shown in comparison with comparative examples. FIG. 9 is a graph showing the average NSRS value in comparison with comparative examples, and FIG. 10 is a graph showing the average RBUR value in comparison with comparative examples. Comparative Example 1 is an example of a case where all RBs are shared between slices without performing network slicing, Comparative Example 2 is an example of a case where RBs are controlled to be equally divided between slices, Comparative Example 3 is an example of a case where RBs are controlled to be divided by weighting the number of packets arriving at the base station BS, and Comparative Example 4 is an example of a case where control is performed by an existing method using deep reinforcement learning (the method described in “R. Li et al., “Deep Reinforcement Learning for Resource Management inNetwork Slicing,” arXiv:1805.06591[cs], May 2018.”). From these evaluation results, it was found that according to the present embodiment, it is possible to realize control that satisfies the requirements of various services for all users, and to reduce the allocation of excessive RBs in various services, thereby maintaining a high RB usage rate.

The present invention is not limited to the above-described embodiment. The configuration of the above embodiment may be modified in various ways.

REFERENCE SIGNS LIST 1 control system, 3 controlled variable calculation device, 5 control device (control unit), 31 ₁ to 31 _N controlled variable derivation unit, 32 training unit, 33 learning data storage unit, RB resource, S ₁ to S _N , S _B slices.

Claims (7)

A control amount calculation device that calculates a control amount for controlling an allocation amount of communication resources to a plurality of slices that are virtualized networks on a communication network, comprising:
A plurality of control variable derivation units are provided corresponding to the plurality of slices, each of which acquires a state value and a reward value related to each of the slices, and each of which includes a reinforcement learning model that inputs the state value, determines an action that is the control variable for each of the slices, and outputs the action;
a learning data storage unit that stores learning data that is a combination of the state value and the reward value acquired in the plurality of control amount derivation units and the action determined corresponding to the state value;
a training unit that optimizes, by training , a parameter shared by the reinforcement learning models included in each of the plurality of control variable derivation units, using the learning data stored in the learning data storage unit;
A control amount calculation device comprising: The state value for the slice includes at least a satisfaction level for a requirement of the slice and a utilization rate of a communication resource allocated to the slice.
The control amount calculation device according to claim 1. The reward value for the slice is a value that takes into account the satisfaction level for the requirements of the slice and the usage rate of the communication resources allocated to the slice.
The control amount calculation device according to claim 1 or 2. A control unit that controls the allocation amounts for the plurality of slices based on the control amounts output by the plurality of control amount derivation units.
The control amount calculation device according to any one of claims 1 to 3. The control unit determines priorities for the slices based on the immediately preceding allocation amount of the communication resources for the slices and the satisfaction level of the requirements for the slices, and executes the control for the slices in accordance with an order indicated by the priorities.
5. The control amount calculation device according to claim 4. the amount of control for the slice relates to a number of resource blocks in the radio access network;
The control amount calculation device according to any one of claims 1 to 5. A control amount calculation method executed by a control amount calculation device that calculates a control amount for controlling an allocation amount of communication resources to a plurality of slices that are virtualized networks on a communication network, comprising:
A plurality of control variable derivation steps are executed corresponding to each of the plurality of slices, and obtain a state value and a reward value for each of the slices, and determine and output the control variable by using a reinforcement learning model that inputs the state value and determines and outputs an action that is the control variable for each of the slices;
a learning data storage step of storing learning data which is a combination of the state value and the reward value acquired in the plurality of control amount derivation steps and the action determined corresponding to the state value;
a training step of optimizing, by training , parameters shared by the reinforcement learning models used in each of the plurality of control variable derivation steps, using the learning data stored in the learning data storage step;
A control amount calculation method comprising:

Images