JP2021158591A - Control amount calculation device and control amount calculation method - Google Patents

Abstract

To realize allocation of a resource to a plurality of slices even when the number of slices is dynamically changed.SOLUTION: A control amount calculation device 3 is a device that calculates a control amount for controlling allocation amounts of RB to a plurality of slices S1 to SN, and comprises: a plurality of control amount lead-out parts 311 to 31N that is provided so as to correspond respectively to the plurality of slices S1 to SN, acquires state values and compensation values related to the respective slices S1 to SN, and determines and outputs behavior to the slices S1 to SN by inputting the state values into a learning model; a learning data storage part 33 that stores learning data which are combinations of the state values and the compensation values acquired in the plurality of control amount lead-out parts 311 to 31N and the behavior determined so as to correspond to the state values; and a training part that optimizes the learning model shared by the plurality of control amount lead-out parts 311 to 31N, through training using the learning data stored in the learning data storage part 33.SELECTED DRAWING: Figure 3

Description

The present invention relates to a control amount calculation device and a control amount calculation method for calculating a control amount for controlling the allocation amount of communication resources for a slice.

The commercialization of 5th generation (5G) mobile communication systems will enable larger capacity, higher speed, and multiple simultaneous connections of communication networks, which will facilitate the provision of various services. Further, as a technology under consideration for provision in a 5G mobile communication system, there is a technology called network slicing that provides an optimum network environment for various services. In this technology, there is a resource management mechanism that allocates communication resources (hereinafter, also simply referred to as resources) such as base stations, communication paths, exchanges, routers, and server CPUs to each slice set in the network. It becomes important.

As a resource management method for allocating resources to each slice in the network, a method using a genetic algorithm (see Non-Patent Document 1 below) and a method using deep reinforcement learning (see Non-Patent Document 2 below) are available. It is being considered.

B. Han, et al., “Slice as an Evolutionary Service: Genetic Optimization for Inter-Slice Resource Management in 5G Networks,” IEEE Access, vol. 6, pp. 33137-33147, 2018. R. Li et al., “Deep Reinforcement Learning for Resource Management in Network Slicing,” arXiv: 1805.06591 [cs], May 2018.

In the methods described in Non-Patent Document 1 and Non-Patent Document 2 described above, it is necessary to relearn the model when the number of slices to be controlled changes, and it is difficult to cope with the dynamic change in the number of slices.

The present invention has been made in view of the above problems, and provides a control amount calculation device and a control amount calculation method capable of allocating resources to a plurality of slices even when the number of slices changes dynamically. With the goal.

In order to solve the above problems, the control amount calculation device according to one embodiment of the present invention is a control amount for calculating a control amount for controlling the allocation amount of communication resources for a plurality of slices which are virtual networks on a communication network. It is a calculation device, which is provided corresponding to each of a plurality of slices, and is a control amount for slices by acquiring a state value related to the slice and a reward value related to the slice and inputting the state value into the reinforcement learning model. Learning data that is a combination of a plurality of control amount deriving units that determine and output actions, state values and reward values acquired by the plurality of control amount deriving units, and actions determined in response to the state values. It includes a learning data storage unit for storing and a training unit for optimizing a reinforcement learning model shared by a plurality of control amount derivation units by training using the learning data stored in the learning data storage unit.

Alternatively, the control amount calculation method according to another embodiment of the present invention is executed by a control amount calculation device that calculates a control amount for controlling the allocation amount of communication resources for a plurality of slices that are virtual networks on the communication network. This is a control amount calculation method that is executed for each of a plurality of slices, obtains a state value related to the slice and a reward value related to the slice, and inputs the state value into the reinforcement learning model for the slice. A combination of a plurality of control amount derivation steps that determine and output an action that is a control amount, a state value and a reward value acquired in the plurality of control amount derivation steps, and an action determined corresponding to the state value. A learning data storage step that stores a certain learning data and a training step that optimizes a reinforcement learning model shared by a plurality of control amount derivation steps by training using the learning data stored by the training data storage step. Be prepared.

According to the above one form or the above other form, by inputting the state value into the reinforcement learning model for each of a plurality of slices, the action for controlling the resource allocation amount for each slice is determined and output, and at that time, the action is determined and output. The combination of the state value and the action used in the above and the reward value for the action is stored as learning data. At this time, the reinforcement learning model shared by controlling the resource allocation of a plurality of slices is optimized by training using the above-mentioned learning data stored in advance. As a result, even when the number of slices in the controlled network changes dynamically, the reinforcement learning model used for resource allocation control for multiple slices is optimized by using the control results for each slice as training data. can do. As a result, resource allocation to multiple slices can be appropriately controlled.

Here, the state value relating to the slice preferably includes at least the satisfaction level regarding the requirement of the slice and the utilization rate of the communication resource allocated to the slice. In this case, when controlling resource allocation for a plurality of slices, it is possible to control so as to satisfy the requirements of each slice, and it is possible to improve the resource utilization efficiency.

It is also preferable that the reward value for the slice is a value that takes into account the satisfaction level regarding the slice requirement and the usage rate of the communication resource allocated to the slice. In this case, when optimizing the reinforcement learning model for controlling resource allocation for multiple slices, it can be optimized to meet the requirements of each slice and improve resource utilization efficiency. Can be optimized for.

It is also preferable to further include a control unit that controls the allocation amount for the plurality of slices based on the control amount output by the plurality of control amount derivation units. In this way, the resource allocation amount for the plurality of slices can be determined based on the control amount output by the plurality of control amount derivation units. As a result, for example, priority control based on a predetermined determination standard can be performed.

In addition, the control unit determines the priority for the plurality of slices based on the allocation amount of the immediately preceding resource for the plurality of slices and the satisfaction with the requirements of the plurality of slices, and controls the plurality of slices according to the order indicated by the priority. It is also preferable to carry out. In this case, priority control of resource allocation based on the priority determined from the immediately preceding resource allocation amount and the satisfaction level of the slice requirement becomes possible. This enables smooth resource allocation for multiple slices.

It is also preferable that the control amount for the slice is related to the number of resource blocks in the radio access network. In this case, the allocation of resource blocks to a plurality of slices can be appropriately controlled.

According to the present invention, resource allocation to a plurality of slices can be realized even when the number of slices changes dynamically.

It is a figure which shows the schematic structure of the control system which concerns on one preferred embodiment of this invention. It is a block diagram which shows the hardware configuration of the control amount calculation apparatus of FIG. It is a block diagram which shows the functional structure of the control amount calculation apparatus of FIG. It is a figure which shows an example of the network structure of the learning model used by the control quantity derivation part 31 _{1 to} 31 _{N of FIG.} It is a figure for demonstrating the whole image of the RB allocation to a slice by a control amount calculation device 3 and control device 5. It is a figure which shows the time change of the RB amount assigned to each slice by the RB allocation to a slice by a control amount calculation device 3 and control device 5. It is a flowchart which shows the procedure of the RB allocation processing in the control amount calculation method which concerns on one preferred embodiment of this invention. It is a flowchart which shows the procedure of learning processing in the control amount calculation method which concerns on one preferred embodiment of this invention. It is a graph which shows the result of having evaluated the performance of the RB allocation control by the control system 1 which concerns on this Embodiment by a simulation calculation. It is a graph which shows the result of having evaluated the performance of the RB allocation control by the control system 1 which concerns on this Embodiment by a simulation calculation.

Hereinafter, preferred embodiments of the control system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are designated by the same reference numerals, and duplicate description will be omitted.

The control system 1, which is a preferred embodiment of the present invention shown in FIG. 1, is a computer system that controls resource allocation to slices for a mobile communication system such as a fifth-generation mobile communication (5G). Is. A slice is one of the virtualized networks in which the resources of the entire communication network are divided into a plurality of parts.

The control target of the control system 1 is a plurality of slices set in the mobile communication system, and the number of these plurality of slices can be dynamically changed. For example, in the present embodiment, as a plurality of slices, the requirements of slices defined by throughput, delay, and reliability are different from each other, slice S ₁ , slice S ₂ , ..., Slice S _N (N is an arbitrary natural number). , and the slice including _{S B} (N + 1) is assumed slices. These slices _S 1 _~S N, _{S B,} respectively, a radio access network including a base station BS (Base Station) or the like: by dividing the resources (RAN Radio Access Network) is constructed, a plurality of user terminals UE ( Shared by User Equipment).

The control system 1 includes a slice S _1, slice S _2, ..., a control amount calculation unit 3 for calculating a control amount for controlling the allocation of resource block is a resource to the slice S _N, is calculated by the control amount calculating device 3 based on the control amount, the slice S _1, slice S _2, ..., slice S _N, the resource block to the slice S _B controller (control unit) that controls the quota (hereinafter also referred to as RB) and 5 include. Here, in the present embodiment, the control amount calculation device 3 and the control device 5 are configured as separate devices, but they may be integrated devices. The control device 5 is configured to be capable of transmitting and receiving various data to and from the mobile communication system so that the mobile communication system can be controlled, and the control amount calculation device 3 transmits and receives data to and from the control device 5. It is possible to receive various data such as a state value or a reward value, which will be described later, via the control device 5. The RB whose allocation is controlled by the control system 1 is a kind of resource in the RAN, and is a two-dimensional structure composed of a frequency axis and a time axis in which the frequency is divided within a certain frequency band and the time axis is also divided. It is a block that divides the area.

FIG. 2 shows the hardware configuration of the computer 20 that constitutes the control amount calculation device 3. As shown in FIG. 2, the computer 20 physically includes a CPU (Central Processing Unit) 101 as a processor, a RAM (Random Access Memory) 102 or a ROM (Read Only Memory) 103 as a recording medium, and a communication module 104. , And a computer or the like including the input / output module 106 and the like, and each of them is electrically connected. The computer 20 may include a display, a keyboard, a mouse, a touch panel display, and the like as the input / output module 106, and may include a data recording device such as a hard disk drive and a semiconductor memory. Further, the computer 20 may be composed of a plurality of computers. The control device 5 also has a similar hardware configuration.

FIG. 3 is a block diagram showing a functional configuration of the control amount calculation device 3. The control amount calculation device 3 includes N control amount derivation units 31 _{1 to} 31 _N , a training unit 32, and a learning data storage unit 33. The control amount deriving units 31 _{1 to} 31 _N are provided in an number corresponding to the number of slices S _{1 to SN to be controlled.} Each functional unit of the control amount calculation device 3 shown in FIG. 3 loads the communication module 104, the input / output module 106, and the like under the control of the CPU 101 by loading the program on the hardware such as the CPU 101 and the RAM 102. It is realized by operating and reading and writing data in the RAM 102. The CPU 101 of the control amount calculation device 3 causes the control amount calculation device 3 to function as each functional unit of FIG. 3 by executing this computer program, and sequentially executes processes corresponding to the control amount calculation method described later. Various data necessary for executing this computer program and various data generated by executing this computer program are all stored in an internal memory such as ROM 103 and RAM 102, or a storage medium such as a hard disk drive.

Control amount derivation unit ₃₁ 1 to 31 _N, respectively, the control amount for controlling the allocation of RB for the corresponding slice _S 1 to S _N (RB weight) was determined, and outputs the control amount to the control unit 5 .. That is, the control amount deriving units 31 _{1 to} 31 _{N acquire the state values related to the corresponding slices S 1 to SN} from the mobile communication system via the control device 5, and apply the state to the common deep reinforcement learning learning model. Enter a value, calculate the action value using the learning model, and select the action that maximizes the action value. The control amount derivation unit ₃₁ 1 to 31 _N is obtained through the control unit 5 a reward Slice _S 1 to S _N to the selected action. Further, the control amount derivation units 31 _{1 to} 31 _N calculate the control amount (RB amount) based on the selected action and output it to the control device 5. In addition, the control amount derivation unit 31 _{1 to} 31 _N stores the combination of the state value and the action and the reward value for the action as learning data in the learning data storage unit 33 for each control amount calculation step. Store.

Specifically, the control amount derivation unit ₃₁ 1 to 31 _N, as a state value, the slice _S 1 and the throughput value and the delay amount shows a to S _N requirements, NSRS indicating the user satisfaction Slice requirements (NS requirement satisfaction ) value and a slice _S 1 ~S and RBUR (RB usage ratio) value indicating a usage rate of _N, slice _S 1 and the actual allocation RB number for ~S _N, the control target slice _S 1 at the base station BS of ~S _The number of arrival packets, the number of transmission packets, and the number of packets in the buffer (waiting for transmission) on N are acquired. Among the above state values, throughput values, delay, and NSRS value is a value that indicates the state regarding requirements slice _S 1 ~S _N, RBUR value, and RB numbers, usage slice _S 1 to S _N The number of incoming packets, the number of transmitted packets, and the number of packets in the buffer are state values that are supplementarily acquired in order to eliminate the ambiguity of these state values.

によって定義される値である。ここで、Suはユーザ毎のスライス要件の満足度の有無を“０”または“１”で示し、uesはスライスに収容されるユーザ数を示す。このＮＳＲＳ値はユーザの平均的な満足度を示し、１に近いほどスライスの要件が満たされていることを表す。 The NSRS value among the above state values is generated by collecting and aggregating the data from the user terminal UE by the base station BS or the like, and the following equation (1);

The value defined by. Here, Su indicates whether or not the satisfaction level of the slice requirement for each user is satisfied by "0" or "1", and ues indicates the number of users accommodated in the slice. This NSRS value indicates the average satisfaction level of the user, and the closer it is to 1, the more the slice requirement is satisfied.

によって定義される値である。ここで、ＵＲＢは実際に使用したＲＢ数を示し、ＭＲＢは、実際に該当するスライスに割り当てたＲＢ数を示す。ＲＢＵＲ値は、該当するスライスの使用率を示し、１に近いほど過剰なＲＢの割り当てが少ないことを意味する。 Further, the RBUR value among the above state values is generated by the base station BS or the like, and the following equation (2);

The value defined by. Here, URB indicates the number of RBs actually used, and MRB indicates the number of RBs actually assigned to the corresponding slice. The RBUR value indicates the utilization rate of the corresponding slice, and the closer it is to 1, the smaller the excess RB allocation.

The control quantity derivation units 31 _{1 to} 31 _N use the Ape-X method in which distributed learning is applied to DQN as a learning model for inputting state values. That is, each of the control amount deriving units 31 _{1 to} 31 _N applies the state values related to the controlled slices S _{1 to SN} to the learning model to determine the action, obtains the reward value for the action, and obtains the state value. , Behavior, and reward values are collected and accumulated as experience (learning data). On the other hand, the training unit 32, which will be described later, optimizes the parameters of the learning model by training (learning) using this learning data.

FIG. 4 shows an example of the network structure of the learning model used by the control quantity derivation units 31 _{1 to} 31 _N. As shown in FIG. 4, the learning model is an input layer N _IN the state value is input, the total binding layer N ₁ state value input from the input layer N _IN is propagated _{sequentially,} batch normalization layer N ₂ , total binding layer _{N 3,} batch normalization layer _{N 4,} and the total binding layer _{N 5} and batch normalization layer _{N 6,,} it is coupled to branch from the batch normalization layer _{N 6,} total binding layer _{N 7,} batch Normalized layer N ₈ , fully bonded layer N ₉ , fully bonded layer N ₁₀ , batch normalized layer N ₁₁ , fully bonded layer N _12, and output layer N _OUT bonded to fully bonded layers N ₉ , N _12. including. In such a learning model, by inputting a state value to the _{input layer N IN} , the action value, which is a reward value expected as a result of the action _{, is output from the output layer N OUT.}

によって算出する。上記式（３）中の

は床関数である。さらに、制御量導出部３１_１〜３１_Ｎは、時間tの割り当てＲＢ量（ＡＲＢ）を、１ステップ前の時刻t-1における割り当てＲＢ量に対して相対量（ＩＤＲＢ）を加算することにより、下記式（４）；
ARB_t＝ARB_t-1＋IDRB_t …（４）
によって計算し、計算した割り当てＲＢ量を制御装置５に出力する。 Further, the control amount derivation unit 31 _{1 to} 31 _N selects the action having the maximum action value based on the action value obtained as a result of inputting the state value into the learning model (greedy method). Specifically, the control amount derivation unit 31 ₁ to 31 _N, using a learning model shown in FIG. 4, the state value s _t and the action a action value for _t Q at time _{t (S t, a t,} θ) and Acquire (θ is a parameter such as the weighting of the neural network), and determine the action a (a is an integer of 1 or more) that maximizes the _{action value Q (S t} , a _{t, θ).} Then, the control amount derivation unit 31 _{1 to} 31 _N sets the relative amount (IDRB) of RB at the time t assigned to the corresponding slice based on the determined action a by the following equation (3);

Calculated by. In the above formula (3)

Is the floor function. Further, the control quantity derivation units 31 _{1 to} 31 _N add the allocated RB amount (ARB) at the time t to the allocated RB amount at the time t-1 one step before by adding the relative quantity (IDRB). The following formula (4);
ARB _t = ARB _t-1 + IDRB _t … (4)
And outputs the calculated allocated RB amount to the control device 5.

The training unit 32 updates the parameter θ of the learning model shared by the _{control quantity derivation units 31 1 to} 31 _N to the optimum value using the learning data stored in the learning data storage unit 33 (training). That is, the training unit 32 acquires the reward value at time t + 1 after one step and the state value at time t + 1 included in one set of learning data, and based on those values, at time t. Calculate the _{target value y t} to be output by the training model. This reward value is a value in which the NSRS value and the RBUR value are added, and is, for example, a value obtained by multiplying the NSRS value and the RBUR value. The training unit 32, action value Q which is determined based on the state value s _t at time _{t (S t, a t,} θ) updates the parameters theta so as to approach the target value y _t.

Here, the training unit 32 randomly extracts the learning data used for training based on the priority from the learning data stored in the learning data storage unit 33. Further, the training unit 32 updates the training data stored in the learning data storage unit 33 so that the priority of the learning data stored in the training data storage unit 33 is lowered as the training is executed. Further, the training unit 32 updates the parameter θ in each of the control quantity derivation units 31 _{1 to} 31 _N by periodically duplicating the parameter θ updated by the training to the control quantity derivation units 31 _{1 to} 31 _N. do. Here, the respective control quantity derivation units 31 _{1 to} 31 _N may acquire the parameter θ updated by the training unit 32 and update the internal parameter θ.

Next, the function of the control device 5 will be described.

The control device 5 has a function of periodically acquiring a state value and a reward value from the mobile communication system and transferring them to the control amount calculation device 3, and a plurality of control amount derivation units 31 ₁ , 31 _{2 of the control amount calculation device 3.} , ... allocation RB quantity at time t output from 31 _N to (ARB) based, each slice _{S 1, S 2, ...,} S N, to determine RB to allocate to time t _{S B} a (MRB) Has a function to control. That is, the control unit 5, the slice _S slices _S 1 except _B, S 2, ..., for _{S N,} a value obtained by multiplying the MRB and NSRS value at time t-1 of the immediately preceding a priority, the priority in ascending order of the sequence, the slice _{S 1,} S 2, ..., performs a control process for determining the MRB time t for _{S N} to a value equal to ARB. At this time, if the ARB is larger than the remaining RB number, the remaining RB number is determined as the MRB. With such priority control, allocation is prioritized for slices with a smaller RB occupancy so that slices requiring a large amount of RB do not occupy the RB. Further, the control unit 5, for the slice _{S B,} slice _{S 1,} S 2, ..., to determine the remaining RB amount as MRB determined by the sum of MRB assigned to _{S N.}

FIG. 5 is a diagram for explaining an overall image of RB allocation to slices by the control amount calculation device 3 and the control device 5, and FIG. 6 is a diagram for RB allocation to slices by the control amount calculation device 3 and the control device 5. by is a graph showing a temporal change of the RB amount allocated to the slice _{S 1, S 2, S N} . Thus, the training unit 32 of the control amount calculating unit 3, the learning data storage unit 33 to the control amount derivation unit 31 _1, 31 2, based on the stored learned data as experience _... 31 _N, learning model Be learned. Then, learning the control amount derivation unit ₃₁ of the parameters of the optimized training model control amount calculating device 3 by, ₃₁ 2, ... 31 _N to be replicated, the control amount derivation unit ₃₁ 1, 31 _2, ... In 31 _N , by inputting a state value into a common learning model, _{actions targeting each slice S 1} , S ₂ , ..., _SN are selected. In the control unit 5, each slice _{S 1,} S 2, ..., based on the action selected by the target _{S N,} each slice _{S 1, S 2, ...,} S N, is RB to allocate to _{S B} determined is, their RB amount of resources each slice _{S 1, S 2, ...,} S N, is assigned to the _{S B.} According to the example shown in FIG. 6, with the passage of time, it slices the _{S 1} as RB amount "0", "3", "0" is assigned in the order, "3 as RB amount in slice _{S 2} "," 2 "," 6 "are assigned in order, and" 2 "," 1 ", and" 0 "are _{assigned to the slice SN in order as the RB amount.}

Next, the procedure of the control amount calculation method executed by the control system 1 described above will be described. FIG. 7 is a flowchart showing the procedure of the RB allocation process in the control amount calculation method, and FIG. 8 is a flowchart showing the procedure of the learning process in the control amount calculation method. The RB allocation process shown in FIG. 7 is repeatedly executed at times t, t + 1, ... Of a plurality of steps, and the learning process shown in FIG. 7 is a periodic timing or the number of experiences accumulated by the RB allocation process. Is repeatedly executed when a certain number is reached.

First, referring to FIG. 7, when the RB assignment process is started, the control amount derivation unit 31 ₁ of the control amount calculating unit 3, the state values for the slice S ₁ is acquired (step S101). Next, the control amount derivation unit 31 ₁ calculates the action value by inputting the acquired state value into the learning model, and selects the action having the maximum action value (step S102). Thereafter, the control amount derivation unit 31 _1, based on the selected action, allocation RB quantity (ARB) is determined relative to the slice _{S 1,} the RB amount is outputted to the control unit 5 (step S103). Furthermore, the control amount derivation unit 31 _1, the acquired compensation value Slice S ₁ for the previous RB assignment process, the state value, action, (step combination of reward values are stored in the learning data storage unit 33 as the learning data S104). A series of processes in steps S101~S104 by the control amount derivation unit 31 ₂ to the control amount derivation unit 31 _N, repeated targeting remaining slice _S 2 to S _N (step S105).

Next, the control unit 5, RB amount at time t-1 of the immediately preceding (MRB) and NSRS value priority Slice _S 1 to S _N based on is determined (step S106). Thereafter, the control unit 5, in order of the determined priority, RB amount Slice _S 1 ~S _N (MRB) is determined allocation RB quantity Slice _S 1 to S _N to (ARB) based on its RB assignment to slices _S 1 to S _N is controlled in order of priority (step S107). Finally, the control unit 5, control is performed to assign the remaining RB on the slice _{S B} (step S108).

Next, referring to FIG. 8, when the learning process is started, the training unit 32 of the control amount calculation device 3 bases the priority on the learning data stored in the learning data storage unit 33. Learning data used for training is randomly extracted while weighting (step S201). Next, the training unit 32 updates the parameter θ of the learning model of the neural network by training using the extracted training data (step S202). Thereafter, the training unit 32, updated parameter θ is the control amount derivation unit ₃₁ 1, ₃₁ 2, by being replicated in ... 31 _N, each of the control amount derivation unit ₃₁ 1, ₃₁ 2, in the ... 31 _N The training model is updated (step S203). Finally, the training unit 32 updates the priority of the learning data stored in the learning data storage unit 33 so that the learning data of the older experience has a lower priority (step S204).

The operation and effect of the control system 1 of the above-described embodiment will be described.

According to the present embodiment, for each of a plurality of slices S ₁ to S _N, by entering the state value on reinforcement learning model, action to control the resource allocation amount for each slice S ₁ to S _N is determined and It is output, and the combination of the state value and action used at that time and the reward value for the action is stored as learning data. In this case, reinforcement learning model shared by the control of the resource allocation of a plurality of slices S ₁ to S _N are optimized by trained using the training data stored in advance. Reinforced Accordingly, even when the number of slices in the network of the control target is changed dynamically, the control results for individual slices as learning data, for use in the resource allocation control with respect to a plurality of slices S ₁ to S _N The learning model can be optimized. As a result, it is possible to properly control the resource allocation to a plurality of slices S ₁ to S _N that does not depend on the number of slices.

Here, the state values for the slice _S 1 to S _N are included NSRS value and RBUR value. In this case, when controlling the resource allocation for a plurality of slices S ₁ to S _N, it is possible to control to meet the requirements of each slice S ₁ to S _N, thereby improving resource use efficiency RB be able to.

Further, reward Slice _S 1 to S _N is a value obtained by adding the NSRS value and RBUR value. With this case, it can be optimized as in optimizing the learning model for controlling resource allocation for a plurality of slices S ₁ to S _N, meet the requirements of each slice S ₁ to S _N , It can be optimized to improve the utilization efficiency of the resource RB.

In addition, the function of the control device 5, a plurality of control amount derivation unit ₃₁ 1, ... 31 RB amount output by _N based on (ARB), allocation RB quantity for a plurality of slices _S 1 ~S _N (MRB) Includes the ability to control. These features, the resource allocation amount for a plurality of slices S ₁ to S _N, may be determined more control amount derivation unit 31 _1, the control amount outputted by ... 31 _N to the group. As a result, for example, priority control based on a predetermined determination standard can be performed.

Specifically, in the present embodiment, the priority of resource allocation for a plurality of slices S ₁ to S _N by the controller 5 is determined immediately before the allocation RB quantity and the NSRS value (MRB) to a group. In this case, since the slice that require more RB so as not occupy RB, it is possible to smooth the resource allocation for a plurality of slices S ₁ to S _N.

Here, the result of evaluating the performance of the RB allocation control by the control system 1 according to the present embodiment by simulation calculation is shown in comparison with the comparative example. FIG. 9 is a graph showing the average NSRS value in comparison with the comparative example, and FIG. 10 is a graph showing the average RBUR value in comparison with the comparative example. Comparative Example 1 is an example in which all RBs are shared between slices without executing network slicing, and Comparative Example 2 is an example in which RBs are controlled to be equally divided among slices. Example 3 is an example in which the number of packets arriving at the base station BS is weighted and the RB is controlled to be divided, and Comparative Example 4 is an existing method using deep reinforcement learning (“R. Li et). This is an example of control by al., “Deep Reinforcement Learning for Resource Management in Network Slicing,” arXiv: 1805.06591 [cs], May 2018. ”). Based on these evaluation results, according to the present embodiment, it is possible to realize highly satisfying control of various service requirements for all users, and reduce excessive RB allocation in various services to reduce RB. It was found that the usage rate of the product can be maintained high.

The present invention is not limited to the above-described embodiments. The configuration of the above embodiment can be changed in various ways.

1 ... Control system, 3 ... Control amount calculation device, 5 ... Control device (control unit), 31 _{1 to} 31 _N ... Control amount derivation unit, 32 ... Training unit, 33 ... Learning data storage unit, RB ... Resource, S ₁ ~S _N, S B _... slice.

Claims (7)

A control amount calculation device that calculates a control amount for controlling the allocation amount of communication resources for a plurality of slices of a virtualized network on a communication network.
It is the control amount for the slice by being provided corresponding to each of the plurality of slices, acquiring the state value related to the slice and the reward value related to the slice, and inputting the state value into the reinforcement learning model. Multiple control amount derivators that determine and output actions,
A learning data storage unit that stores learning data that is a combination of the state value and the reward value acquired by the plurality of control amount derivation units and the action determined in response to the state value.
A training unit that optimizes the reinforcement learning model shared by the plurality of control quantity derivation units by training using the learning data stored in the training data storage unit.
A control amount calculation device including. The state value for the slice includes at least satisfaction with the slice requirements and utilization of communication resources allocated to the slice.
The control amount calculation device according to claim 1. The reward value for the slice is a value that takes into account the satisfaction level regarding the slice requirement and the usage rate of the communication resource allocated to the slice.
The control amount calculation device according to claim 1 or 2. A control unit that controls the allocation amount for the plurality of slices is further provided based on the control amount output by the plurality of control amount derivation units.
The control amount calculation device according to any one of claims 1 to 3. The control unit determines the priority for the plurality of slices based on the amount of the resource allocated immediately before the plurality of slices and the satisfaction level regarding the requirements of the plurality of slices, and the control unit determines the priority for the plurality of slices in accordance with the order indicated by the priority. Performing the control over multiple slices,
The control amount calculation device according to claim 4. The control amount for the slice relates to the number of resource blocks in the radio access network.
The control amount calculation device according to any one of claims 1 to 5. A control amount calculation method executed by a control amount calculation device that calculates a control amount for controlling the allocation amount of communication resources for a plurality of slices of a virtualized network on a communication network.
It is the control amount for the slice by being executed corresponding to each of the plurality of slices, acquiring the state value related to the slice and the reward value related to the slice, and inputting the state value into the reinforcement learning model. Multiple control amount derivation steps that determine and output actions,
A learning data storage step that stores learning data that is a combination of the state value and the reward value acquired in the plurality of control amount derivation steps and the action determined in response to the state value.
A training step of optimizing the reinforcement learning model shared by the plurality of control quantity derivation steps by training using the learning data stored by the learning data storage step.
Control amount calculation method including.

Images