WO2021064768A1 - 制御装置、制御方法及びシステム - Google Patents

制御装置、制御方法及びシステム Download PDF

Info

Publication number
WO2021064768A1
WO2021064768A1 PCT/JP2019/038456 JP2019038456W WO2021064768A1 WO 2021064768 A1 WO2021064768 A1 WO 2021064768A1 JP 2019038456 W JP2019038456 W JP 2019038456W WO 2021064768 A1 WO2021064768 A1 WO 2021064768A1
Authority
WO
WIPO (PCT)
Prior art keywords
control
network
state
learning model
learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/038456
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
亜南 沢辺
孝法 岩井
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to PCT/JP2019/038456 priority Critical patent/WO2021064768A1/ja
Priority to US17/641,183 priority patent/US20220345377A1/en
Priority to JP2021550733A priority patent/JP7251647B2/ja
Publication of WO2021064768A1 publication Critical patent/WO2021064768A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/16—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/14—Network analysis or design
    • H04L41/147—Network analysis or design for predicting network behaviour
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0823—Errors, e.g. transmission errors
    • H04L43/0829—Packet loss
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0852—Delays
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0852—Delays
    • H04L43/087—Jitter
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0876—Network utilisation, e.g. volume of load or congestion level
    • H04L43/0882—Utilisation of link capacity

Definitions

  • the present invention relates to a control device, a control method and a system.
  • Moving image data is distributed from a server on a network, the moving image data is played back on a terminal, and a robot or the like installed in a factory or the like is remotely controlled from the server.
  • Patent Document 1 describes that a wireless communication device capable of allocating one call channel most suitable for wireless communication from a plurality of call channels and supplying good call quality is provided.
  • Patent Document 2 describes that a congestion control device and a congestion control method capable of predicting the behavior of the average buffer length at an early stage and reducing the packet discard rate are provided. ..
  • Patent Document 3 describes that an appropriate communication parameter is selected according to the surrounding conditions of the wireless communication device.
  • Patent Document 4 describes that a facsimile communication device capable of autonomously adjusting communication parameters to prevent the occurrence of communication errors is provided.
  • machine learning In recent years, due to the usefulness of machine learning, the application of machine learning to various fields has been studied. For example, it is being considered to apply machine learning to games such as chess and control of robots and the like.
  • maximization of the score in the game is set as a reward, and the performance of machine learning is evaluated.
  • the realization of the target motion is set as a reward, and the performance of machine learning is evaluated.
  • learning performance is discussed by the sum of immediate reward and episode-based reward.
  • Patent Document 5 describes that an information processing device, an information processing system, an information processing program, and an information processing method that can easily reproduce the delay characteristics of a network are provided.
  • the information processing device disclosed in Patent Document 5 includes a learning processing unit that learns a plurality of parameters of a learning model for predicting a delay time in a network from the amount of data of the traffic and the delay time for each unit time. ..
  • Japanese Unexamined Patent Publication No. 2003-179970 Japanese Unexamined Patent Publication No. 2011-0616999 Japanese Unexamined Patent Publication No. 2013-051520 Japanese Unexamined Patent Publication No. 2019-022055 Japanese Unexamined Patent Publication No. 2019-008554
  • machine learning is incorporated as a part of network control.
  • machine learning is merely used to reproduce the delay characteristics of the network, and the controller selects control parameters according to the state of the network to optimize the state of the network. Has not been realized.
  • a main object of the present invention is to provide a control device, a control method, and a system that contribute to realizing efficient network control using machine learning.
  • control parameters are set to the device included in the network based on the learning unit that learns the behavior for controlling the network and the behavior obtained from the learning model generated by the learning unit.
  • the control unit includes a control unit that controls the network by setting the above, and the control unit determines the control parameter based on the influence of the action obtained from the learning model on the state of the network.
  • Equipment is provided.
  • the device included in the network has control parameters based on the step of learning the action for controlling the network and the action obtained from the learning model generated by the learning step.
  • the control method includes a step of controlling the network by setting the above, and the control step determines the control parameter based on the influence of the behavior obtained from the learning model on the state of the network. Is provided.
  • control parameters are applied to the device included in the network based on the learning means for learning the behavior for controlling the network and the behavior obtained from the learning model generated by the learning means.
  • a control device According to each viewpoint of the present invention, a control device, a control method, and a system that contribute to realizing efficient network control using machine learning are provided.
  • other effects may be produced in place of or in combination with the effect.
  • the control device 100 includes a learning unit 101 and a control unit 102 (see FIG. 1).
  • the learning unit 101 learns an action for controlling the network (step S01 in FIG. 2).
  • the control unit 102 controls the network by setting control parameters in the devices included in the network based on the behavior obtained from the learning model generated by the learning unit 101 (step S02 in FIG. 2). At that time, the control unit 102 determines the control parameters based on the influence of the behavior obtained from the learning model on the state of the network.
  • the control device 100 When controlling the network, the control device 100 does not adopt the behavior obtained from the learning model as it is, but determines the behavior (control parameter) based on the influence of the behavior on the state of the network. That is, the control device 100 does not adopt an action that has little influence on the network even if the action is obtained from the learning model. In other words, the control device 100 positively adopts an action that is expected to be highly effective in controlling the network and controls the network. As a result, unnecessary actions for network control are suppressed, actions useful for network control are promoted, and efficient network control using machine learning is realized.
  • FIG. 3 is a diagram showing an example of a schematic configuration of the communication network system according to the first embodiment.
  • the communication network system includes a terminal 10, a control device 20, and a server 30.
  • the terminal 10 is a device having a communication function.
  • Examples of the terminal 10 include a WEB camera, a surveillance camera, a drone, a smartphone, a robot, and the like.
  • the purpose is not to limit the terminal 10 to the above-mentioned WEB camera or the like.
  • the terminal 10 can be any device having a communication function.
  • the terminal 10 communicates with the server 30 via the control device 20.
  • Various applications and services are provided by the terminal 10 and the server 30.
  • the server 30 analyzes the image data from the WEB camera and manages the materials of the factory or the like.
  • the terminal 10 is a drone
  • a control command is transmitted from the server 30 to the drone, and the drone transports luggage and the like.
  • the terminal 10 is a smartphone
  • the video is distributed from the server 30 to the smartphone, and the user watches the video using the smartphone.
  • the control device 20 is, for example, a communication device such as a proxy server or a gateway, and is a device that controls a network including a terminal 10 and a server 30.
  • the control device 20 controls the network by changing the values of the TCP (Transmission Control Protocol) parameter group and the buffer control parameter group.
  • TCP Transmission Control Protocol
  • buffer control For example, as a control of TCP parameters, changing the flow window size is exemplified.
  • buffer control include changing parameters related to the minimum guaranteed bandwidth, RED (Random Early Detection) loss rate, loss start queue length, and buffer length in queue management of a plurality of buffers.
  • control parameters parameters that affect communication (traffic) between the terminal 10 and the server 30, such as the above TCP parameters and parameters related to buffer control, are referred to as "control parameters”.
  • the control device 20 controls the network by changing the control parameters.
  • the network control by the control device 20 may be performed at the time of packet transfer of the own device (control device 20), or may be performed by instructing the terminal 10 or the server 30 to change the control parameters.
  • control device 20 controls the network by changing the flow window size of the TCP session formed with the terminal 10.
  • the control device 20 may control the network by changing the size of a buffer for storing packets received from the server 30 or changing the cycle of reading packets from the buffer.
  • the control device 20 uses "machine learning” to control the network. More specifically, the control device 20 controls the network based on the learning model obtained by reinforcement learning.
  • control device 20 may control the network based on learning information (Q table) obtained as a result of reinforcement learning called Q-learning.
  • the "agent” is trained so as to maximize the “value” in the given "environment”.
  • the network including the terminal 10 and the server 30 is the "environment”
  • the control device 20 is trained so as to optimize the state of the network.
  • the state s indicates what kind of state the environment (network) is in.
  • traffic for example, throughput, average packet arrival interval, etc.
  • Action a indicates an action that the agent (control device 20) can take with respect to the environment (network). For example, in the case of a communication network system, changing the setting of the TCP parameter group, turning on / off the function, and the like are exemplified as the action a.
  • the reward r indicates how much evaluation can be obtained as a result of the agent (control device 20) executing the action a in a certain state s.
  • the control device 20 is defined as a positive reward if the throughput increases as a result of changing a part of the TCP parameter group, and a negative reward if the throughput decreases.
  • Q-learning learning proceeds so as to maximize the value in the future, instead of maximizing the reward (immediate reward) obtained at the present time (Q-table is constructed).
  • the learning of the agent in Q learning is performed so as to maximize the value (Q value, state action value) when the action a in a certain state s is adopted.
  • the Q value (state action value) is expressed as Q (s, a).
  • Q-learning it is premised that the action of the agent to transition to a high-value state by the action has the same value as the transition destination. Based on such a premise, the Q value at the present time t can be expressed by the Q value at the next time point t + 1 (see equation (1)).
  • Es t + 1 is the expected value relating to the state S t + 1
  • Ea t + 1 denotes the expected value behavioral a t + 1.
  • ⁇ is the discount rate.
  • the Q value is updated according to the result of adopting the action a in a certain state s. Specifically, the Q value is updated according to the following equation (2).
  • ⁇ is a parameter called the learning rate and controls the update of the Q value.
  • "max" in the equation (2) is a function that outputs the maximum value of the possible actions a in the state St + 1.
  • a method for the agent (control device 20) to select the action a a method called ⁇ -greedy can be adopted.
  • an action is randomly selected with a probability of ⁇ , and the most valuable action is selected with a probability of 1- ⁇ .
  • a Q-table as shown in FIG. 4 is generated.
  • the control device 20 may control the network based on a learning model obtained as a result of reinforcement learning using deep learning called DQN (Deep Q Network).
  • DQN Deep Q Network
  • the action value function is expressed by the Q table, but in DQN, the action value function is expressed by deep learning.
  • the optimal action value function is calculated by an approximate function using a neural network.
  • the optimal action value function is a function that outputs the value of performing a certain action a in a certain state s.
  • the neural network includes an input layer, an intermediate layer (hidden layer), and an output layer.
  • the input layer inputs the state s. There is a corresponding weight in the link of each node in the middle layer.
  • the output layer outputs the value of action a.
  • the nodes of the input layer correspond to the network states S1 to S3.
  • the state of the network input to the input layer is weighted by the intermediate layer and output to the output layer.
  • the nodes of the output layer correspond to the actions A1 to A3 that the control device 20 can take.
  • Node of the output layer outputs value of action value function Q (s t, a t) corresponding to each of the actions A1 ⁇ A3.
  • connection parameters weights between the nodes that output the above action value function are learned.
  • the error function shown in the following equation (3) is set and learning is performed by backpropagation.
  • the operation mode of the control device 20 includes two operation modes.
  • the first operation mode is a learning mode for calculating a learning model.
  • a Q table as shown in FIG. 4 is calculated.
  • the control device 20 executes reinforcement learning by "DQN”
  • the weight as shown in FIG. 6 is calculated.
  • the second operation mode is a control mode in which the network is controlled using the learning model calculated in the learning mode. Specifically, the control device 20 in the control mode calculates the current network state s and selects the most valuable action a among the actions a that can be taken in the case of the state s. The control device 20 executes an operation (network control) corresponding to the selected action a.
  • the control device 20 calculates a learning model for each network congestion state. For example, when the congestion state of the network is divided into three stages, three learning models corresponding to each congestion state are calculated. In the following description, the network congestion state will be referred to as "congestion level".
  • the control device 20 calculates a learning model (learning information such as a Q table and weights) corresponding to each congestion level in the learning mode.
  • the control device 20 selects a learning model corresponding to the current congestion level from a plurality of learning models (learning models for each congestion level) and controls the network.
  • FIG. 7 is a diagram showing an example of a processing configuration (processing module) of the control device 20 according to the first embodiment.
  • the control device 20 includes a packet transfer unit 201, a feature amount calculation unit 202, a congestion level calculation unit 203, a network control unit 204, a reinforcement learning execution unit 205, and a storage unit 206. Consists of including.
  • the packet transfer unit 201 is a means for receiving a packet transmitted from the terminal 10 or the server 30 and transferring the received packet to the opposite device.
  • the packet transfer unit 201 performs packet transfer according to the control parameters notified from the network control unit 204.
  • the packet transfer unit 201 performs packet transfer with the notified flow window size.
  • the packet transfer unit 201 delivers a copy of the received packet to the feature amount calculation unit 202.
  • the feature amount calculation unit 202 is a means for calculating the feature amount that characterizes the communication traffic between the terminal 10 and the server 30.
  • the feature amount calculation unit 202 extracts a traffic flow that is a target of network control from the acquired packet.
  • the traffic flow that is the target of network control is a group consisting of packets having the same source IP (Internet Protocol) address, destination IP address, port number, and the like.
  • the feature amount calculation unit 202 calculates the feature amount from the extracted traffic flow. For example, the feature amount calculation unit 202 calculates throughput, average packet arrival interval, packet loss rate, jitter, and the like as feature amounts. The feature amount calculation unit 202 stores the calculated feature amount in the storage unit 206 together with the calculation time. Since existing techniques can be used for calculation of throughput and the like and are obvious to those skilled in the art, detailed description thereof will be omitted.
  • the congestion level calculation unit 203 calculates the congestion level indicating the degree of network congestion based on the feature amount calculated by the feature amount calculation unit 202. For example, the congestion level calculation unit 203 may calculate the congestion level according to the range including the feature amount (for example, throughput). For example, the congestion level calculation unit 203 may calculate the congestion level based on the table information as shown in FIG.
  • the congestion level is calculated as “2”.
  • the congestion level calculation unit 203 may calculate the congestion level based on a plurality of features. For example, the congestion level calculation unit 203 may calculate the congestion level using the throughput and the packet loss rate. In this case, the congestion level calculation unit 203 calculates the congestion level based on the table information as shown in FIG. For example, in the example of FIG. 9, when the throughput T is included in the range of “TH11 ⁇ T ⁇ TH12” and the packet loss rate is included in the range of “TH21 ⁇ L ⁇ TH22”, the congestion level is “. 2 ”is calculated.
  • the congestion level calculation unit 203 delivers the calculated congestion level to the network control unit 204 and the reinforcement learning execution unit 205.
  • the reinforcement learning execution unit 205 is a means for learning actions (control parameters) for controlling the network.
  • the reinforcement learning execution unit 205 executes the Q-learning and the reinforcement learning by DQN described above to generate a learning model.
  • the reinforcement learning execution unit 205 is a module that mainly operates in the learning mode.
  • the reinforcement learning execution unit 205 calculates the network state s at the current time t from the feature amount stored in the storage unit 206.
  • the reinforcement learning execution unit 205 selects the action a from the possible actions a in the calculated state s by a method such as the above-mentioned ⁇ -greedy method.
  • the reinforcement learning execution unit 205 notifies the packet transfer unit 201 of the control content (setting value of the control parameter) corresponding to the selected action.
  • the reinforcement learning execution unit 205 determines the reward according to the change of the network according to the above behavior.
  • the reinforcement learning execution unit 205 sets a positive value in the reward rt + 1 described in the equations (2) and (3) when the throughput increases as a result of taking the action a.
  • the reinforcement learning execution unit 205 sets a negative value in the reward rt + 1 described in the equations (2) and (3) when the throughput decreases as a result of taking the action a.
  • the reinforcement learning execution unit 205 generates a learning model for each congestion level.
  • FIG. 10 is a diagram showing an example of the internal configuration of the reinforcement learning execution unit 205.
  • the reinforcement learning execution unit 205 includes a learning device management unit 211 and a plurality of learning devices 212-1 to 212-N (N is a positive integer, the same applies hereinafter).
  • the learning device management unit 211 is a means for managing the operation of the learning device 212.
  • Each of the plurality of learners 212 learns actions for controlling the network.
  • the learner 212 is prepared for each congestion level. In FIG. 10, the corresponding congestion levels are shown in parentheses.
  • the learning device 212 calculates a learning model (Q table, weight applied to the neural network) for each congestion level and stores it in the storage unit 206.
  • the learner management unit 211 selects the learner 212 corresponding to the congestion level notified from the congestion level calculation unit 203.
  • the learning device management unit 211 instructs the selected learning device 212 to start learning.
  • the learning device 212 that receives the instruction executes the Q-learning and the reinforcement learning by DQN described above.
  • the network control unit 204 is a means for controlling the network based on the behavior obtained from the learning model generated by the reinforcement learning execution unit 205.
  • the network control unit 204 determines the control parameters to be notified to the packet transfer unit 201 based on the learning model obtained as a result of reinforcement learning. At that time, the network control unit 204 selects one learning model from the plurality of learning models, and controls the network based on the behavior obtained from the selected learning model.
  • the network control unit 204 is a module that mainly operates in the control mode.
  • the network control unit 204 selects a learning model (Q table, weight) according to the congestion level notified from the congestion level calculation unit 203. Next, the network control unit 204 reads the latest (current time) feature amount from the storage unit 206.
  • a learning model Q table, weight
  • the network control unit 204 estimates (calculates) the state of the network to be controlled from the read feature amount. For example, the network control unit 204 refers to a table (see FIG. 11) in which the feature amount F and the network state are associated with each other, and calculates the network state corresponding to the current feature amount F.
  • the network state can be regarded as the "traffic state". That is, in the disclosure of the present application, the "traffic state” and the “network state” can be interchanged with each other.
  • FIG. 11 shows a case where the network state is calculated from the feature amount F regardless of the congestion level
  • the feature amount and the network state may be associated with each congestion level.
  • the network control unit 204 refers to the Q-table selected according to the congestion level, and the value Q of each action corresponding to the current network state. Get the highest behavior. For example, in the example of FIG. 4, the calculated traffic state is "state S1", and the value Q (S1, A3) of the values Q (S1, A1), Q (S1, A2), and Q (S1, A3). If A1) is the maximum, the action A1 is read out.
  • the network control unit 204 applies the weight selected according to the congestion level to the neural network as shown in FIG.
  • the network control unit 204 inputs the current network state into the neural network and acquires the most valuable action among the possible actions.
  • fluctuation values of control parameters are mainly learned as actions that the control device 20 can take.
  • the network control unit 204 executes the action acquired from the learning model and controls the network.
  • the network control unit 204 determines the control parameters to be set in the network based on the fluctuation values of the control parameters obtained from the learning model. More specifically, as shown in the following equation (4), the network control unit 204 multiplies the current control parameter P t by the fluctuation amount ⁇ M of the control parameter obtained from the learning model by the weight ⁇ . , Update the control parameter P t + 1 set in the network.
  • the network control unit 204 generates control log information as shown in FIG. 12 and stores it in the storage unit 206.
  • throughput is selected as a feature amount indicating the state of the network.
  • the flow window size is selected as the control parameter.
  • the first line of the control log corresponding to the congestion level 1 in FIG. 12 shows that the traffic increased by B11 Mbps as a result of increasing the flow window size by A11 Mbps when the traffic was T11 Mbps.
  • the network control unit 204 may create a control log for each congestion level.
  • the network control unit 204 determines the control parameters to be set in the packet transfer unit 201 based on the behavior acquired from the learning model.
  • the network control unit 204 controls the network by setting control parameters in the network based on the behavior obtained from the learning model generated by the reinforcement learning execution unit 205. At that time, the network control unit 204 determines the control parameters to be set in the network based on the influence of the behavior obtained from the learning model on the state of the network.
  • the network control unit 204 determines the control parameters to be set in the packet transfer unit 201 based on the log information (control log information) generated by the learner 212 corresponding to the current congestion level.
  • the network control unit 204 extracts the log information stored in the storage unit 206 that matches the following log extraction conditions from the log corresponding to the current congestion level.
  • the log extraction condition is that the state described in the log information is substantially equal to the current state, and the amount of change in the state of the network is larger than a predetermined threshold value. Note that the state is substantially the same, the conditions described in the log information S L, the current state if S t, in the case where the relationship of S L + ⁇ 1 ⁇ S t ⁇ S L + ⁇ 2 is satisfied is there. That is, by appropriately selecting ⁇ 1 and ⁇ 2 , a slight difference between the state SL and the state St can be absorbed.
  • the control log information shown in the upper part of FIG. 12 is selected. If the current network status (throughput) is "T11 Mbps", the logs in the first to third lines from the logs shown in the upper part of FIG. 12 are selected. Further, from the logs in the first to third lines, logs in which the network state change amounts B11 to B13 are larger than a predetermined threshold value are extracted. For example, if the amount of change B11 is larger than a predetermined threshold value, the log of the first line is extracted. When the control device 20 includes two or more logs in which the amount of change in the state of the network is larger than a predetermined threshold value, the control device 20 may extract the log in which the amount of change in the state of the network is the largest.
  • the network control unit 204 changes the control parameter corresponding to the behavior of the extracted log and the control parameter corresponding to the behavior acquired from the learning model corresponding to the current congestion level. Judge differences in direction.
  • the network control unit 204 determines that the change direction of the control parameter is "change in the same direction". On the other hand, the network control unit 204 determines that one control parameter indicates an increase and the other control parameter indicates a decrease, or vice versa, the change direction of the control parameter is "change in the reverse direction”.
  • the network control unit 204 determines that the change direction of the control parameter is "change in the same direction".
  • the network control unit 204 determines that the change direction of the control parameters is "change in the opposite direction".
  • the network control unit 204 When the change direction of the control parameter is determined to be "reverse direction", the network control unit 204 does not adopt the action obtained from the learning model. That is, if the change direction of the control parameter is "reverse direction", the network control unit 204 discards the action (control parameter) obtained from the learning model. In this case, the control of the network is maintained, and the control parameters set in the packet transfer unit 201 are not changed.
  • the network control unit 204 sets the fluctuation value ⁇ L of the control parameter extracted from the log and the control parameter corresponding to the action acquired from the learning model.
  • the difference D between the fluctuation value ⁇ M and the fluctuation value ⁇ M is calculated (see the following equation (5)).
  • the network control unit 204 When the difference is equal to or less than a predetermined threshold value, the network control unit 204 notifies the packet transfer unit 201 of the control parameter P t + 1 determined according to the following equation (6).
  • Delta 1 is a weight to be multiplied by the variation value [delta] M of the control parameter obtained from the learning model.
  • ⁇ 1 is a numerical value less than 1 ( ⁇ 1 ⁇ 1).
  • the network control unit 204 When the difference is larger than a predetermined threshold value, the network control unit 204 notifies the packet transfer unit 201 of the control parameter P t + 1 determined according to the following equation (7).
  • ⁇ 2 is a weight multiplied by the fluctuation value ⁇ M of the control parameter obtained from the learning model.
  • ⁇ 2 is a numerical value of 1 or more ( ⁇ 2 ⁇ 1).
  • the network control unit 204 refers to the control log information obtained when the network is controlled when the network is controlled.
  • the control log information includes the state of the network, the fluctuation value of the control parameter when the network is controlled, and the amount of change in the state caused by the control of the network.
  • the network control unit 204 refers to the control log information and calculates how much the action (change of the control unit parameter) obtained from the learning model affects the state change of the network.
  • the network control unit 204 executes a threshold value process (for example, a process of determining whether the acquired value is equal to or less than the threshold value) with respect to the amount of change in the state of the control log, and the control parameters executed in the past Of these, behaviors (changes in control parameters) that have a high impact on the network are extracted.
  • a threshold value process for example, a process of determining whether the acquired value is equal to or less than the threshold value
  • the network control unit 204 determines how close the behavior (variation amount of the control parameter) obtained from the learning model is to the behavior (variation amount of the control parameter) having a high influence on the network, according to the equation (5). Judging. When the fluctuation amount of the control parameter from the learning model and the fluctuation amount of the control parameter having a high influence are almost the same (difference D is smaller than the threshold value), the network control unit 204 uses a weight ⁇ 1 whose value is less than 1. Weight the control parameters from the training model. For example, by selecting a value such as "0.9" as the weight delta 1, the control of the impact of high had network is reproduced.
  • the network control unit 204 has a weight having a value of 1 or more.
  • the delta 2 weighting control parameters from the learning model For example, by selecting a value such as "1.5" as the weight delta 1, can be brought close to the control of the impact of high was network.
  • the network control unit 204 controls the network state to be optimal by weighting the fluctuation values of the control parameters obtained from the learning model based on the past control history (control log information). .. That is, the network control unit 204 has a fluctuation value of the control parameter obtained from the learning model and a fluctuation value of the control parameter corresponding to the state change in the control log information and caused by the control of the network, which is larger than the threshold value. And, the difference between and is calculated. The network control unit 204 extracts actions having a high degree of influence by calculating the difference. Then, the network control unit 204 executes threshold processing on the calculated difference and changes (adjusts) the weight based on the result of the threshold processing to reproduce the behavior having a high influence in the past. doing.
  • the network control unit 204 discards the action obtained from the learning model when the change direction of the control parameter is determined to be "reverse direction". Such an operation of the network control unit 204 excludes (filters) an action opposite to the action for which a large influence (state change higher than the threshold value) is obtained in the past state in the same state as the present. Based on the idea of preference. Based on the same idea, it is preferable to filter behaviors that have a small effect on the change of state (not involved in the distribution of the state).
  • the network control unit 204 refers to the log information for each congestion level, the current state and the past state are substantially the same, and the amount of state change is low (the amount of change is smaller than a predetermined threshold value). Do not adopt actions that are substantially the same as actions.
  • the network control unit 204 extracts a log having substantially the same state as the current state from the control log information for each congestion level. Further, when the corresponding state change amount of the extracted log is low and the action obtained from the learning model is the same as the action described in the log, the network control unit 204 performs the action from the learning model. Is discarded (filtered). That is, when the amount of change in the state caused by network control is smaller than a predetermined threshold value, the network control unit 204 discards the fluctuation value of the control parameter obtained from the learning model using the corresponding network state. ..
  • control device 20 The operation of the control device 20 according to the first embodiment in the control mode is summarized in the flowchart shown in FIG.
  • the control device 20 acquires the packet and calculates the feature amount (step S101).
  • the control device 20 calculates the congestion level of the network based on the calculated feature amount (step S102).
  • the control device 20 selects a learning model according to the congestion level (step S103).
  • the control device 20 identifies the state of the network based on the calculated features (step S104).
  • the control device 20 controls the network by the most valuable action according to the state of the network by using the learning model selected in step S103 (step S105). At that time, the control device 20 corrects the fluctuation value of the control parameter acquired from the learning model based on the control result (control log) in the past.
  • control device 20 The operation of the control device 20 according to the first embodiment in the learning mode is summarized in the flowchart shown in FIG.
  • the control device 20 acquires the packet and calculates the feature amount (step S201).
  • the control device 20 calculates the congestion level of the network based on the calculated feature amount (step S202).
  • the control device 20 selects the learning device 212 to be learned according to the congestion level (step S203).
  • the control device 20 starts learning of the selected learner 212 (step S204). More specifically, the selected learner 212 sets the packet group (packet group including the packet observed in the past) observed while the condition (congestion level) for which the learner 212 is selected is satisfied. Learn using.
  • the control device 20 corrects the fluctuation value (increase / decrease value) of the control parameter output by the learning model according to the past control log.
  • the control device 20 determines the control parameters based on the influence of the behavior obtained from the learning model on the state of the network.
  • the network targeted by the control device 20 is often controlled by a plurality of and different parameters (control of QoS and the like), and it is necessary to determine which parameter is effective in controlling the network. Therefore, the control device 20 determines the update value of the control parameter according to the strength of the influence of the action (change of the control parameter) in each state of the network on the network from the past control record (control log information) of the network.
  • the state of the network among a plurality of and different parameters will transition (converge) to the intended state (intended QoS) at an early stage.
  • control device 20 handles a large number of flows (traffic flows; packet groups having the same destination and the like), but if the network congestion levels are the same, the same learning model is selected.
  • the behavior applied to each flow is often the same, and even if the update of the control parameter for one flow is small, if the update of the same control parameter is repeated for many flows, resources such as memory Consume a lot. That is, when a plurality of learning models are prepared as disclosed in the present application, the change of the control parameter may have a great influence on the resource.
  • the control device 20 calculates the degree of influence on the reward (change of state to the network) due to the control of the network from the past control information, and does not adopt the control parameter having a small influence on the reward. Further, the control parameter having a large influence on the reward is readjusted by determining the weight for the update value (increase / decrease value) of the control parameter in consideration of the degree of influence.
  • the network control unit 204 sets (updates) the control parameters to be set in the packet transfer unit 201 based on the past network change history (control log information).
  • the update of the control parameter when the control log information does not exist will be described.
  • the network control unit 204 stores the state of the network caused by the action in the storage unit 206.
  • the network control unit 204 stores the control log information as shown in FIG. 16 in the storage unit 206.
  • FIG. 16 shows a change of state of the network when the network control unit 204 performs action A1 (increases the flow window size by A bytes).
  • the network control unit 204 inputs the current network state into the learning model and refers to log information related to the same type of behavior as the obtained behavior. For example, when the current network state is input to the learning model and the action A1 is obtained, the network control unit 204 refers to the log information shown in FIG.
  • the network control unit 204 discards the action obtained from the learning model. In this case, the network control unit 204 does not perform any particular operation. That is, when the action obtained from the learning model is executed, the state of the network is likely to deteriorate, and the network control unit 204 does not adopt such an action.
  • the network control unit 204 executes a threshold value process (for example, a process of determining whether the acquired value is equal to or less than the threshold value) with respect to the state change amount. If the amount of state change is equal to or less than the threshold value as a result of the threshold value processing, the control parameter is determined according to the above-described equation (5). If the amount of state change is larger than the threshold value as a result of the threshold value processing, the control parameter is determined according to the above-described equation (6).
  • a threshold value process for example, a process of determining whether the acquired value is equal to or less than the threshold value
  • the control device 20 when the control device 20 according to the second embodiment has executed the action (update of the control parameter) obtained from the learning model in the past, the reward (network of the network) generated by the update of the control parameter is performed.
  • the control parameter is determined based on the change in the state). That is, as in the first embodiment, the control device 20 determines the weight so as to reproduce the change of the control parameter when the change of the control parameter has a great positive influence on the state of the network. Update control parameters.
  • the weight is determined and the control parameter is updated so as to expand the effect of the change of the control parameter.
  • FIG. 17 is a diagram showing an example of the hardware configuration of the control device 20.
  • the control device 20 can be configured by an information processing device (so-called computer), and includes the configuration illustrated in FIG.
  • the control device 20 includes a processor 311, a memory 312, an input / output interface 313, a communication interface 314, and the like.
  • the components such as the processor 311 are connected by an internal bus or the like so that they can communicate with each other.
  • control device 20 may include hardware (not shown), or may not include an input / output interface 313 if necessary.
  • number of processors 311 and the like included in the control device 20 is not limited to the example of FIG. 17, and for example, a plurality of processors 311 may be included in the control device 20.
  • the processor 311 is a programmable device such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a DSP (Digital Signal Processor). Alternatively, the processor 311 may be a device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The processor 311 executes various programs including an operating system (OS).
  • OS operating system
  • the memory 312 is a RAM (RandomAccessMemory), a ROM (ReadOnlyMemory), an HDD (HardDiskDrive), an SSD (SolidStateDrive), or the like.
  • the memory 312 stores an OS program, an application program, and various data.
  • the input / output interface 313 is an interface of a display device or an input device (not shown).
  • the display device is, for example, a liquid crystal display or the like.
  • the input device is, for example, a device that accepts user operations such as a keyboard and a mouse.
  • the communication interface 314 is a circuit, module, or the like that communicates with another device.
  • the communication interface 314 includes a NIC (Network Interface Card) and the like.
  • the function of the control device 20 is realized by various processing modules.
  • the processing module is realized, for example, by the processor 311 executing a program stored in the memory 312.
  • the program can also be recorded on a computer-readable storage medium.
  • the storage medium may be a non-transient such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. That is, the present invention can also be embodied as a computer program product.
  • the program can be downloaded via a network or updated using a storage medium in which the program is stored.
  • the processing module may be realized by a semiconductor chip.
  • terminal 10 and the server 30 can also be configured by an information processing device like the control device 20, and the basic hardware configuration thereof is not different from that of the control device 20, so the description thereof will be omitted.
  • control device 20 may be separated into a device that controls the network and a device that generates a learning model.
  • the storage unit 206 that stores the learning information (learning model) may be realized by an external database server or the like. That is, the disclosure of the present application may be implemented as a system including learning means, control means, storage means and the like.
  • the weights of the control parameters may be changed according to the network environment. For example, in the case of a network with a large packet loss rate such as a wireless LAN (Local Area Network), the weight of control parameters (for example, transmission rate and transmission power) for suppressing loss is increased. Alternatively, in a network such as PS-LTE (Public Safety Long Term Evolution) or LPWA (Low Power Wide Area) where the bandwidth between one base station and the terminal is narrow, the bandwidth control weight is reduced and the bandwidth control adjustment range (bandwidth control adjustment range). Fluctuation amount) is suppressed. On the other hand, in the case of a fixed network, since there is a margin in the bandwidth, a weight may be set so as to give priority to the bandwidth control.
  • PS-LTE Public Safety Long Term Evolution
  • LPWA Low Power Wide Area
  • the weight of the control parameter may be changed depending on the time zone, the position of the terminal 10, and the like.
  • the weight of the control parameter may be changed in a time zone such as early morning, daytime, evening, and midnight. In this case, since the usage rate (line congestion) of the terminal 10 is higher in the evening than in other time zones, measures such as lowering the weight of the control parameters related to bandwidth control are taken.
  • the weight when determining the control parameter may be changed for each type, service or application of the terminal 10. For example, in the control device 20, since jitter is important in a real-time control system such as a robot or a drone, the weight of a parameter that controls jitter may be increased. Alternatively, since throughput is important in control related to video data such as moving image distribution, the control device 20 may increase the weight of the parameter that controls the throughput. Alternatively, since the packet loss rate is important in the control of the telemetry system such as the measurement control in a remote place, the control device 20 may increase the weight of the parameter for controlling the packet loss.
  • control device 20 may take measures such as increasing the weight of the control parameter changed by the operator. That is, the control device 20 may respect the judgment by the operator so that the control parameter changed by the operator has a great influence on the state of the network.
  • control log information generated by the network control unit 204 is used for modifying the behavior (control parameter) obtained from the learning model.
  • the control log information may be used as a learning log of the learning device 212.
  • control device 20 may control a unit of 10 terminals or a group of a plurality of terminals 10 as a control target. That is, even if the same terminal 10 is used, different applications have different port numbers and the like, and are treated as different flows.
  • the control device 20 may apply the same control (change of control parameters) to packets transmitted from the same terminal 10.
  • the control device 20 may, for example, treat terminals 10 of the same type as one group and apply the same control to packets transmitted from terminals 10 belonging to the same group.
  • the control unit (102, 204) obtains the state of the network when the network is controlled, the fluctuation value of the control parameter when the network is controlled, and the amount of change in the state caused by the control of the network.
  • the control device (20, 100) according to Appendix 2, which weights the fluctuation value of the control parameter obtained from the learning model based on the log information including.
  • the control units (102, 204) The fluctuation value of the control parameter obtained from the learning model and the fluctuation value of the control parameter included in the log information and corresponding to the state change in which the amount of change of the state caused by the control of the network is larger than the first threshold value.
  • Steps to learn actions to control the network A step of controlling the network by setting control parameters in the device included in the network based on the behavior obtained from the learning model generated by the learning step.
  • the control step is a control method in which the control parameters are determined based on the influence of the behavior obtained from the learning model on the state of the network.
  • the control step determines the control parameter based on a fluctuation value of the control parameter obtained from the learning model.
  • the control step is log information including the state of the network obtained when the network is controlled, the fluctuation value of the control parameter when the network is controlled, and the amount of change in the state caused by the control of the network.
  • the control method according to Appendix 8 wherein the fluctuation value of the control parameter obtained from the learning model is weighted based on the above.
  • the control step is The fluctuation value of the control parameter obtained from the learning model and the fluctuation value of the control parameter included in the log information and corresponding to the state change in which the amount of change of the state caused by the control of the network is larger than the first threshold value. Calculate the difference between The control method according to Appendix 9, wherein the weight is changed based on the calculated difference.
  • control step 11 The control step is If the amount of change in the state caused by the control of the network is smaller than the second threshold value, the fluctuation value of the control parameter obtained from the learning model is discarded using the corresponding state of the network, as described in Appendix 10. Control method.
  • Appendix 12 The control step is described in Appendix 8, wherein when the control parameter obtained from the learning model has been updated in the past, the control parameter is determined based on the change of state of the network caused by the update of the control parameter. Control method.
  • the control means (102, 204) is a system that determines the control parameters based on the influence of the behavior obtained from the learning model on the state of the network.
  • the control means (102, 204) obtains when the network is controlled, the state of the network, the fluctuation value of the control parameter when the network is controlled, and the amount of change in the state caused by the control of the network.
  • the control means (102, 204) The fluctuation value of the control parameter obtained from the learning model and the fluctuation value of the control parameter included in the log information and corresponding to the state change in which the amount of change of the state caused by the control of the network is larger than the first threshold value.
  • the process of learning behavior to control the network A process of controlling the network by setting control parameters in the device included in the network based on the behavior obtained from the learning model generated by the learning step.
  • the control process is a program that determines the control parameters based on the influence of the behavior obtained from the learning model on the state of the network.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
PCT/JP2019/038456 2019-09-30 2019-09-30 制御装置、制御方法及びシステム Ceased WO2021064768A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/JP2019/038456 WO2021064768A1 (ja) 2019-09-30 2019-09-30 制御装置、制御方法及びシステム
US17/641,183 US20220345377A1 (en) 2019-09-30 2019-09-30 Control apparatus, control method, and system
JP2021550733A JP7251647B2 (ja) 2019-09-30 2019-09-30 制御装置、制御方法及びシステム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2019/038456 WO2021064768A1 (ja) 2019-09-30 2019-09-30 制御装置、制御方法及びシステム

Publications (1)

Publication Number Publication Date
WO2021064768A1 true WO2021064768A1 (ja) 2021-04-08

Family

ID=75337012

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/038456 Ceased WO2021064768A1 (ja) 2019-09-30 2019-09-30 制御装置、制御方法及びシステム

Country Status (3)

Country Link
US (1) US20220345377A1 (https=)
JP (1) JP7251647B2 (https=)
WO (1) WO2021064768A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023173643A (ja) * 2022-05-26 2023-12-07 株式会社東芝 情報処理装置、情報処理方法、および情報処理プログラム
JP2024543201A (ja) * 2022-07-15 2024-11-19 ▲騰▼▲訊▼科技（深▲セン▼）有限公司 データ伝送制御方法、装置、コンピュータ機器、及びコンピュータプログラム

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115643210A (zh) * 2019-11-30 2023-01-24 华为技术有限公司 控制数据包发送方法及系统
JP7478300B1 (ja) 2023-09-27 2024-05-02 株式会社インターネットイニシアティブ 通信制御装置および通信制御方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009027303A (ja) * 2007-07-18 2009-02-05 Univ Of Electro-Communications 通信装置および通信方法
JP2013106202A (ja) * 2011-11-14 2013-05-30 Fujitsu Ltd パラメータ設定装置、コンピュータプログラム及びパラメータ設定方法
JP2019041338A (ja) * 2017-08-28 2019-03-14 日本電信電話株式会社 無線通信システム、無線通信方法および集中制御局
US20190141113A1 (en) * 2017-11-03 2019-05-09 Salesforce.Com, Inc. Simultaneous optimization of multiple tcp parameters to improve download outcomes for network-based mobile applications

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112913274B (zh) * 2018-09-06 2024-04-23 诺基亚技术有限公司 用于自组织网络的优化的过程

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009027303A (ja) * 2007-07-18 2009-02-05 Univ Of Electro-Communications 通信装置および通信方法
JP2013106202A (ja) * 2011-11-14 2013-05-30 Fujitsu Ltd パラメータ設定装置、コンピュータプログラム及びパラメータ設定方法
JP2019041338A (ja) * 2017-08-28 2019-03-14 日本電信電話株式会社 無線通信システム、無線通信方法および集中制御局
US20190141113A1 (en) * 2017-11-03 2019-05-09 Salesforce.Com, Inc. Simultaneous optimization of multiple tcp parameters to improve download outcomes for network-based mobile applications

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023173643A (ja) * 2022-05-26 2023-12-07 株式会社東芝 情報処理装置、情報処理方法、および情報処理プログラム
JP7785611B2 (ja) 2022-05-26 2025-12-15 株式会社東芝 情報処理装置、情報処理方法、および情報処理プログラム
JP2024543201A (ja) * 2022-07-15 2024-11-19 ▲騰▼▲訊▼科技（深▲セン▼）有限公司 データ伝送制御方法、装置、コンピュータ機器、及びコンピュータプログラム
JP7661629B2 (ja) 2022-07-15 2025-04-14 ▲騰▼▲訊▼科技（深▲セン▼）有限公司 データ伝送制御方法、装置、コンピュータ機器、及びコンピュータプログラム

Also Published As

Publication number Publication date
US20220345377A1 (en) 2022-10-27
JP7251647B2 (ja) 2023-04-04
JPWO2021064768A1 (https=) 2021-04-08

Similar Documents

Publication Publication Date Title
JP7251647B2 (ja) 制御装置、制御方法及びシステム
CN111090631B (zh) 分布式环境下的信息共享方法、装置和电子设备
CN108092791B (zh) 网络控制方法、装置及系统
CN114827032A (zh) 利用强化学习执行网络拥塞控制
CN114500383B (zh) 天地一体化信息网络智能拥塞控制方法、系统及介质
CN114024906B (zh) 优化视频传输用户体验质量的流量控制方法、装置和系统
US12156118B2 (en) Methods and apparatus for data traffic routing
Xu et al. An actor-critic-based transfer learning framework for experience-driven networking
US20250238282A1 (en) Deterministic network architecture and working method for intelligent applications
CN107040961B (zh) 一种无线传感器网络拥塞控制方法
Gomez et al. Intelligent active queue management using explicit congestion notification
WO2024007499A1 (zh) 强化学习智能体训练方法、模态带宽资源调度方法及装置
CN118400337B (zh) 一种面向差异化QoS保障的链路汇聚节点缓存方法
CN111083535A (zh) 视频数据传输码率自适应方法、系统、装置和存储介质
JP7251646B2 (ja) 制御装置、方法及びシステム
JP7259978B2 (ja) 制御装置、方法及びシステム
CN118381765B (zh) 无损网络拥塞控制方法、装置、设备、介质及交换机系统
CN111211984A (zh) 优化cdn网络的方法、装置及电子设备
CN110598871A (zh) 一种微服务架构下的业务流柔性控制的方法及系统
CN116192766B (zh) 用于调整数据发送速率和训练拥塞控制模型的方法及装置
CN113672372A (zh) 一种基于强化学习的多边缘协同负载均衡任务调度方法
Tian et al. Achieving fairness generalizability for learning-based congestion control with jury
WO2019031258A1 (ja) 送信端末、送信方法、情報処理端末、および情報処理方法
Bisoy et al. Design of an active queue management technique based on neural networks for congestion control
Bai et al. MA-CC: Cross-Layer Congestion Control via Multi-agent Reinforcement Learning

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19947459

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021550733

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19947459

Country of ref document: EP

Kind code of ref document: A1