KR102670448B1 - Routing method and device using deep reinforce learning - Google Patents

Abstract

The disclosed technology relates to a deep reinforcement learning-based routing method and device, comprising: a control device of a software defined network receiving a communication service request from a first node among a plurality of nodes in the network; determining, by the control device, a QoS level for the communication service according to the request; And the control device inputs status information about available nodes among the plurality of nodes into a reinforcement learning model to calculate a plurality of routing paths that can provide the communication service, and determines the QoS level among the plurality of routing paths. It includes; selecting at least one corresponding path.

Description

Deep reinforcement learning-based routing method and device {ROUTING METHOD AND DEVICE USING DEEP REINFORCE LEARNING}

The disclosed technology relates to a deep reinforcement learning-based routing method and device that considers the QoS level of network services.

A conventional software defined network (SDN) provides a network management function. By providing administrators with a centralized view of network status, network management could be performed flexibly. However, actual communication networks have a very dynamically changing and complex environment, which requires high computing performance to maintain QoS levels that meet user requirements.

Korean Patent Publication No. 10-2020-0002439

The disclosed technology provides a deep reinforcement learning-based routing method and device that takes into account the QoS level of network services.

The first aspect of the technology disclosed to achieve the above technical problem is the step of a control device of a software defined network receiving a communication service request from a first node among a plurality of nodes in the network, and the control device providing the communication service according to the request. determining a QoS level, and the control device inputs status information about available nodes among the plurality of nodes into a reinforcement learning model to calculate a plurality of routing paths capable of providing the communication service, and calculating a plurality of routing paths capable of providing the communication service. The aim is to provide a deep reinforcement learning-based routing method that includes selecting at least one path corresponding to the QoS level among routing paths.

The second aspect of the technology disclosed to achieve the above technical problem is a communication device that receives a communication service request from a first node among a plurality of nodes in the network, and an enhancement learned to calculate a routing path using a plurality of nodes in the network. A storage device for storing a learning model and a QoS level for the communication service according to the request, and inputting status information on available nodes among the plurality of nodes into the reinforcement learning model to provide the communication service. The purpose of the present invention is to provide a deep reinforcement learning-based routing device that includes a computing unit that calculates a plurality of routing paths and selects at least one path corresponding to the QoS level among the plurality of routing paths.

Embodiments of the disclosed technology may have effects including the following advantages. However, since this does not mean that the embodiments of the disclosed technology must include all of them, the scope of rights of the disclosed technology should not be understood as being limited thereby.

The deep reinforcement learning-based routing method and device according to an embodiment of the disclosed technology has the effect of distributing the load concentrated on a specific routing path by using a routing path that satisfies the requested service among available routing paths.

Additionally, it has the effect of reducing network complexity by tiering QoS for communication services.

Figure 1 is a diagram showing a deep reinforcement learning-based routing process according to an embodiment of the disclosed technology.
Figure 2 is a flowchart of a deep reinforcement learning-based routing method according to an embodiment of the disclosed technology.
Figure 3 is a block diagram of a deep reinforcement learning-based routing device according to an embodiment of the disclosed technology.
Figure 4 is a diagram showing the learning performance of a reinforcement learning model according to the disclosed technology.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention.

Terms such as first, second, A, B, etc. may be used to describe various components, but the components are not limited by the terms, and are only used for the purpose of distinguishing one component from other components. It is used only as For example, a first component may be referred to as a second component, and similarly, the second component may be referred to as a first component without departing from the scope of the present invention. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

As used herein, singular expressions should be understood to include plural expressions unless the context clearly dictates otherwise. And terms such as "include" mean the presence of implemented features, numbers, steps, operations, components, parts, or combinations thereof, but one or more other features, numbers, steps, operations, components, parts, etc. It should be understood that it does not exclude the existence or addition of combinations thereof.

Before providing a detailed description of the drawings, it would be clarified that the division of components in this specification is merely a division according to the main function each component is responsible for. That is, two or more components, which will be described below, may be combined into one component, or one component may be divided into two or more components for more detailed functions.

In addition to the main functions it is responsible for, each of the components described below may additionally perform some or all of the functions handled by other components, and some of the main functions handled by each component may be performed by other components. Of course, it can also be carried out exclusively by . Therefore, the presence or absence of each component described throughout this specification should be interpreted functionally.

Software Defined Network (SDN) is a new networking paradigm that provides on-demand resource allocation, easy reconfiguration, and programmable network management functions. In SDN, network management functions are logically composed of a control plane and a data plane. SDN provides a centralized view of network status, making network management and control more flexible, consistent, and comprehensive. This can improve traffic control and management by dynamically adjusting the allocated bandwidth and paths for user connection services.

The routing module of the SDN control plane collects network state information in real time. Based on this, a path that satisfies QoS is provided between both ends of communication according to Quality of Service (QoS) parameter information such as delay, loss, and bandwidth requested by the user. Various studies have been conducted on QoS routing depending on network conditions, and most of these studies are model-based and conducted on the assumption that user demands and network environments can be modeled appropriately. Additionally, processing multiple QoS parameters requires a high level of computing resources. Meanwhile, communication networks are very dynamic, and as they evolve in complexity, modeling and control are difficult.

After DeepMind proposed DQN (Deep Q-network), the Deep Reinforcement Learning (DRL) method does not require an accurate mathematical modeling process because it learns through experience, and can solve very complex problems, so it has been widely used in various fields. It is being applied. Traditional reinforcement learning has limitations when applied to large-scale systems that require complex sets of states and actions. Deep reinforcement learning can overcome the limitations faced by traditional reinforcement learning by combining deep learning with reinforcement learning theory. Researchers from various fields have become interested in DRL because it has the ability to solve large-scale, complex problems posed in the fields of communications and networking. A QoS-aware adaptive greedy online routing algorithm was proposed by applying reinforcement learning in a software-defined network. Several recent studies have applied various DRL algorithms, such as DDPG (Deep Deterministic Policy Gradient), to routing in communication networks, and have defined the routing problem as a continuous control problem and considered k-shortest paths. That is, the continuous traffic flow traffic matrix is used to determine the shortest path. This DRL method considers only the k-shortest path for communication between each source-destination pair. Therefore, there is a possibility that performance may be limited because there may be other paths that can provide better quality of service.

In the present invention, the routing problem is defined as a control problem. A sender wishing to communicate transmits not only destination information but also the minimum service quality parameters required to receive a smooth service to the service provider. Service quality parameters are converted to appropriate service levels through a mapping function. The SDN controller has information that has converted the current network status information into the service level corresponding to each link. The DRL agent learns to find a path for communication that satisfies the QoS requirements between source and destination pairs based on service level information.

Figure 1 is a diagram showing a deep reinforcement learning-based routing process according to an embodiment of the disclosed technology. Referring to Figure 1, the control device may use a controller of a software defined network. The control device is equipped with a reinforcement learning model and can calculate the routing path by inputting state information about the network into the reinforcement learning model. Conventionally, in this process, the best routing path that can be provided in the current network is provided for communication services, so if a large number of similar services are requested within a short period of time, the corresponding routing path or a significant portion of a plurality of nodes included in the routing path are used. There is a problem in which overlapping requested communication services cannot be provided smoothly. In order to solve this problem, the disclosed technology is a technology that distributes the load across the network by classifying the QoS level for communication services requested from nodes into stages and providing routing paths for each stage.

Meanwhile, as shown in Figure 1, the control device collects network status information in real time. Based on this, a routing path that satisfies QoS can be provided between both ends of communication according to Quality of Service (QoS) parameter information such as delay, loss, and bandwidth requested by the user. The control unit defines the determination of the routing path as a control problem. A sender wishing to communicate transmits not only destination information but also the minimum service quality parameters required to receive smooth service to the service provider. Service quality parameters are converted to appropriate service levels through a mapping function. The control device has information that has converted the current network status information into the service level corresponding to each link. The reinforcement learning model is trained to find a path for communication that satisfies the QoS requirements between source and destination pairs based on service class information.

Meanwhile, referring again to FIG. 1, the entire system consists of a data layer and an SDN control layer consisting of a plurality of nodes requesting communication services. A plurality of nodes may be a combination of users and routers. The data layer transmits data between network devices, and the control layer performs communication between the application layer and the data layer, which performs routing functions, etc., dynamically updates data forwarding rules, and allocates and controls network resources. performs the function of The control layer understands all state information of the current network, and through this, controls route settings and data delivery logic of each router when a user requests communication. The network can be represented as a bidirectional graph G(V, E). Here, V represents the set of all routers and E is defined as the set of links as E = {(i,j)|(i,j) → V × V, i≠j}. Status information for each link is managed separately for delay, loss rate, and bandwidth.

The QoS-aware routing problem is defined as the problem of determining a path that satisfies the flow f representing the QoS requirements between the source node x and the destination node y. In other words, it refers to the problem of determining a path that minimizes delay and loss while maximizing bandwidth, which is a very complex problem. Therefore, mathematical modeling is very difficult, and learning is very slow even when approximating using deep learning. Therefore, in the present invention, the QoS level required by each flow is staged, the minimum QoS level that each link can currently provide in the SDN network is calculated in advance, and only links that satisfy the QoS level required by flow f are found and sent to the source. Replace it with the problem of determining the route to the destination. The procedure for providing communication services in this environment is as follows.

First, a user who wants a communication service transmits destination information and a service profile to the access router. The service profile includes the minimum QoS parameter values required to provide the corresponding service. The QoS parameters considered in the present invention consist of bandwidth, delay, and loss rate. The service profile requested by the user is converted into a service level through a corresponding function. The router requests the routing function within the SDN to set up a route, and transmits it including source and destination node information and service level information. The SDN routing layer uses the same response function to convert the current network state information into a matrix of service levels provided by each link. The DQN agent receives the sender, destination, and service level as input and determines route information that satisfies the service level through interaction with the environment. The minimum QoS level is determined differently depending on the required bandwidth, delay, loss, etc., and can be changed according to each weight.

In the DRL-based routing algorithm to find a path between two nodes that perform communication while satisfying the QoS parameters specified in the service profile, the state, action, and reward are defined as a state space, action space, and reward function, respectively, as follows.

First, the state space must include the current state information of the network, and the network connection information and link state information are primarily determined through the control device. The initial state information is |V| for each QoS parameter. × |V| It is expressed as a two-dimensional matrix of sizes. Since the actual values for each parameter may have different scales, they undergo a normalization process. Afterwards, through the corresponding function, the QoS parameter information for each link is converted into an integrated two-dimensional matrix that is converted into the service level value of the link.

Next, the action space is defined as the set of all links in the network. That is, the action space vector is A = [a ₁ , a ₂ ,. . . , a _|E| ], and each action corresponds to link (i, j) ∈ E of the network. The actions that can be selected at each node are limited to the output link that goes out from that node, but to simplify the model, all links are selectable, and the output link is learned to determine the action through the reward value.

Meanwhile, in order to increase the learning accuracy of the model, the compensation function must be appropriately specified. Because the action set is defined so that the agent can select any link connected to the network, the reward function needs to consider various situations. In an episode of T time steps, the reinforcement learning agent must find a path from source node x to target node y. At a random node z, the agent selects an action at time step t corresponding to link (i, j) and is rewarded by the reward function f((i, j)) as shown in Equation 1 below. receive

here represents the set of valid actions of node z. In other words, it is considered a valid action only if the output link is selected at node z. If the selected action is invalid, the agent receives a reward of -|V|/ 2, i.e. a penalty. Otherwise, compensation is determined by a separately defined function. To achieve this, a very large negative reward is specified when selecting a link heading to an already selected node. for teeth The list is defined as empty at the beginning of each episode and represents a set that is then filled with links. Agents do not get stuck in network loops by repeatedly selecting the same link in an episode. If the target node on the selected link is destination node y, this state is a terminal state, and the environment returns the highest positive reward value |V| to the agent. We limit each episode to T time steps so that the agent does not get stuck in an infinite loop while exploring the action space. If the current time step is greater than the total number of edges, the agent is lost and the episode ends with a high penalty of -|V|. In the remaining cases, the link to the agent's selected action returns appropriate positive and negative rewards compared to the required QoS level.

Meanwhile, the state space, action space, and reward function defined as described above are applied to DQN. The learning algorithm of DQN is defined by Equation 2 below.

The input layer is |V| × |V| It is one-dimensional information of size. Two hidden layers with 512 neurons and a linear unit modified with activation function were used. The size of the output layer is |A| , and the output is a queue function for each action. Among the output queue function values, select the action that represents the maximum queue function value. Additionally, in the training process of the DRL agent, hyperparameters such as batch size 64, learning rate = 0.001, and buffer size = 5000 were used. In addition, an epsilon greedy policy was selected for continuous learning, and the initial epsilon value indicating the probability of selecting a random policy was set to 0.9, and as learning progressed, epsilon decay was used to lower the probability of selecting a random policy. The value was set to 0.99.

At the beginning of the algorithm, an instance of the environment is created by specifying the number of network nodes and link information. Afterwards, the hyperparameters of the playback buffer, main Q-network, and target Q-network are initialized. The algorithm is run for a total of N episodes. At the start of each episode, network link state information is normalized. Through the corresponding function, the QoS parameter information for each link is converted into an integrated two-dimensional matrix that is converted into the service level value of the link. An empty set used to inform the reward function for already visited edges. is created. Each episode has a duration of T time steps (where T = | E |).

At each time step t, act using an epsilon greedy method. is selected. If the selected action is a valid action, it is executed in the environment and compensated according to the reward function. and get to and updates the state space vector. Otherwise, if the selected action is not a valid action, a negative reward value is passed and the same state space is used without change for the next iteration. , , , After getting +1, the transition is stored in the experience playback buffer. When a certain amount of transition information is accumulated in the playback buffer, random mini-batch transitions are sampled from the playback buffer and the weights of the deep neural network are optimized using gradient descent for θ to minimize loss. The target network parameters are updated through the main network parameters at regular intervals. The repetition of an episode occurs when the path to the destination node is completed, or t > |E| If , it ends. Learning of the reinforcement learning model proceeds according to the above-described process, and a routing path that satisfies the QoS level can be calculated according to the learning results.

Figure 2 is a flowchart of a deep reinforcement learning-based routing method according to an embodiment of the disclosed technology. Referring to FIG. 2, the deep reinforcement learning-based routing method 200 includes a step 210 in which the control device receives a communication service request from a first node among a plurality of nodes in the network, and the control device performs QoS for the communication service according to the request. In step 230 of determining the level, the control device calculates a plurality of routing paths that can provide a communication service by inputting status information about available nodes among the plurality of nodes into the reinforcement learning model and selecting one of the plurality of routing paths. It includes a step 230 of selecting at least one path corresponding to the QoS level. The deep reinforcement learning-based routing method 200 can be performed through a control device of a software-defined network.

In step 210, the control device receives a request for a communication service from a first node among a plurality of nodes in the network. A node requesting a communication service from a control device may be a single node or multiple nodes.

In step 220, the control device determines the QoS level for the communication service according to the request. The control device may determine a plurality of QoS levels for each communication service in advance to provide communication services. For example, when requesting transmission of high-resolution video that requires processing a large amount of data, it can be classified as high level, and when requesting transmission of low-capacity data such as text messages, it can be classified as low level. The number of QoS levels may vary depending on the settings of the control device.

In step 230, the control device calculates a plurality of routing paths capable of providing communication services by inputting status information about available nodes among a plurality of nodes into the reinforcement learning model, and calculates at least one routing path corresponding to the QoS level among the plurality of routing paths. Choose one path. The reinforcement learning model installed in the control device is a model that receives state information and calculates optimal behavior. Here, status information refers to network connection information and link status. And the action means defining a set of all links in the network and calculating a plurality of routing paths that can be created using the set of links.

Reinforcement learning models can be trained to calculate the optimal routing path according to the learning process. In other words, the reward can be learned to be the maximum value for each learning round. In the beginning, when learning is not sufficient, the link selected during the last learning can be selected again, and in this case, it can be set to have a very large negative reward value. Naturally, as the learning round is repeated, the reinforcement learning model gradually selects links in the direction with positive reward values. By repeating this process, learning can be performed so that the reward value becomes the maximum. According to this learning process, it is possible to output the optimal path in the later testing stage.

Figure 3 is a block diagram of a deep reinforcement learning-based routing device according to an embodiment of the disclosed technology. Referring to FIG. 3, the deep reinforcement learning-based routing device 300 includes a communication device 310, a storage device 320, and a computing device 330.

The communication device 310 receives a communication service request from a first node among a plurality of nodes in the network. The communication device 310 may be a communication module through which a control device within a software-defined network receives communication service requests from nodes on the network.

The storage device 320 stores a reinforcement learning model learned to calculate a routing path using a plurality of nodes in the network. The storage device 320 may be a memory of the deep reinforcement learning-based routing device 300.

The computing unit 330 determines the QoS level for the communication service upon request, inputs status information about available nodes among the plurality of nodes into the reinforcement learning model, and calculates a plurality of routing paths that can provide the communication service. Among the plurality of routing paths, at least one path corresponding to the QoS level is selected.

Meanwhile, the deep reinforcement learning-based routing device 300 described above may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be stored and provided in a temporary or non-transitory computer readable medium.

A non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, the various applications or programs described above include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM (read-only memory), PROM (programmable read only memory), and EPROM (Erasable PROM, EPROM). Alternatively, it may be stored and provided in a non-transitory readable medium such as EEPROM (Electrically EPROM) or flash memory.

Temporarily readable media include Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), and Enhanced SDRAM (Enhanced RAM). It refers to various types of RAM, such as SDRAM (ESDRAM), synchronous DRAM (Synclink DRAM, SLDRAM), and Direct Rambus RAM (DRRAM).

Figure 4 is a diagram showing the learning performance of a reinforcement learning model according to the disclosed technology. The x-axis represents the number of episodes, and the y-axis represents the accumulated reward value when the episode ends. Since the process of exploring the environment is performed for a considerable period of time in the early stages of learning, the agent may reselect an action that is invalid or has already been selected. It was found that most episodes ended abnormally and thus received very large accumulated negative reward values. As the agent gradually learns, the accumulated reward value it receives increases as it selects actions that avoid invalid actions and network loops. As the accumulated reward value approaches 0, the agent learns routing methods based on a significant portion of the network state. When there are 250 episodes or more, a positive accumulated reward value is received, which means that agent learning has been successful.

The deep reinforcement learning-based routing method and device according to an embodiment of the disclosed technology have been described with reference to the embodiments shown in the drawings to aid understanding, but this is merely an example, and those skilled in the art will understand this. It will be understood that various modifications and equivalent other embodiments are possible. Therefore, the true technical protection scope of the disclosed technology should be determined by the appended patent claims.

Claims (10)

A control device of a software defined network receiving a communication service request from a first node among a plurality of nodes in the network;
determining, by the control device, a QoS level for the communication service according to the request; and
The control device inputs status information about available nodes among the plurality of nodes into the reinforcement learning model to calculate a plurality of routing paths that can provide the communication service, and corresponds to the QoS level among the plurality of routing paths. Including; selecting at least one path to
The control device classifies a plurality of QoS levels for communication services in advance and determines which of the plurality of QoS levels the communication service requested from the node corresponds to,
The control device inputs the network connection information and link status information as the status information, and
A deep reinforcement learning-based routing method in which the reinforcement learning model defines a set of all links in the network and calculates a plurality of routing paths that can be created using the set of links. delete delete delete According to claim 1,
The reinforcement learning model is a deep reinforcement learning-based routing method that is learned so that the reward is maximized in each learning round. A communication device that receives a communication service request from a first node among a plurality of nodes in a network;
a storage device that stores a reinforcement learning model learned to calculate a routing path using a plurality of nodes in the network; and
According to the request, the QoS level for the communication service is determined, and status information on available nodes among the plurality of nodes is input into the reinforcement learning model to calculate a plurality of routing paths capable of providing the communication service. It includes a computing device that selects at least one path corresponding to the QoS level among a plurality of routing paths,
The computing device classifies a plurality of QoS levels for communication services in advance and determines which of the plurality of QoS levels the communication service requested from the node corresponds to,
The computing device inputs the network connection information and link status information as the status information, and
A deep reinforcement learning-based routing device in which the reinforcement learning model defines a set of all links in the network and calculates a plurality of routing paths that can be created using the set of links. delete delete delete According to claim 6,
The reinforcement learning model is a deep reinforcement learning-based routing device that is learned so that the reward is maximized in each learning round.