CN113992595B - SDN data center congestion control method based on priority experience playback DQN - Google Patents

Abstract

The invention discloses a congestion control method of an SDN data center network based on priority experience playback DQN. The method of the present invention is based on the network background of SDN, based on the flow congestion control idea, and has the characteristics of intelligence, centralization and initiative. Introduce and improve the congestion control algorithm based on priority experience playback DQN, and provide an intelligent solution to the congestion control problem of SDN data center network. The present invention distributes the rate globally to the flow of the whole network through the controller, which can not only avoid the congestion of the whole network, but also make the utilization rate of the data link of the network as high as possible, so as to realize the congestion control of the whole data center. Compared with the method based on Q-learning, this method solves the dimensionality disaster of the Q table; compared with the methods based on DQN and DDQN, it has better convergence speed and convergence effect. The invention is a congestion control method adapted to the SDN data center network conforming to the development trend of network intelligence.

Description

A Congestion Control Method for SDN Data Center Based on Priority Experience Playback DQN

technical field

The present invention relates to the technical field of network communication, in particular to an SDN (Software Defined Network, software-defined network) data center network (Data Center Network, DCN) congestion control method.

Background technique

As a future network architecture, SDN architecture has been widely adopted by data center networks. With the development of big data and cloud computing, the number of nodes and the number of flows in the SDN data center network are increasing, and the data center is facing the risk of network congestion. Since DQN needs to gain experience in exploration, the parameters of the neural network are updated through experience playback, but the random uniform sampling strategy it adopts is not the optimal strategy in certain cases. When DQN is introduced into congestion control, due to the large number of action options for each allocation, it is difficult to obtain experience samples that are neither congested nor have high link utilization in the process of random exploration, which brings difficulties to the training of neural networks. Therefore, it is urgent to design a new SDN data center network congestion control method to solve such problems.

Contents of the invention

The purpose of the present invention is to intelligently solve the congestion control problem of the data center network based on the SDN architecture, and proposes a congestion control method of the SDN data center network based on priority experience playback DQN.

The inventive idea of the present invention is to introduce and improve the congestion control algorithm based on priority experience playback DQN, and provide an intelligent solution to the congestion control problem of the SDN data center network. The present invention distributes the rate globally to the flow of the whole network through the controller, which can not only avoid the congestion of the whole network, but also make the utilization rate of the data link of the network as high as possible, so as to realize the congestion control of the whole data center.

The purpose of the present invention is achieved through the following technical solutions:

A kind of SDN data center congestion control method based on priority experience playback DQN, it comprises the following steps:

S1. Deploy the congestion control agent based on the deep Q network in the SDN controller, and introduce the priority experience playback DQN algorithm into the data center based on the software-defined network;

S2. Training the deep Q network, wherein the training process includes S21～S24:

S21. Set the input of the deep Q network as link state information and flow state information, and the output is the Q value corresponding to different actions. The different actions indicate that different rates are allocated to the flow, and the reward function is to balance link utilization and A comprehensive function of link congestion;

S22. Randomly construct any initial link state and any set of flow and rate requirements to realize scene construction;

S23. Construct a SumTree for storing experience, and mark the priority of experience;

S24. Select experience from SumTree according to the priority, and play back the DQN improved algorithm to train the deep Q network according to the priority experience, so that the SDN controller can maximize the data link utilization rate through the deep Q network while ensuring that the data center is not congested. ; Wherein, the improved priority experience playback DQN algorithm is based on the priority experience playback DQN algorithm, adding a judgment on whether the link is congested before the end of each step of each scene of network training, and if it is congested, then directly end the scene , if there is no congestion, continue to the next step; the scenario indicates that the rate is allocated for the entire group of flows, and each step in the scenario indicates that the rate is allocated for a flow;

S3. The SDN controller collects the link state information and the flow state information of the rate to be allocated in real time from the SDN data plane, and inputs it into the trained deep Q network, determines the optimal action according to the Q value of each flow and generates the flow Rate allocation scheme, so as to perform global congestion control on the SDN data center network.

Preferably, the SDN controller is connected to the network equipment of the SDN data plane through a southbound interface to realize centralized control.

Preferably, the reward function is:

In the formula: reward _m represents the reward value, min represents the minimum value operation, LkCap _m represents the link utilization rate,

Preferably, the priority experience replay DQN algorithm is a DQN algorithm in which the experience replay mechanism is replaced by a priority experience replay.

Preferably, the priority mark is an empirical importance mark determined according to TD-error, and the TD-error refers to the absolute value of the difference between the current empirical Q value and the target Q value in the time sequence difference.

As preferably, in said S3, the method for the SDN controller to perform global congestion control includes the following steps:

S31. Obtain the rate requirements and routing information of the N streams currently to be allocated from the SDN data plane, and at the same time obtain the status of each link in the current SDN data center network, that is, the link bandwidth occupancy;

S32. Select a flow from the current N flows to be allocated, input the information of this flow and the current link state into the depth Q network trained by S2, and select the optimal action to execute according to the output of the depth Q network;

S33. Update the current link state, and record the mapping information between the current flow and the allocated rate at the same time.

S34, judging whether all the N streams have been allocated, if not, then return to continue looping S32 and S33 until the rate is allocated for all streams; if all the streams are allocated, then execute S35;

S35. Output the flow rate allocation mapping table of the N flows, and the SDN controller allocates a rate for each flow using the mapping table.

The beneficial effects of the present invention are: the present invention proposes an intelligent solution based on priority experience playback DQN for the congestion control problem of the SDN data center, which can centrally, proactively and intelligently perform congestion according to the load change of the data center network link control. The invention overcomes the disadvantage of weak multi-dimensional perception ability of reinforcement learning, and solves the dimension disaster of Q table; at the same time, by introducing a priority experience playback mechanism, it has better convergence speed and convergence effect than the traditional DQN algorithm. In this method, the controller allocates the rate globally to the flow of the whole network, which can not only avoid the congestion of the whole network, but also make the utilization rate of the data link of the network as high as possible, so as to realize the congestion control of the whole data center.

Description of drawings

Fig. 1 is a block diagram of the congestion control system in the embodiment.

FIG. 2 is a topology diagram of a data center network used in an embodiment.

Figure 3 is a flow chart of the training algorithm.

FIG. 4 is a flowchart of a congestion control method.

Fig. 5 is a bandwidth change diagram of the embodiment.

Figure 6 shows the comparison of link utilization ratios of different algorithms under different flow numbers (each group of three columns represents DQN, DDQN, and PRIO from left to right).

Figure 7 shows the comparison of the convergence speed of different algorithms.

Detailed ways

The present invention will be further elaborated and illustrated below in conjunction with the accompanying drawings and specific embodiments. The technical features of the various implementations in the present invention can be combined accordingly on the premise that there is no conflict with each other.

For ease of description, some basic definitions involved in the present invention will be explained below first.

The SDN data center in the present invention refers to a data center using a software-defined network technology architecture.

In the present invention, the data center congestion control problem of SDN refers to the flow-based congestion control problem, which refers to the overall planning and allocation rate of all flows through the SDN controller, which not only tries to meet the flow rate requirements, but also ensures that the entire data center network does not generate congestion.

In the present invention, the prioritized experience replay DQN algorithm is a DQN algorithm based on prioritized experience reolay, wherein the prioritized experience reolay (prioritized experience reolay) belongs to the prior art, and is an improvement on the experience replay mechanism in the DQN algorithm. Experience refers to the samples under test. The DQN algorithm belongs to the prior art and is a classic algorithm in deep reinforcement learning. The deep reinforcement learning is a learning method that combines the advantages of deep learning and reinforcement learning to solve problems that require perception of high-dimensional original input and decision-making control.

In a preferred embodiment of the present invention, a kind of SDN data center congestion control method based on DQN is provided, and this method comprises the steps:

Step 1: Deploy a deep Q-network-based congestion control agent in the SDN controller, thereby introducing the prior experience replay DQN algorithm into the software-defined network-based data center.

The SDN controller is the control and decision-making part of the SDN network. It is connected to the network equipment of the SDN data plane through the southbound interface, so as to centrally control the rate distribution of the flow in the entire network. The congestion control agent deployed in the SDN controller is based on the deep Q network DQN to allocate the flow rate. The deep Q network in the present invention executes the priority experience playback DQN algorithm, and the purpose is to solve the problem of congestion control in the SDN data center.

Step 2: Based on the priority experience playback DQN improved algorithm, train the deep Q network. The specific training process includes the following steps:

2-1. Determine the input and output of the deep Q network. The input of the deep Q network is link state information and flow state information, and the output is the Q value corresponding to different actions, which represent different rates allocated to the flow.

2-2. Set the reward function. Because the first purpose of congestion control is to avoid link congestion as much as possible, and the second purpose is to maximize link utilization, so the reward function is a comprehensive function set while considering link utilization and link congestion . The form of the reward function is not limited, as long as it can balance link utilization and link congestion. In subsequent examples of the present invention, a reward function form can be set as:

In the formula: reward _m represents the reward value, min() represents the operation of taking the minimum value in parentheses, and LkCap _m represents the link utilization rate.

2-3. By improving the priority experience replay DQN algorithm, an improved priority experience replay DQN algorithm is formed. The so-called improvement is based on the priority experience playback DQN algorithm. The specific improvement is to increase the judgment of whether the link is congested before the end of each step in each scene during the network training process of the algorithm. If congested, then End the scene, do not execute the subsequent steps in the scene, if there is no congestion, continue to the next step. Among them, the so-called step refers to allocating a rate for a flow; the so-called scene refers to completing the rate allocation for the whole group of flows, that is, each flow in a group of flows has completed the rate allocation.

This priority experience playback DQN improved algorithm will be used to train the deep Q network later.

2-4. Construction scenario: Randomly construct any initial link state, randomly construct any set of flows and rate requirements.

2-5. Construct SumTree to store experience and mark the priority of experience. The SumTree here refers to a binary tree data structure, the priority mark is the empirical importance mark determined according to TD-error, and the TD-error refers to the current empirical Q value and the target Q value in the time sequence difference The absolute value of the difference between.

2-6. Select the experience from SumTree according to the priority, and train the deep Q network according to the aforementioned improved priority experience playback DQN algorithm, that is, increase the judgment of whether the link is congested before the end of each step of each scene in the training process, If it is congested, end the scene directly; if not, proceed to the next step. The other training process of the priority experience replay DQN algorithm is the same as the prior art.

Step 3: The SDN controller collects the link state information and the flow state information of the rate to be allocated in real time from the SDN data plane, and inputs it into the trained deep Q network, determines the optimal action according to the Q value of each flow and generates Flow rate allocation scheme, so as to perform global congestion control on the SDN data center network.

In this embodiment, the method for the SDN controller to perform global congestion control in step 3 includes the following steps:

3-1. Obtain the rate requirements and routing information of the N streams currently to be allocated from the SDN data plane, and at the same time obtain the status of each link in the current SDN data center network, that is, the link bandwidth occupancy;

3-2. Select a flow from the current N flows to be allocated, input the flow information and current link status into the deep Q network after S2 training, and select the optimal action to execute according to the output of the deep Q network ;

3-3. Update the current link state, and record the mapping information between the current flow and the allocated rate at the same time.

3-4. Determine whether all the N streams have been allocated: if not, return to step 3-2 and continue the loop until the rate is allocated for all streams; if the allocation is complete, perform step 3-5;

Determine whether all the N streams have been allocated. If not, return to step 3-2 and continue to loop and execute 3-2 and 3-3 until the rate is allocated for all streams; if the allocation is complete, then execute 3-5;

3-5. Output the flow rate allocation mapping table of N flows, and the SDN controller allocates the rate for each flow according to the mapping table. In this way, the allocated rate can not only meet the rate requirements of all flows as much as possible, but also effectively avoid congestion, so as to achieve the purpose of global congestion control for the data center.

The core of the above-mentioned method provided by the present invention is to introduce the improved priority experience playback DQN algorithm into the SDN data center to solve the congestion control problem, so that the SDN controller can centrally, proactively and intelligently Carry out congestion control, overcome the shortcoming of reinforcement learning's weak multi-dimensional perception ability, and solve the dimension disaster of Q table. At the same time, the present invention has better convergence speed and convergence effect compared with the traditional DQN algorithm by introducing a priority experience playback mechanism. In this method, the controller allocates the rate globally to the flow of the whole network, which can not only avoid the congestion of the whole network, but also make the utilization rate of the data link of the network as high as possible, so as to realize the congestion control of the whole data center. Specifically, this method has the following advantages at the same time:

1. Intelligence. Said intelligence refers to because the introduction of priority experience playback DQN possesses intelligence, compared with the congestion control method based on optimization theory, the complexity of the algorithm is reduced; compared with the congestion control method based on reinforcement learning Q-learning, the solution The curse of dimensionality of the Q table.

2. End-to-end congestion control idea. The idea of end-to-end congestion control refers to implementing congestion control by globally allocating the rate of end-to-end flows, instead of considering congestion control enhancement on intermediate nodes such as routers.

3. Centralized network arbitration congestion control. The centralized network arbitration congestion control means that the congestion control method is deployed on the controller of the SDN, fully utilizes the advantages of the centralized control of the SDN, and distributes the overall end-to-end flow rate according to the state of the entire network. Unlike traditional TCP-based congestion control, which is distributed, each end system detects congestion through a TCP connection, and performs congestion avoidance based on the slow-start algorithm.

4. The granularity is flow. The granularity is flow means that the object of congestion control is flow, so as to adapt to the flow-based characteristics of the next generation network architecture SDN, while the traditional idea of congestion control based on TCP is based on IP packets.

5. Active congestion control. The active congestion control means that the idea of the congestion control method of the present invention is that the controller collects the status of the entire network, and the current flow and rate requirements, and combines the two to globally allocate the rate. The goal is to avoid congestion as much as possible while making the link utilization rate of the entire network as high as possible. Therefore, the characteristic is active. The TCP congestion control algorithm takes action after detecting congestion, which is relatively passive.

6. Adopt the priority experience playback strategy. Using the priority experience playback strategy refers to using the proportional priority method for priority experience playback, so as to preferentially obtain experience samples that are neither congested nor have high link utilization. However, the traditional DQN-based congestion control method uses a uniform sampling strategy, which is not conducive to obtaining good empirical samples.

Example

In order to facilitate those of ordinary skill in the art to understand and realize the present invention, and to demonstrate the technical effects of the DQN-based SDN data center congestion control method in the above-mentioned embodiment, the method in the above-mentioned embodiment is now applied to a specific test application example, combined with The figures and data further illustrate the advantages of the invention in practice.

This test application example introduces the priority experience replay DQN algorithm into the data center based on software-defined network to solve the congestion control problem in real time. Figure 1 is the architecture diagram of the congestion control system, in which the SDN controller is the control and decision-making part of the SDN network. It performs centralized control with the network equipment of the SDN data plane through the southbound interface, that is, the control-forward communication interface, and provides flexible programmability. . The present invention introduces the priority experience playback DQN algorithm into the data center based on the software-defined network by deploying the congestion control agent in the SDN controller, collects the link state information and flow state information of the data forwarding plane through the southbound interface, and integrates these state information Input into the neural network to generate a flow rate allocation plan, and then send the allocation plan to the network device on the data plane through the southbound interface. Under the premise of ensuring that the link is not congested, the link utilization rate should be as high as possible.

Figure 2 is a topology diagram of the SDN data center network used in this test application example. There are 8 links in the whole network, and the link bandwidth is 40G. The flow queue length used in this test application example is 28.

In this test application example, the specific DQN-based SDN data center congestion control method includes the following steps:

Step 1: Introduce the DQN algorithm into the congestion control problem of SDN data center.

In the architecture diagram of the congestion control system based on priority experience playback DQN shown in Figure 1, the whole process can be described as follows: First, the SDN controller obtains the status information and status information of each link in the data center network in real time from the data plane through the southbound interface. The flow state information to be allocated; then, by deploying the congestion control agent in the SDN controller, the priority experience playback DQN algorithm is introduced into the data center based on the software-defined network, and the link state information and flow state information collected from the southbound interface, Input into the neural network to generate a flow rate allocation plan; finally, the SDN controller sends the allocation plan to the network devices on the data plane through the southbound interface.

Step 2: Based on the improved DQN algorithm, train the deep Q network according to steps 2-1 to 2-6.

2-1. Determine the input and output of the deep Q network. The input of the deep Q neural network is the state of the link and the state of the current flow to be allocated. The state of the flow is the sequence number of the current flow and the path that the flow passes. The output is the Q value corresponding to different actions that assign different rates to the stream.

Figure 2 is a network topology diagram of this test application example. As shown in the figure, when the bandwidth of each link is 40G, there are 28 flows, passing through L1-L2, L1-L3, ..., L6-L8, L7-L8 respectively, and the bandwidth requirements are all 5G . On the premise of meeting the rate requirements of all flows as much as possible, use the DQN-based congestion control method to allocate a rate for each flow, and ensure that the network is not congested.

In this test application example, the input is the state of 8 links and 28 flows, which is input to the neural network in the DQN algorithm for training, and the output is a mapping table of flows and rates.

2-2. Set the reward function. Since the first purpose of congestion control is to avoid link congestion as much as possible, and the second purpose is to maximize link utilization, the reward function is a function set by comprehensively considering link utilization and link congestion. In this test application example, the given reward function is

In the formula: reward _m represents the reward value, min() represents the operation of taking the minimum value in parentheses, LkCap _m represents the link utilization rate, done is true represents link congestion, and done is false represents link no congestion.

2-3. Improve the DQN algorithm based on priority experience replay. As mentioned above, the improvement here refers to the addition of a judgment on whether the link is congested before the end of each step of each training scenario on the basis of the DQN algorithm based on priority experience playback, and if it is congested, the scenario is ended. The step refers to allocating a rate for a flow, and the scenario refers to completing the rate allocation for the entire group of flows. The improved DQN algorithm based on priority experience replay is called priority experience replay DQN improved algorithm.

The rest of the DQN algorithm based on priority experience replay is the same as the prior art. In order to facilitate understanding, the training flow chart of the improved algorithm of priority experience playback DQN is given in Figure 3, which specifically includes the following steps:

2-3-1. Initialize the neural network that stores training samples, action-value functions, and target action-value functions.

In this test application example, the sample size for storage training is set to 4500, and the neural network of the action-value function and the neural network of the target action-value function both use the basic DNN neural network. The initial load of the 8 links is [21,27,20,28,22,26,23,18].

2-3-2. Select a random action according to the probability ∈, input the current link state, calculate the Q value of each action, and select the action with the largest Q value under the premise of considering the routing information of the current flow ( optimal action) to execute.

In this test application example, the probability ∈ is set to 0.9, the learning rate is 0.001, and the selected action set is [0G, 1G, 2G, 3G, 4G, 5G]. Choose one of six rates to execute.

2-3-4. Obtain the reward r _t after executing a _t , the next input φ _t+1 and the link congestion judgment label done (if the link congestion done is true, otherwise it is false). After continuous iteration, the target action value function parameter is updated as the current action value function parameter.

In this test application example, φ _t , a _t , r _t , φ _t+1 , done represent the current output, current action, reward value, next output and link congestion judgment label respectively. These data are stored in SumTree, and each training takes Batch-size samples from SumTree according to the priority. Calculate the target value of each state and update it by SGD stochastic gradient descent method. The Batch-size here is set to 32, and the number of training steps is 20000.

2-3-4. Loop like this until s is the final state. Get the trained Q neural network.

2-4. Construct the scene. The construction scenario refers to randomly constructing an arbitrary initial link state, and randomly constructing an arbitrary group of flows and rate requirements.

2-5. Build the SumTree storage experience and mark the priority. The SumTree refers to a binary tree data structure, and the marking priority refers to marking the importance of experience according to TD-error, and the TD-error refers to the Q value of the current experience in the timing difference and the target Q value The absolute value of the difference between values.

2-6. According to the priority selection experience, according to the priority experience playback DQN improved algorithm to train the Q neural network.

Furthermore, the training process of the above-mentioned priority experience playback DQN improved algorithm can be described as follows in the form of pseudo-code:

Algorithm input: number of training rounds MAX_EPISODE, number of links M, number of streams to be allocated N, state feature dimension S, action set X, learning rate learning-rate, priority adjustment strength α, sampling weight coefficient β, discount rate γ, Exploration rate ∈, minimal positive value ε, current Q network Q, target Q network Q′, batch gradient descent sample number BT, target Q network parameter update frequency C, and SumTree leaf node number ST.

Output: Q network parameters.

For i from 1 to MAX_EPISODE do

Initialize all states and actions corresponding to the value Q

Randomly initialize all parameters ω of the current Q network

Initialize the parameter ω′=ω of the target Q network Q′

Initialize the default data structure of experience playback SumTree, the priority p _k of all S leaf nodes of SumTree is 1

Initialize S as the first state of the current state sequence, that is, the load status of all links in the current SDN data center network and the information of the flow to be processed, and obtain its feature vector φ(S)

For j from 1 to N do

a) Use φ(S) as input in the Q network to get the Q value output corresponding to all actions of the Q network. Use the ∈-greedy method to select the corresponding action A in the current Q value output, that is, the rate allocated by the jth flow

b) Execute the current action A in the state S, and get the eigenvector φ(S′) corresponding to the new state S′

以及终止状态done，当拥塞发生时，即认为到达终止状态，停止当前episode的训练c) Calculate the reward value reward _m according to the state of each link,

And the termination state done, when congestion occurs, it is considered to have reached the termination state, and the training of the current episode is stopped

d) Store the five-tuple {φ(S), A, R, φ(S′), done} into SumTree

e) S=S'

损失函数权重ω_k＝(N*P(k))^-β/max_j(ω_j)，计算当前目标Q值y_k：f) Sample BT samples from SumTree {φ(S _k ), A _k , R _k ,φ(S′ _k ), done _k }, k=1,2,...,BT, the probability of each sample being sampled yes

Loss function weight ω _k =(N*P(k)) ^-β /max _j (ω _j ), calculate the current target Q value y _k :

通过神经网络的梯度反向传播来更新Q网络的所有参数ωg) Use the mean square error loss function

Update all parameters of the Q-network ω through the gradient back-propagation of the neural network

h) Recalculate the TD error of all samples δ _k =y _k -Q(φ(S _k ),A _k ,ω), and update the priorities of all nodes in SumTree p _k =|δ _k |+ε

i) If i%C=1, then update the target Q network parameter ω′=ω

j) If S' is in the terminated state, that is, all streams are allocated, stop the current episode.

Step 3: Apply the Q neural network trained in step 2 to control the congestion of the SDN data center network.

The specific congestion control method flow chart is shown in Figure 4, which specifically includes the following steps:

3-1. Obtain the rate requirements and routing information of the 28 streams to be allocated, and obtain the status of each link in the current SDN data center network, that is, the link bandwidth occupancy. There are 28 stream requests to be allocated, and the initial load of the 8 links is [21, 27, 20, 28, 22, 26, 23, 18]. The specific links and bandwidth requirements to be occupied are as follows:

Table 1

flow1 flow2 flow3 flow4 ... flow27 flow28 occupied link l ₁ , l ₂ l ₁ ,l ₃ l ₁ , l ₄ l ₁ ,l ₅ ... l ₆ ,l ₈ l ₇ ,l ₈ Required Bandwidth (G) 5 5 5 5 ... 5 5

3-2. Select one stream from all N streams, input the information of this stream and the current link state into the Q neural network trained by the improved algorithm of priority experience playback DQN, and select a stream with The action with the optimal Q value is executed.

3-3. Update the current link state, and record the mapping information between the current flow and the allocated rate at the same time.

3-4. Determine whether all the 28 streams have been allocated: if not, return to step 3-2 and continue the cycle until the allocation rate for all streams; if the allocation is complete, execute step 3-5;

3-5. Output the flow rate allocation mapping table of 28 streams, and the SDN controller allocates the rate for each stream. In this way, the allocated rate can not only meet the rate requirements of all streams, but also effectively avoid congestion, so as to achieve data center The purpose of global congestion control. The flow rate allocation mapping table of 28 flows is as follows:

Table 2

flow1 flow2 flow3 flow4 flow5 ... flow27 flow28 occupied link l ₁ ,l ₂ l ₁ ,l ₃ l ₁ , l ₄ l ₁ ,l ₅ l ₂ ,l ₃ ... l ₆ ,l ₈ l ₇ ,l ₈ Required Bandwidth (G) 5 5 5 5 5 ... 5 5 Allocate Bandwidth (G) 3 3 4 1 5 ... 4 3

Fig. 5 shows a diagram of the bandwidth variation of each link for each allocation. The abscissa indicates the number of allocations, and the ordinate indicates the bandwidth occupancy of each link after the bandwidth is allocated for each flow. It can be seen from Figure 5 that after the rate allocation of 28 streams is completed, no congestion occurs on all links. It shows that the method of the present invention can effectively implement congestion control.

Fig. 6 shows a comparison chart of link utilization ratios of different methods under different flow numbers. Wherein DQN is the algorithm of the original single-objective network, DDQN is to change the single-objective neural network into a double neural network on the basis of DQN, and PRIO is an improved DQN algorithm that has applied priority experience playback in the present invention. It can be seen from Figure 6 that when the initial state of the entire network is relatively saturated and the rate requirement of each flow is large, the DQN algorithm based on priority experience playback has the largest link utilization in the three network states rate, and with the increase of the network state dimension, the advantages are more obvious. Therefore, compared with the DQN algorithm and the DDQN algorithm, the DQN algorithm based on priority experience replay proposed by the present invention has the best congestion control effect and the highest link utilization rate, that is, it can try its best to meet the flow rate requirements in the network.

Figure 7 shows the comparison of the convergence speed of different methods. It can be seen from FIG. 7 that the congestion control method of the improved DQN algorithm based on priority experience replay proposed by the present invention is obviously superior to the DQN algorithm and the DDQN algorithm in terms of convergence speed and convergence effect.

The congestion control method of the present invention has been described above in conjunction with specific embodiments. The embodiment shows that a SDN data center congestion control method based on priority experience playback DQN proposed by the present invention is effective. In this method, the controller allocates the rate globally to the flow of the whole network, which can not only avoid the congestion of the whole network, but also make the utilization rate of the data link of the network as high as possible, so as to realize the congestion control of the whole data center. The present invention overcomes the shortcoming of weak multi-dimensional perception ability of reinforcement learning, uses Q neural network instead of Q table of reinforcement learning to speed up the algorithm convergence speed, uses the target network and experience replay to improve the performance of the algorithm, and uses the priority experience replay mechanism to solve the algorithm The difficulty of obtaining high-quality samples makes it have better convergence speed and convergence effect.

The above-mentioned embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. Various changes and modifications can be made by those skilled in the relevant technical fields without departing from the spirit and scope of the present invention. Therefore, all technical solutions obtained by means of equivalent replacement or equivalent transformation fall within the protection scope of the present invention.

Claims (6)

1. a kind of SDN data center congestion control method based on priority experience playback DQN, is characterized in that, comprises the steps: S1. Deploy the congestion control agent based on the deep Q network in the SDN controller, and introduce the priority experience playback DQN algorithm into the data center based on the software-defined network; S2. Training the deep Q network, wherein the training process includes S21～S24: S21. Set the input of the deep Q network as link state information and flow state information, and the output is the Q value corresponding to different actions. The different actions indicate that different rates are allocated to the flow, and the reward function is to balance link utilization and A comprehensive function of link congestion; S22. Randomly construct any initial link state and any set of flow and rate requirements to realize scene construction; S23. Construct a SumTree for storing experience, and mark the priority of experience; S24. Select experience from SumTree according to the priority, and play back the DQN improved algorithm to train the deep Q network according to the priority experience, so that the SDN controller can maximize the data link utilization rate through the deep Q network while ensuring that the data center is not congested. ; Wherein, the improved priority experience playback DQN algorithm is based on the priority experience playback DQN algorithm, adding a judgment on whether the link is congested before the end of each step of each scene of network training, and if it is congested, then directly end the scene , if there is no congestion, continue to the next step; the scenario indicates that the rate is allocated for the entire group of flows, and each step in the scenario indicates that the rate is allocated for a flow; S3. The SDN controller collects the link state information and the flow state information of the rate to be allocated in real time from the SDN data plane, and inputs it into the trained deep Q network, determines the optimal action according to the Q value of each flow and generates the flow Rate allocation scheme, so as to perform global congestion control on the SDN data center network. 2. The SDN data center congestion control method based on priority experience playback DQN as claimed in claim 1, wherein the SDN controller is connected to the network equipment of the SDN data plane through a southbound interface to realize centralized control. 3. the SDN data center congestion control method based on priority experience playback DQN as claimed in claim 1, is characterized in that, described reward function is:

In the formula: reward _m represents the reward value, min represents the operation of taking the minimum value, and LkCap _m represents the link utilization rate. 4. The SDN data center congestion control method based on priority experience playback DQN according to claim 1, wherein the priority experience playback DQN algorithm is a DQN algorithm that replaces the experience playback mechanism with priority experience playback. 5. the SDN data center congestion control method based on priority experience playback DQN as claimed in claim 1, is characterized in that, the mark of described priority is the empirical importance mark that determines according to TD-error, and described TD-error is Refers to the absolute value of the difference between the current empirical Q value and the target Q value in the temporal difference. 6. the SDN data center congestion control method based on priority experience playback DQN as claimed in claim 1, is characterized in that, in described S3, the method that SDN controller carries out global congestion control comprises the following steps: S31. Obtain the rate requirements and routing information of the N streams currently to be allocated from the SDN data plane, and at the same time obtain the status of each link in the current SDN data center network, that is, the link bandwidth occupancy; S32. Select a flow from the current N flows to be allocated, input the information of this flow and the current link state into the depth Q network trained by S2, and select the optimal action to execute according to the output of the depth Q network; S33. Update the current link state, and record the mapping information between the current flow and the allocation rate; S34, judging whether all the N streams have been allocated, if not, then return to continue looping S32 and S33 until the rate is allocated for all streams; if all the streams are allocated, then execute S35; S35. Output the flow rate allocation mapping table of the N flows, and the SDN controller allocates a rate for each flow using the mapping table.

Images