CN115037667B - Fine-grained network situation awareness and source routing intelligent optimization method and device - Google Patents
Fine-grained network situation awareness and source routing intelligent optimization method and device Download PDFInfo
- Publication number
- CN115037667B CN115037667B CN202210953069.0A CN202210953069A CN115037667B CN 115037667 B CN115037667 B CN 115037667B CN 202210953069 A CN202210953069 A CN 202210953069A CN 115037667 B CN115037667 B CN 115037667B
- Authority
- CN
- China
- Prior art keywords
- network
- data packet
- target
- transmitted
- routing path
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/34—Source routing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/12—Shortest path evaluation
- H04L45/121—Shortest path evaluation by minimising delays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/14—Routing performance; Theoretical aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Abstract
The invention provides a method and a device for intelligently optimizing fine-grained network situation awareness and source routing, which relate to the technical field of communication and comprise the following steps: acquiring state information of all network equipment in a target network and attribute information of a data packet to be transmitted, and processing the state information and the attribute information by using a deep reinforcement learning algorithm to obtain a routing path of the data packet to be transmitted. In the deep reinforcement learning algorithm, the reward of each selectable routing path is the sum of the rewards of each network device on the selectable routing path, and the reward of each network device is the opposite number of the weighted sum of the node time delay of the network device and the maximum link utilization rate of the target network, so that the method can calculate the optimal routing path under the condition of minimizing the maximum link utilization rate and the path time delay, thereby not causing a large amount of data flows to be accumulated in the same path, avoiding the problem of network congestion, ensuring the timeliness of data packet transmission and further improving the overall network transmission performance.
Description
Technical Field
The invention relates to the technical field of communication, in particular to a method and a device for intelligently optimizing fine-grained network situation awareness and source routing.
Background
Based on the above measurement methods, the traditional network management and control technology stays at the level of coarse-grained sensing and control of network equipment and network basic resources, when a network has sudden conditions such as network congestion and single-point failure, a centralized control unit cannot perform timely and effective control rule adaptation and issuance quickly according to information changes of underlying network resources and network equipment, and even manual adjustment is needed, even if a network administrator can quickly respond, the network can generate extremely high burst processing and response time, which can cause the transmission performance of the network to be seriously reduced, so that the transmission of data packets cannot be completed on time as required.
Disclosure of Invention
The invention aims to provide a fine-grained network situation awareness and source routing intelligent optimization method and device, which avoid the problem of network congestion, ensure the timeliness of data packet transmission and further improve the overall network transmission performance.
In a first aspect, the present invention provides a fine-grained network situation awareness and source routing intelligent optimization method, including: acquiring state information of all network equipment in a target network and attribute information of a data packet to be transmitted; wherein the attribute information includes: a source IP address and a destination IP address; processing the state information and the attribute information by using a deep reinforcement learning algorithm to obtain a target routing path of the data packet to be transmitted; the network device where the data packet to be transmitted is located is the state of the agent in the deep reinforcement learning algorithm, and the port number of the next hop network device of the data packet to be transmitted is the action of the agent; the reward of each selectable routing path is the sum of the rewards of each network device on the selectable routing paths, and the reward of each network device is the inverse number of the weighted sum of the node time delay of the network device and the maximum link utilization rate of the target network.
In an optional embodiment, obtaining the status information of all network devices in the target network includes: transmitting a plurality of data packets in the target network based on an in-band telemetry method; wherein a routing path of the plurality of data packets may cover all network devices in the target network; receiving INT metadata of all network equipment in a routing path to which the previous hop network equipment of each data packet receiving end belongs, wherein the INT metadata is fed back by the previous hop network equipment of each data packet receiving end; determining status information for all network devices within the target network based on all the INT metadata.
In an alternative embodiment, the INT metadata of each of the network devices includes: the serial number of the network equipment, the node time delay of the network equipment and the byte number of the data packet to be transmitted by the network equipment.
In an alternative embodiment, the maximum link utilization of the target network represents a maximum of link utilizations of all network devices in the target network; wherein, the link utilization of the target network device is calculated by the following formula:
(ii) a Wherein the content of the first and second substances,
representing the link utilization rate of the target network equipment, B representing the byte number of the data packet to be transmitted by the target network equipment,
representing a node latency of the target network device, the target network device representing any network device in the target network.
In an alternative embodiment, the reward for each alternative routing path is expressed as:
(ii) a Wherein the content of the first and second substances,
indicating the node delay of the ith network device on the alternative routing path,
the node delay correction factor is represented by a time delay correction factor,
representing the total number of network devices on the alternative routing path,
representing a maximum link utilization for the target network.
In an optional embodiment, after obtaining the target routing path of the data packet to be transmitted, the method further includes: and issuing the target routing path to an initial switching node of the data packet to be transmitted in a source routing manner, so that the data packet to be transmitted is forwarded in the target network in the source routing manner according to the target routing path.
In a second aspect, the present invention provides an intelligent fine-grained network situational awareness and source routing optimization apparatus, including: the acquisition module is used for acquiring the state information of all network equipment in a target network and the attribute information of the data packet to be transmitted; wherein the attribute information includes: a source IP address and a destination IP address; the processing module is used for processing the state information and the attribute information by utilizing a deep reinforcement learning algorithm to obtain a target routing path of the data packet to be transmitted; the network device where the data packet to be transmitted is located is the state of the agent in the deep reinforcement learning algorithm, and the port number of the next hop network device of the data packet to be transmitted is the action of the agent; the reward of each selectable routing path is the sum of the rewards of each network device on the selectable routing path, and the reward of each network device is the inverse number of the weighted sum of the node time delay of the network device and the maximum link utilization rate of the target network.
In an alternative embodiment, the obtaining module includes: a transmitting unit for transmitting a plurality of data packets in the target network based on an in-band telemetry method; wherein a routing path of the plurality of data packets may cover all network devices in the target network; the receiving unit is used for receiving INT metadata of all network equipment in a routing path to which the previous hop of network equipment of each data packet receiving end belongs, wherein the INT metadata is fed back by the previous hop of network equipment; a determining unit for determining status information of all network devices within the target network based on all the INT metadata.
In a third aspect, the present invention provides an electronic device, including a memory and a processor, where the memory stores a computer program operable on the processor, and the processor implements the steps of the fine-grained network situational awareness and source routing intelligent optimization method described in any one of the foregoing embodiments when executing the computer program.
In a fourth aspect, the present invention provides a computer-readable storage medium, which stores computer instructions, and when executed by a processor, the computer instructions implement the fine-grained network situational awareness and source routing intelligence optimization method of any one of the foregoing embodiments.
The invention provides a fine-grained network situation awareness and source routing intelligent optimization method, which comprises the following steps: acquiring state information of all network equipment in a target network and attribute information of a data packet to be transmitted; wherein the attribute information includes: a source IP address and a destination IP address; processing the state information and the attribute information by utilizing a deep reinforcement learning algorithm to obtain a target routing path of the data packet to be transmitted; the network equipment where the data packet to be transmitted is located is the state of the intelligent agent in the deep reinforcement learning algorithm, and the port number of the next hop network equipment of the data packet to be transmitted is the action of the intelligent agent; the reward of each selectable routing path is the sum of the rewards of each network device on the selectable routing path, and the reward of each network device is the inverse of the weighted sum of the node delay of the network device and the maximum link utilization rate of the target network.
The invention provides a fine-grained network situation awareness and source routing intelligent optimization method, which comprises the steps of firstly obtaining state information of all network devices in a target network and attribute information of a data packet to be transmitted, then processing the state information and the attribute information by using a deep reinforcement learning algorithm to obtain a routing path of the data packet to be transmitted, and calculating an optimal routing path under the condition of minimizing the maximum link utilization rate and the path delay because the reward of each selectable routing path is the sum of the rewards of each network device on the selectable routing path and the reward of each network device is the weighted sum of the node delay of the network device and the maximum link utilization rate of the target network.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a fine-grained network situation awareness and source routing intelligent optimization method according to an embodiment of the present invention;
fig. 2 is a schematic network management and control diagram of an SDN architecture according to an embodiment of the present invention;
FIG. 3 is a flow chart of the basic process of an in-band telemetry technique provided by an embodiment of the present invention;
fig. 4 is a schematic diagram of a data packet transmission system according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a packet header format according to an embodiment of the present invention;
fig. 6 is a schematic diagram illustrating a format of a source routing packet header in a data packet header according to an embodiment of the present invention;
fig. 7 is a flowchart of a packet forwarding based on a source routing protocol according to an embodiment of the present invention;
fig. 8 is a flow chart of packet transmission according to an embodiment of the present invention;
fig. 9 is a functional block diagram of an intelligent fine-grained network situation awareness and source routing optimization apparatus according to an embodiment of the present invention;
fig. 10 is a schematic diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.
Some embodiments of the invention are described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
The traditional network measurement methods include Ping protocol, IP measurement protocol (IPMP), MPLS packet loss/delay measurement protocol, etc., which belong to out-of-band based network measurement. The out-of-band is that the management and control information of the network and the information carrying the user service are transmitted in different logical channels, so that the management and control information actively sends a special protocol data packet To the network To count the network information, which may cause extra traffic detection overhead, and at the same Time, only coarse-grained network performance indexes such as packet loss rate, delay, TTL (Time To Live), and the like may be measured.
Based on the above measurement methods, the conventional network management and control technology stays at the level of coarse-grained sensing and control of network devices and network basic resources, when a network has an emergency situation such as network congestion and single-point failure, a centralized control unit cannot perform timely and effective control rule adaptation and issuance quickly according to information changes of underlying network resources and network devices, even if manual adjustment and correction are needed, even if a network administrator can quickly respond, the network can generate extremely high emergency processing and response time, which can cause the transmission performance of the network to be seriously reduced, so that the transmission of data packets cannot be completed on time as required. In view of the above, embodiments of the present invention provide a fine-grained network situation awareness and source routing intelligent optimization method, so as to alleviate the technical problems mentioned above.
Example one
Fig. 1 is a flowchart of a fine-grained network situation awareness and source routing intelligent optimization method provided by an embodiment of the present invention, and as shown in fig. 1, the method specifically includes the following steps:
step S102, acquiring state information of all network devices in the target network and attribute information of the data packet to be transmitted.
The method provided by the embodiment of the invention is based on an SDN (Software Defined Network) Network architecture and a programmable data plane technology to realize Network control on a target Network, wherein the Network control refers to fine-grained and differentiated management and control on each equipment terminal in the Network, and most of common Network control schemes are realized based on a centralized controller under an SDN Network structure. The network management and control technology based on the SDN architecture can be mainly divided into the work of a data plane and a control plane, and the specific flow is shown in fig. 2. Firstly, a data plane needs to collect state information of equipment in a network and uploads the state information to a controller of a control plane through a southbound interface; and then the controller designs a control strategy according to the information of the data plane and issues a flow table to change the forwarding behavior of the data plane, so that a network control mode integrating high perception and quick response is realized, and the network performance is improved.
Therefore, to transmit the data packet to be transmitted through the target network, not only the attribute information of the data packet to be transmitted but also the status information of all network devices in the target network need to be obtained. The attribute information of the data packet to be transmitted includes: a source IP address and a destination IP address. In the embodiment of the invention. The data plane under the SDN network architecture is responsible for information collection, and the control plane is responsible for determining a routing strategy.
And step S104, processing the state information and the attribute information by using a deep reinforcement learning algorithm to obtain a target routing path of the data packet to be transmitted.
The traditional network management and control means has low controllability on a data plane, and the forwarding rule of the data stream is difficult to change according to the network traffic distribution condition; and the way of performing route optimization only for delay may cause a large amount of data flows to be accumulated on the same path, thereby causing network congestion. In order to solve the above problem, the method provided in the embodiment of the present invention uses a deep reinforcement learning algorithm to process the state information and the attribute information, so as to determine a target routing path of a data packet to be transmitted.
The network equipment where the data packet to be transmitted is located is the state of the intelligent agent in the deep reinforcement learning algorithm, and the port number of the next hop network equipment of the data packet to be transmitted is the action of the intelligent agent; the reward of each selectable routing path is the sum of the rewards of each network device on the selectable routing path, and the reward of each network device is the inverse of the weighted sum of the node delay of the network device and the maximum link utilization rate of the target network.
Specifically, when a data packet to be transmitted is located in each switch and each host in a target network, the data packet is regarded as that an agent is in one state, a source host is in an initial state, and a destination host is a destination; the action of the agent is set as the port number of the next hop network device of the data packet to be transmitted, namely the port number of the switch, the action space is discrete, and the action selects which number value to represent that the port is taken as the output port to transmit the data packet.
The deep reinforcement learning algorithm needs to maximize rewards, and the solution provided by the embodiment of the present invention aims to find an optimal route in the case of minimizing the maximum link utilization and the path delay, so the embodiment of the present invention uses the link utilization of the network and the path delay of the data packet together as reward values, and sets the reward of each network device as the inverse number of the weighted sum of the node delay of the network device and the maximum link utilization of the target network. Further, in order to maximize the reward, in the deep reinforcement learning algorithm, when the next state of the data packet to be transmitted reaches the host except the destination host (destination IP address), the obtained reward is negative, and when the next state of the data packet to be transmitted reaches the destination host, the obtained reward is positive.
That is to say, with the method provided by the embodiment of the present invention, an optimal route can be calculated while minimizing the maximum link utilization and the path delay, and then a controller of the SDN network architecture issues a flow table to deploy an optimal route forwarding rule, which can manage and control the forwarding of network traffic, optimize the routing manner of data flows, and prevent data flows that need to be route optimized from being stacked on the same path, thereby improving the overall network performance.
The invention provides a fine-grained network situation awareness and source routing intelligent optimization method, which comprises the steps of firstly obtaining state information of all network devices in a target network and attribute information of a data packet to be transmitted, then processing the state information and the attribute information by using a deep reinforcement learning algorithm to obtain a routing path of the data packet to be transmitted, and calculating an optimal routing path under the condition of minimizing the maximum link utilization rate and the path delay because the reward of each selectable routing path is the sum of the rewards of each network device on the selectable routing path and the reward of each network device is the weighted sum of the node delay of the network device and the maximum link utilization rate of the target network.
With the rise of software-defined networks, besides the out-of-band telemetry technology, another kind of network measurement technology appears, which obtains information of network internal devices directly from the periphery of the network through an SDN controller, and this way can obtain the perception of global state, but since a large amount of data exchange needs to be performed between the controller and the switching device, the transmission of network state information is performed between the controller and the network device, and thus, a large south-north overhead is generated.
In-band Telemetry (INT) is an emerging Network monitoring technology, and In-band refers to transmission of Network control information and information carrying user services through the same logical channel. The basic idea is to record and add network equipment information passed by a data packet to a packet head bit hop by hop, so as to form equipment related information on a source node-destination node complete path at an end point, thereby realizing fine-grained network sensing capability.
Specifically, fig. 3 is a basic processing flow diagram of an in-band telemetry according to an embodiment of the present invention, and as shown in fig. 3, a sending end sends out a data packet, and adds an INT header and first INT metadata to an INT source end (a network device that can insert the INT header into the data packet). The required device state information INT Metadata (i.e., INT Metadata, which is network state information to be collected) is inserted into the packet header bits of the data packet by parsing the INT packet header every time one switch passes. And transmitting all the acquired INT Metadata to the controller in the previous hop of the data packet receiving end, and simultaneously recovering the data packet to be in the initial state.
In view of this, in an optional implementation manner, in the step S102, the obtaining of the state information of all network devices in the target network specifically includes the following steps:
step S1021, transmitting a plurality of data packets in the target network based on the in-band telemetry method.
Wherein the routing path of the plurality of data packets may cover all network devices in the target network.
Step S1022, the INT metadata of all network devices in the routing path to which the previous hop of network device of each data packet receiving end belongs, which is fed back by the previous hop of network device of each data packet receiving end, is received.
In step S1023, the status information of all network devices within the target network is determined based on all INT metadata.
Based on the above description of the solution, in the embodiment of the present invention, the data plane of the SDN architecture employs an in-band telemetry technology to collect state information of all networks in the target network. The network information measurement by adopting the in-band telemetry mode can fully utilize the data packet which is already transmitted in the network, and when the data packet passes through one network device, the state information related to the network device is added on the data packet. And extracting the newly added network state information at the previous hop of the data packet transmitted to the destination IP address, and sending the newly added network state information to the controller. The in-band telemetry method is equivalent to the fact that the sensing capability of the network is expanded on the basis of the basic transmission capability of the data packet, and is a high-efficiency and low-overhead network management and control method.
Therefore, only a plurality of data packets need to be transmitted in the target network, and it is ensured that the routing paths of the plurality of data packets can cover all network devices in the target network, that is, after all the data packets are transmitted to the corresponding target IP addresses, INT metadata of all network devices in the routing path to which the previous hop network device of each data packet receiving end belongs, which is fed back by the previous hop network device of each data packet receiving end, is obtained, and a union of the INT metadata of all network devices in the routing path is the state information of all network devices in the target network.
The existing telemetry method adopts Ping protocol, IP measurement protocol (IPMP), MPLS packet loss/delay measurement protocol and the like, adopts a new protocol to send a detection packet, can only carry out coarse-grained measurement, and increases extra traffic detection overhead based on an out-of-band telemetry mode. And the in-band telemetry can insert the telemetry information into the normally forwarded data packet, does not cause extra detection overhead, and can realize fine-grained measurement at the level of the data packet.
In an alternative embodiment, the INT metadata of each network device includes: the serial number of the network equipment, the node time delay of the network equipment and the byte number of the data packet to be transmitted by the network equipment.
In the target network, in order to distinguish network devices and quickly locate the specified network devices, each network device in the target network is provided with a unique number, namely a swid, and optionally, the swid occupies 32 bits. In addition, it is known that a delay exists when a network device transmits a data packet, and the quality of a routing path is related to the transmission delay of the data packet, so that a node delay (hop _ delay) is also used as data to collect, wherein the unit of the node delay is us, and occupies 32 bits. In the embodiment of the present invention, it is known that, in addition to the time delay, the link utilization of the network device is used as a part of the routing path reward, and the number of bytes of the to-be-transmitted data packet currently processed by the network device is associated with the link utilization of the network device, so the INT metadata further includes the number of bytes byte _ cnt of the to-be-transmitted data packet of the network device, and optionally, the byte _ cnt occupies 32 bits.
In an alternative embodiment, the maximum link utilization of the target network represents the maximum of the link utilizations of all network devices in the target network.
Wherein, the link utilization of the target network device is calculated by the following formula:
(ii) a Wherein, the first and the second end of the pipe are connected with each other,
the link utilization rate of the target network equipment is shown, B shows the byte number of the data packet to be transmitted by the target network equipment,
representing the node delay of the target network device, which represents any network device in the target network.
In the embodiment of the present invention, after the INT metadata of each network device in the target network is acquired, the state information of each network device is determined, and the link utilization rate corresponding to any data transmission port of any network device can be further calculated by using the acquired INT metadata and the formula of the link utilization rate. Assume that there are 5 network devices in the target network, and the link utilization rates are: 60%,80%,20%,50%,30%, then the maximum link utilization of the target network is 80%.
In an alternative embodiment, the reward for each alternative routing path is expressed as:
(ii) a Wherein the content of the first and second substances,
indicating the node delay of the ith network device on the alternative routing path,
the node delay correction coefficient is represented, and can be set to be a proper value according to the link utilization rate, so that the node delay and the link utilization rate have the same effect on route optimization;
representing the total number of network devices on the alternative routing path,
representing the maximum link utilization of the target network.
In the embodiment of the invention, the input of the deep reinforcement learning algorithm is in-band telemetering information acquired by a data plane, and the algorithm aims to find a path to maximize the reward. After the target routing path of the data packet to be transmitted is determined, the egress port of each hop switch which the path passes through can be determined.
Reinforcement Learning is a branch of machine Learning, and the greatest characteristic of reinforcement Learning is Learning from Interaction. And the intelligent agent continuously learns knowledge according to the acquired reward or punishment in the interaction with the environment so as to adapt to the environment. The motivation for deep reinforcement learning is that the storage space of the traditional reinforcement learning Q-learning algorithm is limited, and for a large number of states in a complex environment, a Q table (reward value table) which can store an oversized state space and represent the state quality cannot be constructed. Therefore, a neural network is introduced to form a deep Q network. The deep reinforcement learning technology is continuously improved, and nearly ten classical algorithms including DQN, DDPG, TRPO, A2C, ACER and PPO are realized. The deep reinforcement learning algorithm is divided into three learning frames: value function (Value Based), policy Gradient (Policy Gradient), and Actor-Critic (Actor-Critic). The Actor-Critic reinforcement learning algorithm integrates a value function and a strategy gradient, and a strategy is guided to be updated by using a value function error, so that the learning speed is accelerated.
The embodiment of the invention does not limit the specific algorithm form of the deep reinforcement learning algorithm, a user can select the algorithm according to actual requirements, and the mechanism of the PPO (proximity Policy Optimization) near-end strategy Optimization algorithm is summarized below. In general, PPO is an Actor-Critic structure, critic adopts a dominant function to evaluate the quality of actions, and strategy
Gradient by expectation of reward
Update, the expected reward gradient is:
wherein a represents an action, s represents a current state,
indicating the reward for action a taken for the s-state under policy pi.
As PPO is an on-policy algorithm, in order to utilize original sample data and improve the sample utilization rate, importance sampling and old strategy are introduced
To sample, we can get:
。
further, due to
Thus, it is possible to obtain:
. The optimization objective function corresponding to the gradient is:
。
in the PPO algorithm, the goal of the Critic network is to minimize the loss function, i.e. minimize the estimation error; the goal of Actor is to maximize
. The Actor network will be in the old policy
On the basis of A, a new strategy is modified according to A
. When A is large, the modification amplitude is large, and a new strategy is enabled
More likely to occur. In practical application, the CLIP method can be used for limiting the strategy updating amplitude and ensuring the training stability, namely,
wherein, in the step (A),
the ratio of the old and new policies is shown,
representing a clipping magnitude hyperparameter.
In view of the fact that the specific contents of the state, the action and the reward of the agent in the deep reinforcement learning algorithm are defined, the optimal route of the data package to be transmitted, namely the target route path of the data package to be transmitted, can be rapidly calculated by applying the PPO algorithm.
In the conventional routing method, the network switching device determines a forwarding path hop by hop according to the destination of the data packet, it is difficult to remove the fault position in time when the network fails, so as to replan the route, and the controllability of the forwarding path of the data packet is poor, so that it is difficult to deploy a network management and control mechanism. In addition, under the SDN network architecture, although the controller can issue the flow table to change the forwarding behavior of the data plane switching device, applying this method to route optimization may cause expensive north-south traffic overhead because the controller needs to frequently issue the flow table to each switching device to send a control instruction.
In view of this, in an optional embodiment, after obtaining the destination routing path of the data packet to be transmitted, the method of the present invention further includes the following steps:
and transmitting the target routing path to an initial switching node of the data packet to be transmitted in a source routing manner, so that the data packet to be transmitted is transmitted in the target network in the source routing manner according to the target routing path.
Specifically, the source routing technology allows a route for forwarding a data packet in a network to be specified at a data packet sending end, that is, the source route can customize a data packet forwarding path, so that not only can network failure removal be realized more quickly, but also the source routing technology can be used for network route optimization. By adopting a source routing mode, the controller can realize the forwarding behavior of the data packet in the network only by sending the flow table to the data packet sending end, thereby greatly reducing the north-south overhead.
Fig. 4 is a schematic diagram of a frame of a data packet transmission system according to an embodiment of the present invention, where a control plane needs to deploy a default forwarding path for each switching device initially to implement network device interconnection, and after the control plane calculates an optimal route based on a deep reinforcement learning algorithm, the control plane issues a flow table to an initial switching device (i.e., an initial switching node) of a data packet to be sent, so that the data packet to be sent is forwarded according to an optimized routing path in a source routing manner, thereby implementing fine-grained network sensing and routing optimization, and improving data flow forwarding performance.
Because the data plane in the SDN architecture needs to implement an in-band telemetry mechanism and a source routing mechanism, the embodiment of the present invention designs a packet header of a data packet to a format as shown in fig. 5 based on a programmable data plane, and the INT packet header is used for storing multi-hop INT information; the IP packet header is used for storing the IP information of the next hop node; a UDP (User Datagram Protocol) header, that is, a User Datagram Protocol header. By means of the programmable data plane, the advantage of the data packet processing logic can be customized, and the source routing protocol directly stores the forwarding ports of all switches (network equipment) on the target routing path in the source routing packet header; the source routing header consists of forwarding port ports of multiple switches and bos for indicating whether the switch is the last hop switch. The source routing header format is shown in fig. 6.
Referring to fig. 7, a source routing packet header of a first hop switching device includes n bos and port information, which indicates that a data packet needs to be forwarded through the n hop switching devices (n =3 in the example of fig. 7), because each time the data packet passes through one hop switching device, one piece of bos and port information needs to be analyzed, and the piece of source routing information is deleted after the analysis. And a bos indicates whether the current data packet is the last hop, and if the current data packet is the last hop (bos = 1), the data packet is recovered to be a normal data packet after leaving the current switching equipment.
To facilitate understanding of the technical solution of the present invention, the following description illustrates that, as shown in fig. 8, the data plane collects INT information by using an in-band telemetry technique and uploads the INT information to the control plane. The control plane calculates an optimal route based on a route optimization module of a deep reinforcement learning algorithm PPO, and then the controller sends the optimized routing rule down the flow table to the initial switching device S1 of the data flow, and a source routing packet header is inserted into the flow table, so that each hop can analyze the source routing packet header and forward the routing packet header according to the source routing rule. In fig. 8, ETH denotes an ethernet packet header, and the remaining types of packet headers, except for the source routing packet header, are not shown in fig. 8. S1, sending a data packet by taking a port as 1 according to first source routing information, and deleting the source routing information after sending the data packet; s2, obtaining first source routing information according to the current data packet, sending the data packet by taking a port as 2, and deleting the source routing information after sending the data packet; s3 is the same, but since the bos value is 1 at this time, the source route is the last hop, and therefore, after forwarding according to port 3, the packet is restored to a normal packet, and then, the packet is analyzed according to the normal packet format.
Therefore, by using the scheme of the invention, fine-grained in-network sensing can be realized, and the PPO algorithm can calculate the optimal route according to the sensing information under the condition of minimizing the maximum link utilization rate and the path delay. Compared with the method for optimizing the route only aiming at the single target with the shortest time delay, the embodiment of the invention can obtain the optimal route and can not cause a large amount of data streams to be accumulated in the same path. The forwarding rule of the data flow is updated by using a source routing mode, and finally the transmission performance of the data flow is optimized with low north-south overhead.
In summary, the embodiment of the present invention provides a technical solution that combines in-band telemetry with a network management and control technology based on an SDN architecture, where the in-band telemetry does not cause extra detection overhead by inserting information into a data packet normally forwarded in a network, and achieves fine-grained measurement at the level of the data packet; the controller issues a flow table according to the information to implement a corresponding route optimization strategy, so that the performance of the network is improved; furthermore, the embodiment of the invention provides the route optimization based on the deep reinforcement learning algorithm PPO and the source route, firstly, the PPO algorithm can show the advantages of an intelligent decision mode under the condition of considering a plurality of optimization targets, and compared with a method that a controller issues a flow table updating forwarding rule to each switching device, the source route mode can greatly reduce the cost in the north and south directions, so that the route optimization can be realized through smaller cost in the north and south directions.
Example two
The embodiment of the invention also provides an intelligent optimization device for sensing the situation of the fine-grained network and routing the source, which is mainly used for executing the intelligent optimization method for sensing the situation of the fine-grained network and routing the source provided by the embodiment of the invention.
Fig. 9 is a functional block diagram of an intelligent fine-grained network situation awareness and source routing optimization apparatus according to an embodiment of the present invention, and as shown in fig. 9, the apparatus mainly includes: an obtaining module 10 and a processing module 20, wherein:
an obtaining module 10, configured to obtain status information of all network devices in a target network and attribute information of a data packet to be transmitted; wherein the attribute information includes: a source IP address and a destination IP address.
And the processing module 20 is configured to process the state information and the attribute information by using a deep reinforcement learning algorithm to obtain a target routing path of the data packet to be transmitted.
The network equipment where the data packet to be transmitted is located is the state of the intelligent agent in the deep reinforcement learning algorithm, and the port number of the next-hop network equipment of the data packet to be transmitted is the action of the intelligent agent; the reward of each selectable routing path is the sum of the rewards of each network device on the selectable routing path, and the reward of each network device is the inverse of the weighted sum of the node delay of the network device and the maximum link utilization rate of the target network.
The invention provides a fine-grained network situation awareness and source route intelligent optimization method executed by a fine-grained network situation awareness and source route intelligent optimization device.
Optionally, the obtaining module 10 includes:
a transmitting unit for transmitting a plurality of data packets in a target network based on an in-band telemetry method; wherein the routing path of the plurality of data packets may cover all network devices in the target network.
And the receiving unit is used for receiving INT metadata of all network equipment in the routing path to which the previous hop of network equipment of each data packet receiving end belongs, wherein the INT metadata is fed back by the previous hop of network equipment.
A determining unit for determining status information of all network devices within the target network based on all INT metadata.
Optionally, the INT metadata of each network device comprises: the serial number of the network equipment, the node time delay of the network equipment and the byte number of the data packet to be transmitted by the network equipment.
Optionally, the maximum link utilization of the target network represents a maximum of link utilizations of all network devices in the target network.
Wherein, the link utilization of the target network device is calculated by the following formula:
(ii) a Wherein the content of the first and second substances,
the link utilization rate of the target network equipment is shown, B represents the byte number of the data packet to be transmitted by the target network equipment,
the node latency of the target network device is represented, and the target network device represents any network device in the target network.
Optionally, the reward for each alternative routing path is expressed as:
(ii) a Wherein the content of the first and second substances,
indicating the node delay of the ith network device on the alternative routing path,
the node delay correction factor is represented by a time delay correction factor,
representing the total number of network devices on the alternative routing path,
representing the maximum link utilization of the target network.
Optionally, the apparatus further comprises:
and the issuing module is used for issuing the target routing path to the initial switching node of the data packet to be transmitted in a source routing manner so as to forward the data packet to be transmitted in a target network in the source routing manner according to the target routing path.
EXAMPLE III
Referring to fig. 10, an embodiment of the present invention provides an electronic device, including: a processor 60, a memory 61, a bus 62 and a communication interface 63, wherein the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is arranged to execute executable modules, such as computer programs, stored in the memory 61.
The Memory 61 may include a Random Access Memory (RAM) and a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
The bus 62 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 10, but this does not indicate only one bus or one type of bus.
The memory 61 is configured to store a program, and the processor 60 executes the program after receiving an execution instruction, where the method performed by the apparatus defined by the process disclosed in any of the foregoing embodiments of the present invention may be applied to the processor 60, or implemented by the processor 60.
The processor 60 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 60. The Processor 60 may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in ram, flash, rom, prom, or eprom, registers, etc. as is well known in the art. The storage medium is located in a memory 61, and the processor 60 reads the information in the memory 61 and completes the steps of the method in combination with the hardware.
The computer program product of the fine-grained network situation awareness and source routing intelligent optimization method and apparatus provided by the embodiments of the present invention includes a computer-readable storage medium storing a non-volatile program code executable by a processor, where instructions included in the program code may be used to execute the method described in the foregoing method embodiments, and specific implementation may refer to the method embodiments, and will not be described herein again.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present invention or a part thereof which substantially contributes to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined or explained in subsequent figures.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings or the orientations or positional relationships that the products of the present invention are conventionally placed in use, and are only used for convenience in describing the present invention and simplifying the description, but do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," "third," and the like are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.
Furthermore, the terms "horizontal", "vertical", "overhang" and the like do not imply that the components are required to be absolutely horizontal or overhang, but may be slightly inclined. For example, "horizontal" merely means that the direction is more horizontal than "vertical" and does not mean that the structure must be perfectly horizontal, but may be slightly inclined.
In the description of the present invention, it should also be noted that, unless otherwise explicitly stated or limited, the terms "disposed," "mounted," "connected," and "connected" are to be construed broadly and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. A fine-grained network situation awareness and source routing intelligent optimization method is characterized by comprising the following steps:
acquiring state information of all network equipment in a target network and attribute information of a data packet to be transmitted; wherein the attribute information includes: a source IP address and a destination IP address; the status information of each of the network devices includes: the serial number of the network equipment, the node time delay of the network equipment and the byte number of a data packet to be transmitted by the network equipment;
processing the state information and the attribute information by using a deep reinforcement learning algorithm to obtain a target routing path of the data packet to be transmitted;
the network device where the data packet to be transmitted is located is the state of the agent in the deep reinforcement learning algorithm, and the port number of the next hop network device of the data packet to be transmitted is the action of the agent; the reward of each selectable routing path is the sum of the rewards of each network device on the selectable routing path, and the reward of each network device is the opposite number of the weighted sum of the node time delay of the network device and the maximum link utilization rate of the target network; the goal of the deep reinforcement learning algorithm is to find a path that maximizes the reward.
2. The fine-grained network situation awareness and source routing intelligent optimization method according to claim 1, wherein obtaining state information of all network devices in a target network comprises:
transmitting a plurality of data packets in the target network based on an in-band telemetry method INT; wherein a routing path of the plurality of data packets may cover all network devices in the target network;
receiving INT metadata of all network equipment in a routing path to which the previous hop network equipment of each data packet receiving end belongs, wherein the INT metadata is fed back by the previous hop network equipment of each data packet receiving end;
determining status information for all network devices within the target network based on all the INT metadata.
3. The fine-grained network situational awareness and source routing intelligent optimization method of claim 1,
the reward for each alternative routing path is expressed as:
(ii) a Wherein the content of the first and second substances,
indicating alternative routing pathsThe node delay of the last i-th network device,
the node delay correction factor is represented by a time delay correction factor,
representing the total number of network devices on the alternative routing path,
representing a maximum link utilization for the target network.
4. The fine-grained network situation awareness and source routing intelligent optimization method according to claim 1, further comprising, after obtaining the target routing path of the data packet to be transmitted:
and issuing the target routing path to an initial switching node of the data packet to be transmitted in a source routing manner, so that the data packet to be transmitted is forwarded in the target network in the source routing manner according to the target routing path.
5. A fine-grained network situation awareness and source routing intelligent optimization device is characterized by comprising:
the acquisition module is used for acquiring the state information of all network equipment in a target network and the attribute information of a data packet to be transmitted; wherein the attribute information includes: a source IP address and a destination IP address; the state information of each of the network devices includes: the serial number of the network equipment, the node time delay of the network equipment and the byte number of a data packet to be transmitted by the network equipment;
the processing module is used for processing the state information and the attribute information by utilizing a deep reinforcement learning algorithm to obtain a target routing path of the data packet to be transmitted;
the network device where the data packet to be transmitted is located is the state of the agent in the deep reinforcement learning algorithm, and the port number of the next hop network device of the data packet to be transmitted is the action of the agent; the reward of each selectable routing path is the sum of the rewards of each network device on the selectable routing path, and the reward of each network device is the opposite number of the weighted sum of the node time delay of the network device and the maximum link utilization rate of the target network; the goal of the deep reinforcement learning algorithm is to find a path that maximizes the reward.
6. The fine-grained network situation awareness and source routing intelligent optimization device according to claim 5, wherein the obtaining module comprises:
a transmitting unit for transmitting a plurality of data packets in the target network based on an in-band telemetry method INT; wherein a routing path of the plurality of data packets may cover all network devices in the target network;
the receiving unit is used for receiving INT metadata of all network equipment in a routing path to which the previous hop of network equipment of each data packet receiving end belongs, and the INT metadata is fed back by the previous hop of network equipment;
a determining unit for determining status information of all network devices within the target network based on all the INT metadata.
7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program operable on the processor, and wherein the processor when executing the computer program implements the steps of the fine-grained network situational awareness and source routing intelligence optimization method of any one of claims 1 to 4.
8. A computer readable storage medium, wherein the computer readable storage medium stores computer instructions, which when executed by a processor, implement the fine-grained network situational awareness and source routing intelligence optimization method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210953069.0A CN115037667B (en) | 2022-08-10 | 2022-08-10 | Fine-grained network situation awareness and source routing intelligent optimization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210953069.0A CN115037667B (en) | 2022-08-10 | 2022-08-10 | Fine-grained network situation awareness and source routing intelligent optimization method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115037667A CN115037667A (en) | 2022-09-09 |
| CN115037667B true CN115037667B (en) | 2022-11-08 |
Family
ID=83131303
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210953069.0A Active CN115037667B (en) | 2022-08-10 | 2022-08-10 | Fine-grained network situation awareness and source routing intelligent optimization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115037667B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117041139B (en) * | 2023-10-09 | 2024-03-26 | 腾讯科技（深圳）有限公司 | Data packet transmission method, device, computer equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1435033A (en) * | 2000-03-15 | 2003-08-06 | 英佛西莫信息技术股份有限公司 | Method and system for communication of data via optimum data path in network |
| CN109039942A (en) * | 2018-08-29 | 2018-12-18 | 南京优速网络科技有限公司 | A kind of Network Load Balance system and equalization methods based on deeply study |
| CN111010294A (en) * | 2019-11-28 | 2020-04-14 | 国网甘肃省电力公司电力科学研究院 | Electric power communication network routing method based on deep reinforcement learning |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8565117B2 (en) * | 2008-01-15 | 2013-10-22 | Alcatel Lucent | Systems and methods for network routing |
| US9137161B2 (en) * | 2013-05-29 | 2015-09-15 | Telefonaktiebolaget L M Ericsson (Publ) | Method and system of bandwidth-aware service placement for service chaining |
-
2022
- 2022-08-10 CN CN202210953069.0A patent/CN115037667B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1435033A (en) * | 2000-03-15 | 2003-08-06 | 英佛西莫信息技术股份有限公司 | Method and system for communication of data via optimum data path in network |
| CN109039942A (en) * | 2018-08-29 | 2018-12-18 | 南京优速网络科技有限公司 | A kind of Network Load Balance system and equalization methods based on deeply study |
| CN111010294A (en) * | 2019-11-28 | 2020-04-14 | 国网甘肃省电力公司电力科学研究院 | Electric power communication network routing method based on deep reinforcement learning |
Non-Patent Citations (1)
| Title |
|---|
| 基于MPLS的广域保护通信系统路由算法;熊小萍等;《电工技术学报》;20130630;第28卷(第6期);第3节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115037667A (en) | 2022-09-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7375068B2 (en) | Data packet detection methods, devices, and systems | |
| JP4774357B2 (en) | Statistical information collection system and statistical information collection device | |
| US8427958B2 (en) | Dynamic latency-based rerouting | |
| JP5660198B2 (en) | Network system and switching method | |
| US11496399B2 (en) | Dynamically balancing traffic in a fabric using telemetry data | |
| US10075371B2 (en) | Communication system, control apparatus, packet handling operation setting method, and program | |
| CN113676376B (en) | In-band network telemetry method based on clustering | |
| US20120201140A1 (en) | Network system, controller, method, and program | |
| US7944924B2 (en) | Handling of received implicit null packets | |
| EP2552059A1 (en) | Packet transfer system, control apparatus, transfer apparatus, method of creating processing rules, and program | |
| WO2011118575A1 (en) | Communication system, control device and traffic monitoring method | |
| CN113242177B (en) | Method and device for multi-path concurrent data transmission, storage medium and electronic equipment | |
| JP4823156B2 (en) | Remote traffic monitoring method | |
| CN115037667B (en) | Fine-grained network situation awareness and source routing intelligent optimization method and device | |
| US20180309647A1 (en) | Length Control For Packet Header Sampling | |
| KR101500251B1 (en) | Communication system, node, packet forwarding method and computer-readable recording medium recording a program | |
| US20210306267A1 (en) | Optimized network latency using in-band telemetry | |
| JP4382122B2 (en) | Relay device and bandwidth control program | |
| US9521066B2 (en) | vStack enhancements for path calculations | |
| US9270577B2 (en) | Selection of one of first and second links between first and second network devices | |
| CN114157595A (en) | Communication system, data processing method and related equipment | |
| WO2014006920A1 (en) | Communication system, control apparatus, communication method, and program | |
| CN110933002B (en) | Method and device for realizing switching chip of MPLS in-band detection OAM | |
| WO2023206165A1 (en) | Multicast data message sending method and apparatus, and device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |