CN118540272A - Congestion control method, device, equipment, medium and program product - Google Patents
Congestion control method, device, equipment, medium and program product Download PDFInfo
- Publication number
- CN118540272A CN118540272A CN202310202979.XA CN202310202979A CN118540272A CN 118540272 A CN118540272 A CN 118540272A CN 202310202979 A CN202310202979 A CN 202310202979A CN 118540272 A CN118540272 A CN 118540272A
- Authority
- CN
- China
- Prior art keywords
- message
- traffic
- sending end
- buffer queue
- data center
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 230000005540 biological transmission Effects 0.000 claims abstract description 101
- 238000004590 computer program Methods 0.000 claims description 19
- 238000012545 processing Methods 0.000 claims description 16
- 238000003860 storage Methods 0.000 claims description 12
- 238000004891 communication Methods 0.000 description 18
- 238000005516 engineering process Methods 0.000 description 14
- 238000010586 diagram Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 10
- 238000011217 control strategy Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000001360 synchronised effect Effects 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000012790 confirmation Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000002708 enhancing effect Effects 0.000 description 2
- 230000008713 feedback mechanism Effects 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides a congestion control method, a congestion control device, congestion control equipment, congestion control media and a congestion control program product. The method comprises the following steps: receiving a first flow message sent by a first sending end in a data center network; if the message buffer queue is congested and the first traffic message is an external traffic message of the data center network, or if the message buffer queue is congested and the first traffic message is an internal traffic message of the data center network, and meanwhile, no external traffic message of the data center network exists in the message buffer queue, discarding the first traffic message; if the message buffer queue is congested and the first traffic message is an internal traffic message of the data center network, and meanwhile, an external traffic message of the data center network exists in the message buffer queue, discarding the second traffic message in the message buffer queue, and inserting the first traffic message into the message buffer queue. To reduce packet loss of the internal flow of the DCN, thereby optimizing the transmission performance of the internal flow of the DCN.
Description
Technical Field
The embodiment of the application relates to the technical field of clouds, in particular to a congestion control method, a congestion control device, a congestion control equipment, a congestion control medium and a congestion control program product.
Background
The data center Network (DATA CENTER Network, DCN) is an important foundation for constructing the cloud service technology, and is an important condition for improving the business experience of clients and enhancing the competitiveness of enterprises. DCNs are typically formed by connecting a plurality of servers through a network, and communication between the servers is realized through a routing switch device inside the DCN, so that low latency, high throughput, packet loss sensitivity and the like are typical features of communication inside the DCN; meanwhile, the server inside the DCN may also directly serve the client through the internal network and the external wide area network (Wide Area Network, WAN), for example, the server inside the DCN may respond to the client request and send the traffic message to the client, in which case, high latency, low throughput, and relatively insensitive packet loss are important features of the cross-WAN communication.
The traffic message sent from the sending end inside the DCN to the receiving end inside the DCN may be referred to as an internal traffic message of the DCN. The traffic message sent from the sender inside the DCN to the receiver inside the WAN may be referred to as an external traffic message of the DCN. At present, when a routing exchange device in a DCN is congested, an indiscriminate congestion control mode is adopted for an internal flow message and an external flow message, namely, the internal flow message and the external flow message are discarded indiscriminately. When the route switching device adopts a non-differential congestion control mode, the receiving end of the internal flow message can quickly send a message confirmation message to the sending end to confirm whether the flow message is received or not due to smaller internal communication time delay of the DCN, and at the moment, the sending end quickly adjusts a sending window aiming at the internal flow so as to relieve the congestion pressure of the route switching device. However, because the communication delay from the DCN to the WAN is increased, the receiving end of the external traffic packet may not receive the message confirmation packet, and based on this, the sending end does not adopt a congestion control policy for the external traffic, which will cause the congestion condition to be mainly borne by the internal traffic of the DCN, and affect the performance of the internal traffic transmission of the DCN.
Disclosure of Invention
The application provides a congestion control method, a congestion control device, congestion control equipment, congestion control media and a congestion control program product. To reduce packet loss of the internal flow of the DCN, thereby optimizing the transmission performance of the internal flow of the DCN.
In a first aspect, an embodiment of the present application provides a congestion control method, where the method is applied to a routing switch device in a data center network, and the method includes: receiving a first flow message sent by a first sending end in a data center network; if the message buffer queue is congested and the first traffic message is an external traffic message of the data center network, or if the message buffer queue is congested and the first traffic message is an internal traffic message of the data center network, and meanwhile, no external traffic message of the data center network exists in the message buffer queue, discarding the first traffic message; if the message buffer queue is congested and the first traffic message is an internal traffic message of the data center network and an external traffic message of the data center network exists in the message buffer queue, discarding the second traffic message in the message buffer queue and inserting the first traffic message into the message buffer queue; the second flow message is an external flow message of the data center network in the message buffer queue.
In a second aspect, an embodiment of the present application provides a congestion control apparatus, where the apparatus is located inside a data center network, and the apparatus includes: a transceiver module and a processing module. The receiving and transmitting module is used for receiving a first flow message sent by a first sending end in the data center network; a processing module for: if the message buffer queue is congested and the first traffic message is an external traffic message of the data center network, or if the message buffer queue is congested and the first traffic message is an internal traffic message of the data center network, and meanwhile, no external traffic message of the data center network exists in the message buffer queue, discarding the first traffic message; if the message buffer queue is congested and the first traffic message is an internal traffic message of the data center network and an external traffic message of the data center network exists in the message buffer queue, discarding the second traffic message in the message buffer queue and inserting the first traffic message into the message buffer queue; the second flow message is an external flow message of the data center network in the message buffer queue.
In a third aspect, embodiments of the present application provide a computer-readable storage medium storing a computer program that causes a computer to perform a method as in the first aspect or implementations thereof.
In a fourth aspect, embodiments of the present application provide a computer program product comprising computer program instructions for causing a computer to perform the method as in the first aspect or implementations thereof.
In a fifth aspect, embodiments of the present application provide a computer program for causing a computer to perform the method as in the first aspect or implementations thereof.
By the technical scheme provided by the embodiment of the application, a differential congestion control strategy can be adopted for the internal flow messages and the external flow messages, namely when the message buffer queue of the routing switching equipment is congested, the priority of the DCN internal flow is ensured to be higher than that of the external flow crossing the WAN, and the DCN internal flow with high priority is supported to borrow the external flow crossing the WAN so as to reduce packet loss of the DCN internal flow, thereby optimizing the transmission performance of the DCN internal flow.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of DCN internal communication and WAN external communication;
Fig. 2 is a schematic diagram of an application scenario provided in an embodiment of the present application;
Fig. 3 is a flowchart of a congestion control method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a message buffer queue according to an embodiment of the present application;
Fig. 5 is a flowchart of another congestion control method according to an embodiment of the present application;
fig. 6 is a schematic diagram of a congestion control apparatus 600 according to an embodiment of the present application;
Fig. 7 is a schematic block diagram of an electronic device provided by an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Before introducing the technical scheme of the application, the technical terms related to the technical scheme of the application are described below:
1. Cloud technology (Cloud technology) refers to a hosting technology for integrating hardware, software, network and other series resources in a wide area network or a local area network to realize calculation, storage, processing and sharing of data. Cloud technology (Cloud technology) is based on the general terms of network technology, information technology, integration technology, management platform technology, application technology and the like applied by Cloud computing business models, and can form a resource pool, so that the Cloud computing business model is flexible and convenient as required. Cloud computing technology will become an important support. Background services of technical networking systems require a large amount of computing, storage resources, such as video websites, picture-like websites, and more portals. Along with the high development and application of the internet industry, each article possibly has an own identification mark in the future, the identification mark needs to be transmitted to a background system for logic processing, data with different levels can be processed separately, and various industry data needs strong system rear shield support and can be realized only through cloud computing.
2. DCN is an important foundation for constructing cloud service technology, and is an important condition for improving business experience of clients and enhancing enterprise competitiveness. DCNs are typically made up of multiple servers connected by a network, with communication between the servers being accomplished through routing switches within the DCN, low latency, high throughput, packet loss sensitivity, etc. being typical features of DCN internal communications.
3. WANs, also known as extranets, public networks, are remote networks that connect computers in different local or metropolitan area networks, principally using packet-switched technology, as opposed to the internet.
4. And an internal flow message is sent from a sending end in the DCN to a receiving end in the DCN.
5. And an external flow message is sent from a sending end in the DCN to a receiving end in the WAN.
6. The sending window refers to the number of flow messages that can be sent by the sending end at one time.
The technical problems and the inventive concept to be solved by the present application will be described below:
At present, when a routing switch device in a DCN is congested with a message, it adopts a non-differential congestion control manner for an internal traffic message and an external traffic message, so as to better understand the different effects of packet loss inside the DCN on transmission between the inside of the DCN (i.e. the traffic from a sender in the DCN to a receiver in the DCN) and the outside of the WAN (i.e. the traffic from the sender in the DCN to the receiver in the WAN), which is described below with reference to fig. 1:
fig. 1 is a schematic diagram of communication between the inside of the DCN and communication outside of the WAN, as shown in fig. 1, a DCN server (e.g. sending ends 1 and 2) may send a traffic message to a server (receiving ends 1 and 2) inside the DCN through a routing switch device inside the DCN, that is, R1 in fig. 1, and may also send a traffic message to a client (e.g. receiving ends 3 and 4) inside the WAN through a routing switch device inside the DCN and a routing switch device inside the WAN, where different connections in fig. 1 are labeled by different numbers, and one connection corresponds to one sending window. In this process, after receiving the traffic from the sending ends 1 and 2, the receiving ends 1 and 2 in the DCN may send these traffic messages to other user terminals. Assuming that the communication latency is less than 10 milliseconds (millisecond, ms) inside the DCN; in this case, if congestion packet loss occurs in the routing switch device R1, on one hand, the sending ends 1 and 2 react rapidly to the traffic in the DCN, for example, reduce the sending window, etc., so as to further alleviate the congestion condition at the R1; on the other hand, the sending ends 1 and 2 cannot timely receive the message acknowledgement messages from the receiving ends 3 and 4 in the WAN due to larger communication delay aiming at the external traffic of the WAN, and based on the fact, the sending ends cannot perform congestion control on the external traffic sent to the receiving ends 3 and 4. The traffic transmission performance inside the DCN is reduced, but the traffic transmission performance outside the WAN is not effectively reduced, instead, the sending end misuses the packet loss condition that no external traffic appears at present, and further increases the external traffic transmission, and further causes the congestion condition to be mainly borne by the traffic inside the DCN, and influences the traffic transmission performance inside the DCN.
In order to solve the technical problem, the application provides that a differential congestion control strategy can be adopted for the internal flow message and the external flow message, namely the priority of the internal flow message is improved so as to reduce packet loss of the internal flow of the DCN, thereby optimizing the transmission performance of the internal flow of the DCN.
Fig. 2 is a schematic diagram of an application scenario provided in an embodiment of the present application, as shown in fig. 2, each server 210 in the DCN is connected through a network formed by a plurality of routing switch devices, and communication between the servers 210 in the DCN can be implemented through the routing switch devices, where communication between any two servers 210 in the DCN is implemented through the routing switch device 220. In addition, the server 210 inside the DCN may also communicate with the client 230 inside the WAN through the routing switch 220 and the routing switch inside the WAN.
In some implementations, the server 210 may be a stand-alone physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content delivery network (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.
In some implementations, the routing switch 220 may be a router, a switch, or the like, but is not limited thereto.
In some implementations, the client 230 may be, but is not limited to, a smart phone, tablet, notebook, desktop, smart speaker, smart watch, etc.
It should be understood that fig. 2 illustrates three servers and one client, and that the application scenario may alternatively include other numbers of servers and clients, which are not limited by the embodiment of the present application. Each server may be connected to at least one client, so as to send a traffic message to a corresponding client through the connection. In other words, each server may be used as a sending end of the traffic message, or may be used as a receiving end of the traffic message, and each client may be used as a receiving end of the traffic message.
The technical scheme of the application will be described in detail as follows:
Fig. 3 is a flowchart of a congestion control method according to an embodiment of the present application, where the method is applied to a routing switch device in a data center network, for example, may be the routing switch device 220 in fig. 2, and as shown in fig. 3, the method may include:
S301: the routing switching equipment receives a first flow message sent by a first sending end in a data center network;
s302: the route switching equipment detects whether congestion exists in the message buffer queue; if the message buffer queue has no congestion, executing S303, and if the message buffer queue has congestion, executing S304;
s303: the route switching equipment normally forwards the first flow message;
S304: the route switching device detects whether the first traffic message is an internal traffic message of the data center network; if the first traffic message is an internal traffic message of the data center network, executing S305; if the first traffic message is not an internal traffic message of the data center network, that is, the first traffic message is an external traffic message of the data center network, S308 is executed;
S305: the route switching equipment detects whether an external flow message of the data center network exists in the message buffer queue; if there is an external traffic message of the data center network in the message buffer queue, executing S306; if no external traffic message of the data center network exists in the message buffer queue, executing S308;
s306: the routing switching equipment discards a second flow message in the message buffer queue, wherein the second flow message is an external flow message of the data center network in the message buffer queue;
S307: the route switching equipment inserts the first flow message into a message buffer queue;
s308: the routing switch discards the first traffic message.
S301 is described in detail below:
Let it be assumed that the receiving end of the first traffic message is referred to as a first receiving end. In order to facilitate understanding the situations of the first transmitting end and the first receiving end, the following describes the two devices in conjunction with the application scenario shown in fig. 2: for example, in the application scenario shown in fig. 2, the first sending end may be any one of the servers 210 in fig. 2, and the receiving end of the first traffic packet may be another server 210 different from the first sending end in fig. 2, or any one of the clients 230 in fig. 2.
The sending manner of the first sending end to the first flow message may include: passive transmission mode and active transmission mode, but not limited thereto:
In some implementations, the first receiving end may first send a traffic request message to the first sending end, based on which the first sending end responds to the traffic request message to send the first traffic message to the first receiving end.
In other implementations, the first receiving end may actively send the first traffic message to the first sending end.
In some implementations, the first traffic message includes at least one of: an internet protocol (Internet Protocol, IP) address, port number, IP address of the first transmitting end, port number, other data information, etc. of the first receiving end.
It should be understood that, in the embodiment of the present application, the traffic message may also be referred to as a message, and the message may be any form, and any content of the message, which is not limited in this embodiment of the present application.
S302 is described in detail below:
the routing switch device may detect whether the message buffer queue is congested by adopting the following implementation manners, but is not limited to the following:
In some implementations, the routing switch device may detect whether the message buffer queue is full, if so, indicate that there is congestion in the message buffer queue, and if not, indicate that there is no congestion in the message buffer queue.
For example, assuming that the message buffer queue can carry at most 20 traffic messages, when the routing switch device detects that the current message buffer queue has carried 20 traffic messages, it indicates that congestion exists in the message buffer queue, and if the message buffer queue is smaller than 20 traffic messages, it indicates that congestion does not exist in the message buffer queue.
In other implementations, the routing switch device may detect whether the carrying capacity in the packet buffer queue reaches a preset carrying capacity, if so, indicate that the packet buffer queue has congestion, and if not, indicate that the packet buffer queue has no congestion.
In some realizable modes, the bearing capacity in the message buffer queue can be the ratio between the current bearing flow message quantity in the message buffer queue and the maximum bearing capacity of the message buffer queue, and can also be the current bearing flow message quantity in the message buffer queue; correspondingly, the preset bearing capacity can be a preset ratio or the number of preset flow messages.
In some implementations, the preset load may be 80%,85%,90%,95%, or the like, or the preset load may be 18, 19, or the like, and in any case, if the load in the message buffer queue reaches the preset load, the message buffer queue is about to be fully loaded.
For example, it is assumed that the message buffer queue can carry at most 20 traffic messages, and the number of traffic messages currently carried in the message buffer queue is 10, and the preset carrying capacity is 80%, and since 10/20<80%, it indicates that there is no congestion in the message buffer queue.
For example, it is assumed that the message buffer queue can carry at most 20 traffic messages, and the number of traffic messages currently carried in the message buffer queue is 19, and the preset carrying capacity is 18, and because 19>18, it indicates that there is congestion in the message buffer queue.
S304 is described in detail below:
the routing switch device may detect whether the first traffic packet is an internal traffic packet of the data center network by, but is not limited to, the following implementation manner:
In one implementation manner, the routing switch device detects whether the first traffic packet carries a fourth identifier for identifying that the first traffic packet is an internal traffic packet of the data center network, so as to detect whether the first traffic packet is an internal traffic packet of the data center network.
It should be understood that the premise of the first implementation is that: any one of the flow messages may or may not carry a fourth identifier for identifying that the first flow message is an internal flow message of the data center network. Based on this, if the fourth identifier is carried in the first traffic packet, it indicates that the first traffic packet is an internal traffic packet of the data center network, and if the fourth identifier is not carried in the first traffic packet, it indicates that the first traffic packet is not an internal traffic packet of the data center network, that is, the first traffic packet is an external traffic packet of the data center network.
In some implementations, for any one traffic message, a bit may be set for the fourth identifier, and the bit may be 1 bit (bit) in length and may be located in the extension field of the traffic message header. When the bit is not empty, it is indicated that it carries the fourth identification, and when the bit is empty, it is indicated that it does not carry the fourth identification.
In a second implementation manner, the routing exchange device detects the value of the fifth identifier in the first flow message; and detecting whether the first traffic message is an internal traffic message of the data center network based on the value of the fifth identifier.
In some implementations, for any one traffic message, a bit may be set for the fifth identifier, where the bit may be 1bit in length and may be located in the extension field of the traffic message header.
In some implementations, if the value of the fifth identifier in the first traffic packet is 1, it indicates that the first traffic packet is an internal traffic packet of the data center network; if the value of the fifth identifier in the first traffic message is 0, it indicates that the first traffic message is not an internal traffic message of the data center network, that is, the first traffic message is an external traffic message of the data center network.
In other implementations, if the value of the fifth identifier in the first traffic packet is 0, it indicates that the first traffic packet is an internal traffic packet of the data center network; if the value of the fifth identifier in the first traffic message is 1, it indicates that the first traffic message is not an internal traffic message of the data center network, that is, the first traffic message is an external traffic message of the data center network.
In a third implementation manner, the routing switch device detects whether the first traffic packet carries a sixth identifier for identifying that the first traffic packet is an external traffic packet of the data center network, so as to detect whether the first traffic packet is an external traffic packet of the data center network, and further determine whether the first traffic packet is an internal traffic packet of the data center network.
It should be understood that the precondition for the third implementation is: any one of the flow messages may or may not carry a sixth identifier for identifying that the first flow message is an external flow message of the data center network. Based on this, if the sixth identifier is carried in the first traffic packet, it indicates that the first traffic packet is an external traffic packet of the data center network, and if the sixth identifier is not carried in the first traffic packet, it indicates that the first traffic packet is not an external traffic packet of the data center network, that is, the first traffic packet is an internal traffic packet of the data center network.
In some implementations, for any one traffic message, a bit may be set for the sixth identifier, where the bit may be 1bit in length and may be located in an extended field of the traffic message header. When the bit is not empty, it is indicated that it carries the sixth identification, and when the bit is empty, it is indicated that it does not carry the sixth identification.
In a fourth implementation manner, the routing switch device detects the value of the seventh identifier in the first flow packet; and detecting whether the first flow message is an external flow message of the data center network based on the value of the seventh identifier, and further determining whether the first flow message is an internal flow message of the data center network.
In some implementations, for any one traffic message, a bit may be set for the seventh identifier, where the bit may be 1bit in length and may be located in an extended field of the traffic message header.
In some implementations, if the value of the seventh identifier in the first traffic packet is 1, it indicates that the first traffic packet is an external traffic packet of the data center network; if the value of the seventh identifier in the first traffic message is 0, it indicates that the first traffic message is not an external traffic message of the data center network, that is, the first traffic message is an internal traffic message of the data center network.
In other implementations, if the value of the seventh identifier in the first traffic packet is 0, it indicates that the first traffic packet is an external traffic packet of the data center network; if the value of the seventh identifier in the first traffic message is 1, it indicates that the first traffic message is not an external traffic message of the data center network, that is, the first traffic message is an internal traffic message of the data center network.
S305 is described in detail below:
the routing switch device may detect whether an external traffic message of the data center network exists in the message buffer queue in the following manner, but is not limited thereto:
In some implementations, the routing switch device may scan each traffic message in the message buffer queue and detect whether the scanned traffic message is an external traffic message of the data center network. The detection manner of detecting whether the flow message is the external flow message of the data center network is similar to the detection manner of detecting whether the flow message is the internal flow message of the data center network in S304, and the embodiment of the present application will not be repeated.
In some implementations, the route switch device may start scanning from the tail of the packet buffer queue until the head of the packet buffer queue is scanned, but the scanning manner of the route switch device for the packet buffer queue is not limited thereto.
S306 and S307 are described in detail below:
the routing switch device may determine the second traffic message by, but is not limited to, the following realizable means:
In one implementation manner, the route switching device starts scanning from the tail of the message buffer queue; and taking the external flow message of the first scanned data center network as a second flow message.
In the second implementation manner, the route switching device starts scanning from the opposite ends of the message buffer queue; and taking the external flow message of the first scanned data center network as a second flow message, or taking the external flow message of the last scanned data center network as the second flow message.
In some implementations, the routing switch device may insert the first traffic message into the end of the message buffer queue after discarding the second traffic message.
For example, fig. 4 is a schematic diagram of a message buffer queue provided in an embodiment of the present application, as shown in fig. 4, a route switch device starts scanning from the tail of the message buffer queue, and an external traffic message of a data center network scanned by the first is a traffic message pointed by a downward arrow, where the route switch device may discard the traffic message and insert an internal traffic message to be inserted into the tail of the message buffer queue.
The embodiment of the application provides that a differential congestion control strategy can be adopted for an internal flow message and an external flow message, namely when a message buffer queue of a routing switching device is congested, the priority of the internal flow of the DCN is ensured to be higher than that of the external flow crossing the WAN, and the external flow crossing the WAN is borrowed by the internal flow of the DCN with high priority, so that the packet loss of the internal flow of the DCN is reduced, and the transmission performance of the internal flow of the DCN is optimized.
It should be understood that when the message buffer queue of the routing switch device is congested, the receiving end may send a message confirmation message to the sending end at present to feedback whether the traffic message is received. Further, if the sending end confirms that the receiving end does not receive the traffic message according to the message, that is, there is a packet loss condition, the sending end can adjust the congestion control policy, such as adjusting the sending window, and retransmit the packet loss. However, the feedback mechanism is slower, so that the sending end cannot adjust the congestion control strategy in time, or the congestion response speed of the sending end is slower, or the packet loss repair speed of the sending end is slower, thereby causing lower resource utilization rate of the sending end and lower traffic transmission rate.
In order to solve the technical problem, an embodiment of the present application proposes a packet loss and timely feedback mechanism, which will be described in detail below based on the previous embodiment:
as shown in fig. 5, the congestion control method further includes:
In some implementations, at S308: after the routing switch device discards the first traffic message, the congestion control method further includes:
s309: the route switching equipment sends a first congestion feedback message to a first sending end;
S310: the first transmitting end adjusts a transmitting window of the first transmitting end based on the first congestion message and retransmits the lost packet of the first transmitting end.
It should be understood that the first congestion packet is used by the first sender to adjust a sending window of the first sender, and retransmit a packet loss of the first sender.
In some implementations, the first congestion feedback message includes at least one of, but is not limited to: the message type, the first identifier, the first numerical value and the first message number of the first congestion feedback message. The first identifier is used for identifying whether congestion packet loss exists or not; the first value is the packet loss number of the first transmitting end in a first preset time period; the first message number is the message number of the packet lost by the first sending end in a first preset duration.
In some implementations, the message type of the first congestion feedback message may occupy 2 bits, but is not limited thereto. For example, the value of the message type field in the first congestion feedback message may be binary 00.
In some implementations, the first identifier may occupy 1bit, but is not limited thereto.
In some implementations, when the value of the first identifier is 1, it indicates that there is a congestion packet loss; when the value of the first identifier is 0, the condition that congestion packet loss does not exist is indicated.
In other realizations, when the value of the first identifier is 0, it indicates that there is a congestion packet loss; when the value of the first identifier is 1, the condition that congestion packet loss does not exist is indicated.
In some implementations, the routing switch device may count the packet loss of the first sending end within a period of time, for example, within 10ms from the time when the first traffic packet is received, and feed back the congestion packet loss to the first sending end through the first congestion feedback packet. Based on this, the first value may be an integer greater than or equal to 0.
In some implementations, the value of the first preset duration may be 10ms, 20ms, etc., but is not limited thereto.
In some implementations, the first value is fixed in length, such as, but not limited to, 8 bits. For example, the first value may take on the value 00000011.
In other implementations, the first value is variable in length. For example, if the first congestion feedback packet carries the first identifier and the first value at the same time, and the first identifier indicates that there is no congestion packet loss, the first value may occupy 1bit, and the value may be 0. If the first identifier indicates that there is a congestion packet loss, the first value may occupy 8 bits and may be 00000011.
It should be understood that the number of the first packet numbers is a first numerical value.
In some implementations, the first number is a fixed length, e.g., 80 bits, of the first message number. Wherein, the length of each first message code may be fixed, for example, 16 bits.
In some implementations, the first number of first message numbers is variable in length. For example, if the first congestion feedback packet carries the first identifier and the first number of first packet numbers at the same time, and the first identifier indicates that there is no congestion packet loss, the first number of first packet numbers may occupy 1bit, and the value may be 0. If the first identifier indicates that there is a congestion packet loss, the first packet number of the first value may occupy the first value by 16 bits.
The adjustment manner of the first transmitting end to the transmitting window of the first transmitting end may be as follows, but is not limited thereto:
In some implementations, the first transmitting end reduces a transmitting window of the first transmitting end to a first preset size.
It should be understood that, as described above, the routing switch device sends the first congestion feedback message to the first sending end after discarding the first traffic message. In other words, when the first sender receives the first congestion feedback packet, it indicates that there is a congestion packet loss, and in this case, no matter what information the first congestion feedback packet carries, the first sender may reduce the sending window of the first sender to a first preset size.
In some implementations, the first preset size may be half the size of the original transmission window of the first transmitting end, but is not limited thereto. For example, assume that the original transmission window of the first transmitting end can transmit 10 traffic messages at a time, and the transmission window adjusted by the first transmitting end can transmit 5 traffic messages at a time.
In some implementations, if the first congestion feedback message includes a first message number, the message retransmitted by the first transmitting end is a message of the first message number. For example, the first congestion feedback message includes two first message numbers, which are 00000001 and 00000011 respectively, and after the first sending end receives the first congestion feedback message, the first sending end parses the two first congestion feedback messages to obtain two first message numbers, and then retransmits the traffic messages corresponding to the two first message numbers.
In some implementations, if the first congestion feedback packet does not include the first packet number, the first transmitting end may send a request message for requesting to obtain the first packet number to the routing switch device after receiving the first congestion feedback packet, where the routing switch device responds to the request message and sends the first packet number to the first transmitting end.
In some implementations, S307: after the route switching device inserts the first flow message into the message buffer queue, the congestion control method further comprises the following steps:
S311: the route switching equipment sends a second congestion feedback message to the first sending end;
S312: the first sending end adjusts a sending window of the first sending end based on the second congestion feedback message and transmits a flow message to be transmitted.
It should be understood that the second congestion feedback message is used by the first sending end to adjust the sending window of the first sending end, and to transmit the traffic message to be transmitted.
In some implementations, the second congestion feedback message includes at least one of, but is not limited to: the message type, the second identifier, the second numerical value and the second message number of the second congestion feedback message. The second identifier is used for identifying whether the connection between the first sending end and the receiving end of the first flow message has the condition of borrowing other connection transmission resources within a second preset duration; the second value is the number of messages transmitted by the connection between the first transmitting end and the receiving end of the first flow message by using other connection transmission resources within a second preset duration; the second message number is a message number of a message transmitted by the connection between the first sending end and the receiving end of the first flow message by using other connection transmission resources within a second preset duration.
It should be understood that, in the embodiment of the present application, the process of deleting an external traffic message in the message buffer queue and inserting an internal traffic message may be referred to as "borrowing" the transmission resource of the connection in which the external traffic message is located by the connection in which the internal traffic message is located.
Let it be assumed that the receiving end of the first traffic message is referred to as a first receiving end.
In some implementations, the routing switch may count whether the connection between the first transmitting end and the first receiving end borrows transmission resources of other connections within a period of time, such as 10ms from a time of receiving the first traffic message.
In some implementations, the message type of the second congestion feedback message may occupy 2 bits, but is not limited thereto. For example, the value of the message type field in the second congestion feedback message may be binary 10.
In some implementations, the second identifier may occupy 1bit, but is not limited thereto.
In some realizable modes, when the value of the second identifier is 1, the connection between the first sending end and the first receiving end is represented to have the condition of borrowing other connection transmission resources in a second preset duration; when the value of the second identifier is 0, the connection between the first sending end and the first receiving end does not borrow other connection transmission resources within a second preset duration.
In other realizable modes, when the value of the second identifier is 0, the connection between the first sending end and the first receiving end is represented to have the condition of borrowing other connection transmission resources in a second preset duration; when the value of the second identifier is 1, the connection between the first sending end and the first receiving end does not borrow other connection transmission resources within a second preset duration.
In some implementations, the second preset duration may have a value of 10ms, 20ms, or the like, but is not limited thereto.
In some implementations, the second value is fixed in length, such as, but not limited to, 8 bits. For example, the second value may take a value of 00000011, which indicates that the connection between the first transmitting end and the first receiving end borrows other connection transmission resources within the second preset duration, and the number of the messages transmitted by the first transmitting end is 3, where 3 is a binary value corresponding to 00000011.
In other implementations, the second value may be variable in length. For example, if the second congestion feedback packet carries the second identifier and the second value at the same time, and the second identifier indicates that the connection between the first sending end and the first receiving end does not borrow other connection transmission resources within a second preset duration, the second value may occupy 1bit and may be 0. If the second identifier indicates that the connection between the first sending end and the first receiving end has a condition of borrowing other connection transmission resources in a second preset duration, the second value can occupy 8 bits and can be 00000011.
It should be understood that the number of the second packet numbers is a second value.
In some implementations, the second number of second message numbers is fixed in length, e.g., 80 bits. Wherein the length of each second message code may be fixed, for example 16 bits.
In some implementations, the second number of second message numbers is variable in length. For example, if the second congestion feedback packet carries the second identifier and the second number of second packets, and the second identifier indicates that the connection between the first sending end and the first receiving end does not borrow other connection transmission resources within a second preset duration, the second number of second packets may occupy 1bit, and the value may be 0. If the second identifier indicates that the connection between the first sending end and the first receiving end has a condition of borrowing other connection transmission resources in a second preset duration, the second number of second message numbers may occupy a second number of 16 bits.
The adjustment manner of the first transmitting end to the transmitting window of the first transmitting end may be as follows, but is not limited thereto:
in some implementations, the first transmitting end reduces the transmitting window of the first transmitting end to a second preset size.
It should be understood that, as described above, the routing switch device sends the second congestion feedback message to the first sending end after discarding the second traffic message and inserting the first traffic message into the message buffer queue. In other words, when the first sending end receives the second congestion feedback message, it indicates that there is a congestion packet loss, and in this case, no matter what information the second congestion feedback message carries, the first sending end may reduce the sending window of the first sending end to a second preset size.
In some implementations, the second preset size may be half the size of the original transmission window of the first transmitting end, but is not limited thereto. For example, assume that the original transmission window of the first transmitting end can transmit 10 traffic messages at a time, and the transmission window adjusted by the first transmitting end can transmit 5 traffic messages at a time.
It should be understood that, because the connection between the first sending end and the first receiving end borrows the transmission resources of other connections, in order to ensure that the traffic messages on other connections can be normally transmitted in the following, the first sending end may adjust the congestion control policy to restore the transmission resources thereof, and this elastic congestion control policy may fully utilize the network transmission resources, so as to ensure that the internal traffic of the DCN and the external traffic across the WAN can be better transmitted. The following will describe the elastic control policy executed by the first transmitting end:
In other implementations, if the second congestion feedback packet includes the second value, the adjustment manner of the transmission window of the first transmitting end is: firstly, reducing the sending window of the first sending end to a second preset size, and then subtracting a second value from the sending window of the first sending end.
It should be understood that, regarding the value of the second preset size, reference may be made to the above, and the embodiments of the present application will not be repeated.
For example, assuming that the original transmission window of the first transmitting end is 10 traffic messages capable of being transmitted at a time, the first transmitting end may first halve the transmission window, and the obtained transmission window is 5 traffic messages capable of being transmitted at a time, and further assuming that the second value is 2, the final obtained transmission window is 3 traffic messages capable of being transmitted at a time.
It should be understood that the traffic message to be transmitted in S312 refers to a traffic message after a message sent by a connection between the first sending end and the first receiving end by using transmission resources of other connections. For example, the number of the message sent by the connection between the first sending end and the first receiving end by using the transmission resources of other connections is 00000001, and then the traffic message to be transmitted may be 00000010.
In some implementations, S306: after the routing switch device discards the second traffic message in the message buffer queue, the congestion control method further includes:
s313: the route switching equipment sends a third congestion feedback message to a second sending end of the second traffic message;
S314: and the second sending end adjusts a sending window of the second sending end based on the third congestion feedback message and retransmits the lost packet of the second sending end.
It should be understood that, the third congestion feedback packet is used by the second sending end to adjust the sending window of the second sending end, and retransmit the packet loss of the second sending end.
In some implementations, the third congestion feedback message includes at least one of: the message type, the third identifier, the third numerical value and the third message number of the third congestion feedback message; the third identifier is used for identifying whether the connection between the second sending end and the receiving end of the second traffic message has the condition of borrowing transmission resources by other connections within a third preset duration; the third value is the number of packet loss of the second sending end caused by borrowing transmission resources by other connections in a third preset duration by the connection between the second sending end and the receiving end of the second traffic message; the third message number is a message number of a packet lost by the second sending end due to the fact that the connection between the second sending end and the receiving end of the second traffic message is borrowed by other connections for transmission resources within a third preset duration.
Let it be assumed that the receiving end of the second traffic message is referred to as a second receiving end.
In some implementations, the routing switch may count whether the connection between the second sending end and the second receiving end has borrowed transmission resources by other connections within a period of time, such as 10ms from a time of receiving the second traffic message.
In some implementations, the message type of the third congestion feedback message may occupy 2 bits, but is not limited thereto. For example, the value of the message type field in the third congestion feedback message may be binary 01.
In some implementations, the third identifier may occupy 1bit, but is not limited thereto.
In some implementations, when the value of the third identifier is 1, it indicates that the connection between the second sending end and the receiving end of the second traffic message has a situation that transmission resources are borrowed by other connections within a third preset duration; when the value of the third identifier is 0, the connection between the second sending end and the receiving end of the second traffic message does not exist in the third preset time period, and the situation that transmission resources are borrowed by other connections is indicated.
In other realizable modes, when the value of the third identifier is 0, the condition that the connection between the second sending end and the receiving end of the second traffic message is borrowed by other connections exists in a third preset time period is indicated; when the value of the third identifier is 1, the connection between the second sending end and the receiving end of the second traffic message does not exist in the third preset time period, and the situation that transmission resources are borrowed by other connections is indicated.
In some implementations, the third preset duration may have a value of 10ms, 20ms, or the like, but is not limited thereto.
In some implementations, the third value is fixed in length, such as, but not limited to, 8 bits. For example, the third value may take a value of 00000011, which indicates that the number of packet losses of the second transmitting end caused by borrowing transmission resources by other connections in the third preset duration by the connection between the second transmitting end and the second receiving end is 3, where 3 is a binary value corresponding to 00000011.
In other implementations, the third value may be variable in length. For example, if the third congestion feedback packet carries the third identifier and the third value at the same time, and the third identifier indicates that the connection between the second sending end and the second receiving end does not have a situation of borrowing transmission resources by other connections within a third preset duration, the third value may occupy 1bit and may have a value of 0. If the third identifier indicates that the connection between the second sending end and the receiving end of the second traffic message has a situation that transmission resources are borrowed by other connections within a third preset duration, the third value may occupy 8 bits and may be 00000011.
It should be understood that the number of the third packet numbers is a third numerical value.
In some implementations, the third number is fixed in length, for example 80 bits. Wherein the length of each third message code may be fixed, for example 16 bits.
In some implementations, the third number of third message numbers is variable in length. For example, if the third congestion feedback packet carries the third identifier and the third number of third packet numbers at the same time, and the third identifier indicates that the connection between the second sending end and the second receiving end does not have a condition of borrowing transmission resources by other connections within a third preset duration, the third number of third packet numbers may occupy 1bit, and the value may be 0. If the third identifier indicates that the connection between the second transmitting end and the second receiving end has a situation of borrowing transmission resources by other connections within a third preset duration, the third number of third packet numbers of the third numerical value may occupy a third number of 16 bits.
The adjustment manner of the second transmitting end to the transmitting window of the second transmitting end may be as follows, but is not limited thereto:
In some implementations, the second sender reduces the sending window of the second sender to a third preset size.
It should be understood that, as described above, the routing switch device sends the third congestion feedback packet to the second sender after discarding the second traffic packet. In other words, when the second sending end receives the third congestion feedback packet, it indicates that there is a congestion packet loss, and in this case, no matter what information the third congestion feedback packet carries, the second sending end may reduce the sending window of the second sending end to a third preset size.
In some implementations, the third preset size may be half the size of the original transmission window of the second transmitting end, but is not limited thereto. For example, assume that the original transmission window of the second transmitting end can transmit 10 traffic messages at a time, and the transmission window adjusted by the second transmitting end can transmit 5 traffic messages at a time.
It should be understood that, since the connection between the second sending end and the second receiving end is borrowed by other connections, in order to ensure that the traffic message on the connection between the second sending end and the second receiving end can be normally transmitted subsequently, the second sending end can adjust the congestion control policy to return to its transmission resource, and this elastic congestion control policy can fully utilize the network transmission resource, so as to ensure that the internal traffic of the DCN and the external traffic across the WAN can be better transmitted. The following will describe the elastic control policy executed by the second transmitting end:
In other implementations, if the third congestion feedback packet includes the third value, the adjustment manner of the transmission window of the second transmitting end is: firstly, reducing the sending window of the second sending end to a third preset size, and then adding a third numerical value to the sending window of the second sending end.
It should be understood that, regarding the value of the third preset size, reference may be made to the above, and this will not be repeated in the embodiments of the present application.
For example, assuming that the original transmission window of the second transmitting end is 10 traffic messages capable of being transmitted at a time, the second transmitting end may first halve the transmission window, and the obtained transmission window is 5 traffic messages capable of being transmitted at a time, and further assuming that the third value is 2, the final obtained transmission window is 7 traffic messages capable of being transmitted at a time.
In some implementations, if the third congestion feedback packet includes a third packet number, the packet retransmitted by the second transmitting end is a packet of the third packet number. For example, the third congestion feedback message includes two third message numbers, which are 00000001 and 00000011 respectively, and the second sending end, after receiving the third congestion feedback message, parses the two message numbers, and then can retransmit the traffic messages corresponding to the two message numbers.
In some implementations, after the first sending end receives the third congestion feedback packet, the first sending end may count the third number of 16 bits at the starting position corresponding to the third packet number, so as to parse all the third packet numbers. For example, assume that the format of the third congestion feedback message is: { message type, third identifier, third value, third message number of third congestion feedback message }, wherein, message type accounts for 2 bits, third identifier accounts for 1bit, third value accounts for 8 bits, and third value is 2, then from 12 th bit of third congestion feedback message, the first 16 bits are one third message code, the second 16 bits are another third message code.
The following is an exemplary illustration of an embodiment of the application by way of an example:
(1) After receiving the flow request message, the transmitting end transmits a flow message pkt_new to the corresponding receiving end;
In some implementations, it is assumed that the identification of the internal traffic messages used to identify the data center network is represented by pkt_in, where pkt_in occupies 1bit, located in the extended field of the traffic message header. In this case, the pkt_in corresponds to the fourth identifier above.
In other implementations, it is assumed that the identifier for identifying whether the internal traffic packet is the data center network is denoted by pkt_in, and when pkt_in is binary 1, it indicates that the traffic packet pkt_new is the internal traffic packet; when pkt_in is binary 0, it indicates that the flow packet pkt_new is an external flow packet. In this case, the pkt_in corresponds to the fifth identifier above.
(2) The route switching device detects the congestion condition of the message buffer queue, and if congestion does not occur (namely the buffer queue is not full or the state of being about to be fully loaded is not reached), the flow message pkt_new is forwarded to the next hop network device; if congestion occurs (i.e. the buffer queue is full or about to be full), performing differential processing according to the type of the flow message pkt_new and whether an external flow message exists in the message buffer queue, which is specifically as follows:
1) If the extension domain of the head of the flow message pkt_new does not have the pkt_in mark or the value of the extension domain is binary 0, executing the step (3);
2) If the extension domain of the head of the flow message pkt_new has a pkt_in identifier, or the value of the extension domain is binary 1, executing the step (5);
(3) The route switching device performs packet loss on the flow message pkt_new, and sends a congestion feedback message pkt_ cong _ack to a sending end of the flow message pkt_new, wherein the message carries the following information:
1) The message type ack_type of the congestion feedback message pkt_ cong _ack occupies 2 bits, and at this time, the ack_type is binary 00, and the format of pkt_con_ack is shown in formula (1):
Pkt_cong_ack = {ack_type, cong_mark, loss_amount, pkt_loss_num} (1)
Wherein, the cong _mark corresponds to the first identifier, the loss_value corresponds to the first value, and the pkt_loss_num corresponds to the first message number. I.e. cong _mark is used for identifying whether congestion packet loss exists or not; the loss_count is the number of lost packets of the sending end in a first preset duration; pkt_loss_num is the packet number of the packet lost by the sending end in the first preset duration.
2) In the formula (1), cong _mark occupies 1bit, and when cong _mark is binary 1, the congestion packet loss is shown to occur currently; when cong _mark is binary 0, it indicates that no congestion packet loss phenomenon occurs currently, and at this time, the values of loss_current and pkt_loss_num are both binary 0.
3) In equation (1), when cong _mark is binary 1, loss_amounts occupies 8 bits;
4) In formula (1), when cong _mark is binary 1 and loss_current is not 0, the pkt_loss_num field has a length of 16×loss_current;
5) After receiving the congestion feedback message pkt_ cong _ack from the routing switch device, the sending end of the flow message pkt_new executes packet loss retransmission operation, and the step (4) is specifically shown;
(4) The sending end of the flow message pkt_new adjusts congestion control parameters (such as halving a sending window) according to the received congestion feedback message pkt_ cong _ack and retransmits the flow message pkt_new;
(5) If the routing switch device detects that the message buffer queue is congested and the received traffic message pkt_new is an internal traffic message, the routing switch device needs to further detect whether the message buffer queue contains an external traffic message or not, and execute differential processing based on the detection result, which specifically includes:
1) If the message buffer queue is detected to contain the external flow message, executing the step (6);
2) If the fact that the message buffer queue does not contain the external flow message is detected, executing the step (3);
(6) The route switching device deletes an external traffic message from the message buffer queue and inserts a traffic message pkt_new, which is specifically as follows:
1) The route switching device starts to scan external flow messages from the tail part of the message buffer queue;
2) Positioning and scanning the position of an external flow message, and deleting the external flow message from the message buffer queue;
3) Inserting the flow message pkt_new into the tail of the message buffer queue;
4) As shown in fig. 4, the routing switch device detects that a message buffer queue has a congestion state, where the queue is from the tail to the head: internal, external, etc. traffic messages;
In the application, after the route switching device deletes the located external flow message (pkt_buffer), the flow message pkt_new is added to the tail part listed in the message buffer queue, and the process is called as borrowing of transmission resources;
5) The route switching equipment forwards the flow message in the latest message buffer queue to the next hop network node;
6) The message buffer queue returns a congestion feedback message pkt_ cong _ack to the transmitting end of the deleted external flow message pkt_buffer, and the step (7) is specifically described;
7) The route switching device returns a congestion feedback message pkt_ cong _ack to the sending end of the flow message pkt_new, and the step (9) is specifically shown;
(7) The route switching device returns a congestion feedback message pkt_ cong _ack to the sending end of the pkt_buffer message, wherein the message carries the following information:
1) Congestion feedback message type ack_type occupies 2 bits, where ack_type is binary 01, and the format of pkt_con_ack is shown in formula (2):
Pkt_cong_ack = {ack_type, lend, lend_amount, pkt_lend_num} (2)
Wherein, the length corresponds to the third identifier, the length_count corresponds to the third value, and the pkt_length_num corresponds to the third message number. Namely, lend is used for identifying whether the connection between the sending end and the receiving end of the pkt_buffer message has the condition of borrowing transmission resources by other connections within a third preset duration; the lend_current is the number of packet loss of the sending end caused by borrowing transmission resources by other connections in a third preset duration by the connection between the sending end and the receiving end of the pkt_buffer message; the pkt_length_num is a message number of a packet lost by the sending end due to the fact that the connection between the sending end and the receiving end of the pkt_buffer message is borrowed by other connections to transmit resources within a third preset duration.
2) In the formula (2), the identifier lend occupies 1bit, when the lend is binary 1, it indicates that the connection between the sending end and the receiving end of the pkt_buffer message has a situation that transmission resources are borrowed by other connections within a third preset duration, see steps 3) and 4) below for details; when the lend is binary 0, the connection between the sending end and the receiving end of the pkt_buffer message does not have the condition of borrowing transmission resources by other connections within a third preset duration, and at this time, the values of the lend_current and pkt_lend_num are both binary 0;
3) In formula (2), when lend is binary 1, the lend_count occupies 8 bits;
4) In formula (2), when lend is binary 1 and lend_count is non-0, the total length of pkt_lend_num is 16×lend_count, where 16 represents the length of each packet number;
5) In the application, after receiving congestion feedback message pkt_ cong _ack from route switching equipment, a sending end of pkt_buffer message executes packet loss retransmission operation, and sets corresponding congestion control parameters according to step (8);
(8) After receiving the congestion feedback message pkt_ cong _ack sent by the routing switch device, the sending end of the pkt_buffer message performs the following operations:
1) Judging the specific lost message number according to the information such as lend_current and pkt_lend_num;
2) According to the recalculation congestion control window cwnd_cpt; it should be noted that, in the congestion control algorithm based on packet loss, the value is reduced (e.g. halved) compared with the original window value;
3) Based on the computed cwnd_cpt, in combination with transmission resources "borrowed" by other connections, a final congestion window value cwnd_final is computed as shown in equation (3):
cwnd_final = cwnd_cpt + lend_amount (3)
4) The sender of the pkt_buffer message retransmits the lost messages based on the cwnd_final, wherein the messages are messages corresponding to the pkt_lend_num message numbers;
(9) The route switching device returns a congestion feedback message pkt_ cong _ack to the sending end of the flow message pkt_new, wherein the message carries the following information:
1) Congestion feedback message type ack_type occupies 2 bits, where ack_type is binary 10, and the format of pkt_con_ack is shown in formula (4):
Pkt_cong_ack = {ack_type, borrow, borrow_amount, pkt_borrow_num} (4)
Wherein borrow corresponds to the second identifier above, borrow _count corresponds to the second value above, and pkt_ borrow _num corresponds to the second message number above. Borrow is used for identifying whether the connection between the sending end and the receiving end of the flow message pkt_new is borrowed from other connection transmission resources or not within a second preset duration; borrow _current is the number of messages transmitted by the connection between the transmitting end and the receiving end of the flow message pkt_new by using other connection transmission resources within a second preset duration; pkt_ borrow _num is the message number of the message transmitted by the connection between the sending end and the receiving end of the flow message pkt_new by using other connection transmission resources within the second preset duration.
2) In the formula (4), borrow occupies 1bit, when borrow is binary 1, it indicates that the connection between the sending end and the receiving end of the flow packet pkt_new has a condition of borrowing other connection transmission resources in a second preset duration, see steps 3) and 4) below; when borrow is binary 0, it indicates that the connection between the sending end and the receiving end of the flow packet pkt_new does not borrow other connection transmission resources within the second preset duration, and at this time, both the borrow _current value and the pkt_ borrow _num value are binary 0;
3) In equation (4), borrow _count occupies 8 bits when borrow is binary 1;
4) In formula (4), when borrow is binary 1 and borrow _count is non-0, the total length of pkt_ borrow _num is 16×narrow_count, where 16 represents the length of each packet number;
5) In the application, after receiving congestion feedback message pkt_ cong _ack from route switching equipment, a sending end of flow message pkt_new executes packet loss retransmission operation, and sets corresponding congestion control parameters according to step (10);
(10) After receiving the congestion feedback message pkt_ cong _ack sent by the routing switch device, the sending end of the pkt_new message performs the following operations:
1) Calculating a congestion control window cwnd_cpt; it should be noted that, in the congestion control algorithm based on packet loss, the value is reduced (e.g. halved) compared with the original window value;
2) Based on the computed cwnd_cpt, a final congestion window value cwnd_final is computed in combination with "borrowing" the transmission resources of the other connection, as shown in formula (5):
cwnd_final = cwnd_cpt - borrow_amount (5)
3) The sending end of the flow message pkt_new transmits a subsequent message to be transmitted based on the cwnd_final.
In the embodiment of the application, the route switching equipment can feed back the congestion feedback message to the sending end, so that the sending end can timely adjust the congestion control strategy, or can improve the congestion response speed of the sending end, or can improve the packet loss repairing speed of the sending end, thereby improving the resource utilization rate of the sending end and further improving the traffic transmission rate.
Further, in the embodiment of the present application, the sending end may adopt an elastic congestion control policy, including: for a sender borrowing transmission resources of other connections, it may adjust congestion control policies to return the transmission resources. For a sender borrowed with other connection transmission resources, it may adjust the congestion control policy to return the transmission resources. The elastic congestion control strategy can fully utilize network transmission resources, and ensures that the internal traffic of the DCN and the external traffic crossing the WAN can be better transmitted.
The preferred embodiments of the present application have been described in detail above with reference to the accompanying drawings, but the present application is not limited to the specific details of the above embodiments, and various simple modifications can be made to the technical solution of the present application within the scope of the technical concept of the present application, and all the simple modifications belong to the protection scope of the present application. For example, the specific features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various possible combinations are not described further. As another example, any combination of the various embodiments of the present application may be made without departing from the spirit of the present application, which should also be regarded as the disclosure of the present application.
It should be further understood that, in the various method embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic of the processes, and should not constitute any limitation on the implementation process of the embodiments of the present application.
The method provided by the embodiment of the application is described above, and the device provided by the embodiment of the application is described below.
Fig. 6 is a schematic diagram of a congestion control apparatus 600 according to an embodiment of the present application, where the apparatus 600 is located inside a data center network, and as shown in fig. 6, the apparatus 600 includes: a transceiver module 610 and a processing module 620. The transceiver module 610 is configured to receive a first traffic message sent by a first sending end in the data center network; the processing module 620 is configured to: if the message buffer queue is congested and the first traffic message is an external traffic message of the data center network, or if the message buffer queue is congested and the first traffic message is an internal traffic message of the data center network, and meanwhile, no external traffic message of the data center network exists in the message buffer queue, discarding the first traffic message; if the message buffer queue is congested and the first traffic message is an internal traffic message of the data center network and an external traffic message of the data center network exists in the message buffer queue, discarding the second traffic message in the message buffer queue and inserting the first traffic message into the message buffer queue; the second flow message is an external flow message of the data center network in the message buffer queue.
In some implementations, the transceiver module 610 is further configured to send a first congestion feedback message to the first sending end after the processing module 620 discards the first traffic message; the first congestion message is used for the first sending end to adjust a sending window of the first sending end and retransmit lost packets of the first sending end.
In some implementations, the first congestion feedback message includes at least one of: the method comprises the steps of message type, first identification, first numerical value and first message number of a first congestion feedback message; the first identifier is used for identifying whether congestion packet loss exists or not; the first value is the packet loss number of the first transmitting end in a first preset time period; the first message number is the message number of the packet lost by the first sending end in a first preset duration.
In some implementations, the adjustment manner of the transmission window of the first transmitting end is: and reducing the sending window of the first sending end to a first preset size.
In some implementations, if the first congestion feedback message includes a first message number, the message retransmitted by the first transmitting end is a message of the first message number.
In some implementations, the transceiver module 610 is further configured to send a second congestion feedback message to the first sending end after the processing module 620 inserts the first traffic message into the message buffer queue; the second congestion feedback message is used for the first sending end to adjust the sending window of the first sending end and transmit the flow message to be transmitted.
In some implementations, the second congestion feedback message includes at least one of: the message type, the second identifier, the second numerical value and the second message number of the second congestion feedback message; the second identifier is used for identifying whether the connection between the first sending end and the receiving end of the first flow message has the condition of borrowing other connection transmission resources within a second preset duration; the second value is the number of messages transmitted by the connection between the first transmitting end and the receiving end of the first flow message by using other connection transmission resources within a second preset duration; the second message number is a message number of a message transmitted by the connection between the first sending end and the receiving end of the first flow message by using other connection transmission resources within a second preset duration.
In some implementations, if the second congestion feedback packet includes the second value, the transmission window adjustment manner of the first transmitting end is: firstly, reducing the sending window of the first sending end to a second preset size, and then subtracting a second value from the sending window of the first sending end.
In some implementations, the transceiver module 610 is further configured to send a third congestion feedback message to the second sending end of the second traffic message after the processing module 620 discards the second traffic message in the message buffer queue; the third congestion feedback message is used for the second sending end to adjust the sending window of the second sending end, and retransmits the packet loss of the second sending end.
In some implementations, the third congestion feedback message includes at least one of: the message type, the third identifier, the third numerical value and the third message number of the third congestion feedback message; the third identifier is used for identifying whether the connection between the second sending end and the receiving end of the second traffic message has the condition of borrowing transmission resources by other connections within a third preset duration; the third value is the number of packet loss of the second sending end caused by borrowing transmission resources by other connections in a third preset duration by the connection between the second sending end and the receiving end of the second traffic message; the third message number is a message number of a packet lost by the second sending end due to the fact that the connection between the second sending end and the receiving end of the second traffic message is borrowed by other connections for transmission resources within a third preset duration.
In some implementations, if the third congestion feedback packet includes the third value, the adjustment manner of the transmission window of the second transmitting end is: firstly, reducing the sending window of the second sending end to a third preset size, and then adding a third numerical value to the sending window of the second sending end.
In some implementations, if the third congestion feedback packet includes a third packet number, the packet retransmitted by the second transmitting end is a packet of the third packet number.
In some implementations, the processing module 620 is further to: detecting whether the first flow message carries a fourth identifier for identifying that the first flow message is an internal flow message of the data center network or not so as to detect whether the first flow message is the internal flow message of the data center network or not; or detecting the value of a fifth identifier in the first flow message; and detecting whether the first traffic message is an internal traffic message of the data center network based on the value of the fifth identifier.
In some implementations, the processing module 620 is further to: before discarding the second traffic message in the message buffer queue, starting scanning from the tail of the message buffer queue; and taking the external flow message of the first scanned data center network as a second flow message.
In some implementations, the processing module 620 is specifically configured to: and inserting the first flow message into the tail of the message buffer queue.
It should be understood that apparatus embodiments and method embodiments may correspond with each other and that similar descriptions may refer to the method embodiments. To avoid repetition, no further description is provided here. Specifically, the apparatus 600 shown in fig. 6 may perform the method embodiments corresponding to fig. 3 and 5, and the foregoing and other operations and/or functions of each module in the apparatus 600 are respectively for implementing the corresponding flows in each method in fig. 3 and 5, and are not described herein for brevity.
The apparatus 600 of the embodiment of the present application is described above in terms of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module may be implemented in hardware, or may be implemented by instructions in software, or may be implemented by a combination of hardware and software modules. Specifically, each step of the method embodiment in the embodiment of the present application may be implemented by an integrated logic circuit of hardware in a processor and/or an instruction in a software form, and the steps of the method disclosed in connection with the embodiment of the present application may be directly implemented as a hardware decoding processor or implemented by a combination of hardware and software modules in the decoding processor. Alternatively, the software modules may be located in a well-established storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, and the like. The storage medium is located in a memory, and the processor reads information in the memory, and in combination with hardware, performs the steps in the above method embodiments.
Fig. 7 is a schematic block diagram of an electronic device provided by an embodiment of the present application.
As shown in fig. 7, the electronic device may include:
A memory 710 and a processor 720, the memory 710 being configured to store a computer program and to transfer the program code to the processor 720. In other words, the processor 720 may call and run a computer program from the memory 710 to implement the method in the embodiment of the present application.
For example, the processor 720 may be configured to perform the above-described method embodiments according to instructions in the computer program.
In some embodiments of the application, the processor 720 may include, but is not limited to:
A general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field programmable gate array (Field Programmable GATE ARRAY, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like.
In some embodiments of the application, the memory 710 includes, but is not limited to:
Volatile memory and/or nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM) which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as static random access memory (STATIC RAM, SRAM), dynamic random access memory (DYNAMIC RAM, DRAM), synchronous Dynamic Random Access Memory (SDRAM), double data rate Synchronous dynamic random access memory (Double DATA RATE SDRAM, DDR SDRAM), enhanced Synchronous dynamic random access memory (ENHANCED SDRAM, ESDRAM), synchronous link dynamic random access memory (SYNCH LINK DRAM, SLDRAM), and Direct memory bus RAM (DR RAM).
In some embodiments of the application, the computer program may be partitioned into one or more modules that are stored in the memory 710 and executed by the processor 720 to perform the methods provided by the application. The one or more modules may be a series of computer program instruction segments capable of performing the specified functions, which are used to describe the execution of the computer program in the electronic device.
As shown in fig. 7, the electronic device further includes:
A transceiver 730, the transceiver 730 being connectable to the processor 720 or the memory 710.
The processor 720 may control the transceiver 730 to communicate with other devices, and in particular, may send information or data to other devices or receive information or data sent by other devices. Transceiver 730 may include a transmitter and a receiver. Transceiver 730 may further include antennas, the number of which may be one or more.
It will be appreciated that the various components in the electronic device are connected by a bus system that includes, in addition to a data bus, a power bus, a control bus, and a status signal bus.
The present application also provides a computer storage medium having stored thereon a computer program which, when executed by a computer, enables the computer to perform the method of the above-described method embodiments. Alternatively, embodiments of the present application also provide a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of the method embodiments described above.
When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a Solid State Drive (SSD)), or the like.
Those of ordinary skill in the art will appreciate that the various illustrative modules and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.
The modules illustrated as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, functional modules in various embodiments of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module.
The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (19)
1. A congestion control method, applied to a routing switch in a data center network, comprising:
Receiving a first flow message sent by a first sending end in a data center network;
discarding the first traffic message if the message buffer queue is congested and the first traffic message is an external traffic message of the data center network or if the message buffer queue is congested and the first traffic message is an internal traffic message of the data center network and no external traffic message of the data center network exists in the message buffer queue;
If the message buffer queue is congested and the first flow message is an internal flow message of the data center network, and meanwhile, an external flow message of the data center network exists in the message buffer queue, discarding a second flow message in the message buffer queue and inserting the first flow message into the message buffer queue;
Wherein the second traffic message is an external traffic message of the data center network in the message buffer queue.
2. The method of claim 1, wherein after discarding the first traffic message, further comprising:
sending a first congestion feedback message to the first sending end;
the first congestion message is used for the first sending end to adjust a sending window of the first sending end, and retransmits the packet loss of the first sending end.
3. The method of claim 2, wherein the first congestion feedback message comprises at least one of: the message type, the first identifier, the first numerical value and the first message number of the first congestion feedback message;
The first identifier is used for identifying whether congestion packet loss exists or not; the first value is the packet loss number of the first sending end in a first preset duration; the first message number is a message number of the packet lost by the first sending end within the first preset duration.
4. A method according to claim 2 or 3, wherein the adjustment manner of the transmission window of the first transmitting end is: and reducing the sending window of the first sending end to a first preset size.
5. The method of claim 3 wherein if the first congestion feedback message includes the first message number, the message retransmitted by the first sender is the message of the first message number.
6. The method of claim 1, wherein after inserting the first traffic message into the message buffer queue, further comprising:
Sending a second congestion feedback message to the first sending end;
The second congestion feedback message is used for the first sending end to adjust a sending window of the first sending end and transmit a flow message to be transmitted.
7. The method of claim 6, wherein the second congestion feedback message comprises at least one of: the message type, the second identifier, the second numerical value and the second message number of the second congestion feedback message;
the second identifier is used for identifying whether the connection between the first sending end and the receiving end of the first flow message has the condition of borrowing other connection transmission resources within a second preset duration; the second value is the number of messages transmitted by the connection between the first sending end and the receiving end of the first flow message by using other connection transmission resources within the second preset duration; the second message number is a message number of a message transmitted by the connection between the first sending end and the receiving end of the first flow message by using other connection transmission resources within the second preset duration.
8. The method of claim 7, wherein if the second congestion feedback message includes the second value, the transmission window adjustment manner of the first transmitting end is: firstly reducing the sending window of the first sending end to a second preset size, and then subtracting the second numerical value from the sending window of the first sending end.
9. The method of claim 1, wherein after discarding the second traffic message in the message buffer queue, further comprising:
sending a third congestion feedback message to a second sending end of the second traffic message;
The third congestion feedback message is used for the second sending end to adjust a sending window of the second sending end, and retransmits the packet loss of the second sending end.
10. The method of claim 9, wherein the third congestion feedback message comprises at least one of: the message type, the third identifier, the third numerical value and the third message number of the third congestion feedback message;
The third identifier is used for identifying whether the connection between the second sending end and the receiving end of the second flow message has the condition of borrowing transmission resources by other connections within a third preset duration; the third value is the number of packet loss of the second sending end caused by borrowing transmission resources by other connections in the third preset duration by the connection between the second sending end and the receiving end of the second flow message; and the third message number is a message number of a packet lost by the second sending end due to the fact that the connection between the second sending end and the receiving end of the second flow message is borrowed by other connections to transmit resources within the third preset duration.
11. The method of claim 10, wherein if the third congestion feedback message includes the third value, the adjustment manner of the transmission window of the second transmitting end is: firstly reducing the sending window of the second sending end to a third preset size, and then adding the third numerical value to the sending window of the second sending end.
12. The method of claim 11, wherein if the third congestion feedback message includes the third message number, the message retransmitted by the second sender is the message of the third message number.
13. A method according to any one of claims 1-3, further comprising:
Detecting whether the first flow message carries a fourth identifier for identifying that the first flow message is an internal flow message of the data center network or not so as to detect whether the first flow message is the internal flow message of the data center network or not; or alternatively
Detecting the value of a fifth identifier in the first flow message; and detecting whether the first traffic message is an internal traffic message of the data center network based on the value of the fifth identifier.
14. A method according to any one of claims 1-3, wherein before discarding the second traffic message in the message buffer queue, further comprising:
scanning from the tail of the message buffer queue;
and taking the external flow message of the first scanned data center network as the second flow message.
15. A method according to any one of claims 1-3, wherein said inserting said first traffic message into said message buffer queue comprises:
and inserting the first flow message into the tail of the message buffer queue.
16. A congestion control apparatus, wherein the apparatus is located within a data center network, the apparatus comprising:
the receiving and transmitting module is used for receiving a first flow message sent by a first sending end in the data center network;
A processing module for:
discarding the first traffic message if the message buffer queue is congested and the first traffic message is an external traffic message of the data center network or if the message buffer queue is congested and the first traffic message is an internal traffic message of the data center network and no external traffic message of the data center network exists in the message buffer queue;
If the message buffer queue is congested and the first flow message is an internal flow message of the data center network, and meanwhile, an external flow message of the data center network exists in the message buffer queue, discarding a second flow message in the message buffer queue and inserting the first flow message into the message buffer queue;
Wherein the second traffic message is an external traffic message of the data center network in the message buffer queue.
17. An electronic device, comprising:
A processor and a memory for storing a computer program, the processor being for invoking and running the computer program stored in the memory to perform the method of any of claims 1 to 15.
18. A computer readable storage medium storing a computer program for causing a computer to perform the method of any one of claims 1 to 15.
19. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the method of any one of claims 1 to 15.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202310202979.XA CN118540272A (en) | 2023-02-23 | 2023-02-23 | Congestion control method, device, equipment, medium and program product |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202310202979.XA CN118540272A (en) | 2023-02-23 | 2023-02-23 | Congestion control method, device, equipment, medium and program product |
Publications (1)
Publication Number | Publication Date |
---|---|
CN118540272A true CN118540272A (en) | 2024-08-23 |
Family
ID=92383408
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202310202979.XA Pending CN118540272A (en) | 2023-02-23 | 2023-02-23 | Congestion control method, device, equipment, medium and program product |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN118540272A (en) |
-
2023
- 2023-02-23 CN CN202310202979.XA patent/CN118540272A/en active Pending
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111740808B (en) | Data transmission method and device | |
US11558302B2 (en) | Data transmission method and apparatus | |
US10826830B2 (en) | Congestion processing method, host, and system | |
US7369498B1 (en) | Congestion control method for a packet-switched network | |
CN109412946B (en) | Method, device, server and readable storage medium for determining back source path | |
KR100785293B1 (en) | System and Method for TCP Congestion Control Using Multiple TCP ACKs | |
CN110677221B (en) | Retransmission control method, communication interface and electronic equipment | |
JP2024509728A (en) | Data retransmission processing method, device, computer equipment and computer program | |
KR20130082070A (en) | Communication apparatus and communication method | |
CN111435866B (en) | Data transmission method and related device | |
US8589586B2 (en) | Method and apparatus for managing transmission of TCP data segments | |
US10897725B2 (en) | System and method for managing data transfer between two different data stream protocols | |
US10136375B2 (en) | Method for service data management, apparatus, and system | |
CN111314961A (en) | TCP transmission method, device and system | |
CN116827853A (en) | Path processing method and device and electronic equipment | |
CN111385069A (en) | Data transmission method and computer equipment | |
CN118540272A (en) | Congestion control method, device, equipment, medium and program product | |
US7490160B2 (en) | Method of efficiently transmitting/receiving data using transport layer in a mobile ad hoc network, and network device using the method | |
CN113424578B (en) | Acceleration method and device for transmission control protocol | |
KR100756183B1 (en) | Network transmission apparatus having fake-ack layer and tcp packet sending/receiving method using there of | |
WO2021100178A1 (en) | Communication device, communication system, communication method, and non-temporary computer-readable medium storing program | |
CN118381839A (en) | Computer device and transmission control protocol message processing method thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication |