CN109120454B - QoS flow rate limiting system and method - Google Patents
QoS flow rate limiting system and method Download PDFInfo
- Publication number
- CN109120454B CN109120454B CN201811024132.2A CN201811024132A CN109120454B CN 109120454 B CN109120454 B CN 109120454B CN 201811024132 A CN201811024132 A CN 201811024132A CN 109120454 B CN109120454 B CN 109120454B
- Authority
- CN
- China
- Prior art keywords
- bridge
- flow
- qos
- flow table
- speed limit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0823—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/54—Organization of routing tables
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/24—Traffic characterised by specific attributes, e.g. priority or QoS
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/25—Flow control; Congestion control with rate being modified by the source upon detecting a change of network conditions
Abstract
The invention provides a QoS flow rate limiting system and a QoS flow rate limiting method. The method comprises the steps that a policy management unit receives an external configuration request comprising a request type and a speed limit requirement aiming at QoS flow; when the request type is newly increased, the flow guide flow table management unit sends a corresponding flow guide flow table down on a br-int bridge of a corresponding data plane node, and the execution flow table management unit sends a corresponding flow execution flow table including a speed limit requirement down on a br-qos bridge of a corresponding data plane node; each data plane node establishes a corresponding speed limit queue aiming at the speed limit requirement of each flow execution flow table on the br-qos bridge; forwarding the QoS flow for each flow drainage flow table to a br-QoS bridge through the br-int bridge; and (4) pumping each QoS flow into a corresponding speed limit queue through the br-QoS bridge to carry out speed limit processing. The scheme can support QoS flow rate limit with different granularities and has high flexibility of rate limit.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a QoS flow rate limiting system and a QoS flow rate limiting method.
Background
The Service Quality of the virtualized network directly affects the user Service Quality in the cloud computing environment, and the Quality of Service (QoS) guarantee capability of the virtualized network becomes an important basic Service capability in the cloud computing system. In the actual use process, the QoS guarantee generally refers to the guarantee of the bandwidth.
Openstack has gained wide attention and general popularity in the industry as a mainstream product of current open source Iaas cloud computing, but there is still room for improvement in the aspect of specific implementation technology, for example, the QoS bandwidth guarantee implementation mode of Openstack is about to be improved.
At present, the QoS bandwidth guarantee implementation manner of openstack may be a global speed limit for a virtual machine port. For example, the whole bandwidth of a certain virtual machine port is limited directly through libvirt.
However, limiting the speed directly to the virtual machine port results in an excessively large granularity of the speed limit, and therefore the speed limit lacks sufficient flexibility.
Disclosure of Invention
The invention provides a QoS flow rate limiting system and a method, which can support QoS flow rate limiting with different granularities and have high rate limiting flexibility.
In order to achieve the purpose, the invention is realized by the following technical scheme:
in one aspect, the present invention provides a flow rate limiting system, including:
the system comprises at least one data plane node, a policy management unit, a drainage flow table management unit and an execution flow table management unit;
at least one virtual machine is correspondingly established on each data plane node, and each virtual machine comprises at least one QoS flow;
the policy management unit is used for receiving an externally input configuration request aiming at target QoS flow, and the configuration request comprises a request type and a speed limit requirement;
the flow guide flow table management unit is used for issuing a flow guide flow table aiming at the target QoS flow on an integrated bridge br-int bridge in a target data plane node when the request type is newly added, wherein a virtual machine comprising the target QoS flow is established on the target data plane node;
the execution flow table management unit is used for issuing a flow execution flow table which aims at the target QoS flow and comprises the speed limit requirement on a service quality bridge Br-QoS bridge in the target data plane node when the request type is newly increased;
each data plane node is configured to execute, for each flow execution flow table issued to a first service quality bridge Br-qos bridge pre-introduced inside: establishing a speed limit queue corresponding to the QoS flow aimed at by the current flow execution flow table according to the speed limit requirement included by the current flow execution flow table; respectively forwarding QoS (quality of service) flows, which are issued to each flow diversion flow table on a first integrated bridge Br-int bridge, to the first quality of service bridge Br-QoS bridge through the first integrated bridge Br-int bridge integrated inside; executing, by the first quality of service bridge Br-QoS bridge, for each QoS traffic forwarded by the first integrated bridge Br-int bridge: and the current QoS flow is injected into a speed limit queue corresponding to the current QoS flow so as to carry out speed limit treatment.
Further, when the request type is newly added, the speed limit requirement includes: a speed limit threshold and a speed limit type;
the speed limit type comprises the following steps: and any one or more of a source media access control layer Mac, a destination media access control layer Mac, a virtual local area network Vlan number, a source port, a source IP, a destination port, a destination IP and a transport layer protocol type of the virtual machine including the target QoS flow.
Further, the flow rate limiting system further comprises: a policy database;
the policy management unit is further configured to log the configuration request into the policy database, and respectively trigger the drainage flow table management unit and the execution flow table management unit to process the configuration request;
the flow guide flow table management unit is further configured to delete the flow guide flow table, which is issued to the br-int bridge of the integrated network bridge in the target data plane node and is directed to the target QoS flow, when the request type is deletion; when the request type is modification or viewing, ending;
the execution flow table management unit is further configured to delete the flow execution flow table for the target QoS flow, which is issued to the service quality bridge Br-QoS bridge in the target data plane node, when the request type is deletion; when the request type is modified, updating a flow execution flow table aiming at the target QoS flow and issued to a service quality bridge Br-QoS bridge in the target data plane node according to the speed limit requirement; and ending when the request type is checking.
Further, the drainage flow table management unit is further configured to generate a path correction flow table for the target QoS flow according to an original path flow table for the target QoS flow, which is shorted by the flow drainage flow table for the target QoS flow, where a match field uses a virtual local area network Vlan number, an action field uses an action field and a metadata field of the original path flow table;
each data plane node is further configured to send the current QoS traffic after rate limiting back to the first integrated bridge Br-int bridge through the first QoS bridge Br-QoS bridge after performing rate limiting on the current QoS traffic; and correcting a flow table according to the path aiming at the current QoS flow through the first integrated bridge br-int bridge, and reintroducing the current QoS flow after speed limiting into the normal forwarding flow of the data plane node.
Further, the flow rate limiting system further comprises: a setting unit configured to execute, for each of the data plane nodes: introducing a service quality bridge Br-qos bridge into a current data plane node by using a qos queue mechanism of an open virtual switch (Ovs) (openvswitch); and connecting the introduced service quality bridge Br-QoS bridge with the integrated bridge Br-int bridge integrated in the current data plane node through two pairs of virtual Ethernet Veth ports so as to control the access of QoS flow between the service quality bridge Br-QoS bridge and the integrated bridge Br-int bridge based on the two pairs of virtual Ethernet Veth ports.
On the other hand, the invention provides a flow rate limiting method based on any one of the flow rate limiting systems, which comprises the following steps:
receiving a configuration request aiming at target QoS flow input from outside through the policy management unit, wherein the configuration request comprises a request type and a speed limit requirement;
when the request type is newly increased, issuing a flow diversion flow table aiming at the target QoS flow on an integrated bridge br-int bridge in a target data plane node through the diversion flow table management unit, wherein a virtual machine comprising the target QoS flow is established on the target data plane node; issuing a flow execution flow table which aims at the target QoS flow and comprises the speed limit requirement on a service quality bridge Br-QoS bridge in the target data plane node through the execution flow table management unit;
executing, by each data plane node, each flow execution flow table issued to a first service quality bridge Br-qos bridge pre-introduced inside: establishing a speed limit queue corresponding to the QoS flow aimed at by the current flow execution flow table according to the speed limit requirement included by the current flow execution flow table; respectively forwarding QoS (quality of service) flows, which are issued to each flow diversion flow table on a first integrated bridge Br-int bridge, to the first quality of service bridge Br-QoS bridge through the first integrated bridge Br-int bridge integrated inside; executing, by the first quality of service bridge Br-QoS bridge, for each QoS traffic forwarded by the first integrated bridge Br-int bridge: and the current QoS flow is injected into a speed limit queue corresponding to the current QoS flow so as to carry out speed limit treatment.
Further, when the request type is newly added, the speed limit requirement includes: a speed limit threshold and a speed limit type;
the speed limit type comprises the following steps: and any one or more of a source media access control layer Mac, a destination media access control layer Mac, a virtual local area network Vlan number, a source port, a source IP, a destination port, a destination IP and a transport layer protocol type of the virtual machine including the target QoS flow.
Further, the flow rate limiting method further comprises the following steps: the configuration request is recorded into a policy database which is also included in the flow rate limiting system through the policy management unit;
when the request type is deletion, deleting the flow guide flow table which is issued to the integrated bridge br-int bridge in the target data plane node and aims at the target QoS flow through the guide flow table management unit; deleting a flow execution flow table aiming at the target QoS flow, which is issued to a service quality bridge Br-QoS bridge in the target data plane node, by the execution flow table management unit;
when the request type is modified, updating a flow execution flow table aiming at the target QoS flow, which is issued to a service quality bridge Br-QoS bridge in the target data plane node, by the execution flow table management unit according to the speed limit requirement;
and ending when the request type is checking.
Further, the flow rate limiting method further comprises the following steps: generating, by the drainage flow table management unit, a path correction flow table for the target QoS flow according to an original path flow table for the target QoS flow, which is short-circuited by a flow drainage flow table for the target QoS flow, wherein a virtual local area network Vlan number is used for a match domain, and an action domain and a metadata domain of the original path flow table are used for an action domain;
after the current QoS flow is subjected to speed limiting processing through each data plane node, the current QoS flow subjected to speed limiting is sent back to the first integrated bridge Br-int bridge through the first service quality bridge Br-QoS bridge; and correcting a flow table according to the path aiming at the current QoS flow through the first integrated bridge br-int bridge, and reintroducing the current QoS flow after speed limiting into the normal forwarding flow of the data plane node.
Further, the flow rate limiting method further comprises the following steps: through a setting unit included in the flow rate limiting system, for each data plane node, executing: introducing a service quality bridge Br-qos bridge into the current data plane node by using a qos queue mechanism of the open virtual switch Ovs; and connecting the introduced service quality bridge Br-QoS bridge with the integrated bridge Br-int bridge integrated in the current data plane node through two pairs of virtual Ethernet Veth ports so as to control the access of QoS flow between the service quality bridge Br-QoS bridge and the integrated bridge Br-int bridge based on the two pairs of virtual Ethernet Veth ports.
The invention provides a QoS flow rate limiting system and a QoS flow rate limiting method. The method comprises the steps that a policy management unit receives an external configuration request comprising a request type and a speed limit requirement aiming at QoS flow; when the request type is newly increased, the flow guide flow table management unit sends a corresponding flow guide flow table down on an integrated bridge Br-int bridge of a corresponding data plane node, and the execution flow table management unit sends a corresponding flow execution flow table including speed limit requirements down on a service quality bridge Br-qos bridge of the corresponding data plane node; each data plane node establishes a corresponding speed limit queue aiming at the speed limit requirement of each flow execution flow table on the service quality bridge Br-qos bridge; forwarding the QoS flow for each flow diversion flow table to a service quality bridge Br-QoS bridge through an integrated bridge Br-int bridge; and (4) pumping each QoS flow into a corresponding speed limit queue through a service quality bridge Br-QoS bridge to carry out speed limit treatment. The invention can support QoS flow rate limit with different granularities and has high flexibility of rate limit.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic diagram of a QoS traffic speed limiting system according to an embodiment of the present invention;
fig. 2 is a schematic diagram illustrating an internal flow guiding and speed limiting principle of a QoS data plane node according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating another QoS traffic speed limiting system according to an embodiment of the present invention;
fig. 4 is a flowchart of another QoS traffic rate limiting method according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer and more complete, the technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention, and based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative efforts belong to the scope of the present invention.
As shown in fig. 1 and referring to fig. 2, an embodiment of the present invention provides a flow rate limiting system, which may include:
at least one data plane node 101, a policy management unit 102, a drainage flow table management unit 103, an execution flow table management unit 104;
at least one virtual machine is correspondingly established on each data plane node 101, and each virtual machine comprises at least one QoS flow;
the policy management unit 102 is configured to receive an externally input configuration request for a target QoS traffic, where the configuration request includes a request type and a speed limit requirement;
the flow-guiding flow table management unit 103 is configured to issue a flow-guiding flow table for the target QoS flow on an integrated bridge br-int bridge in a target data plane node when the request type is newly added, where a virtual machine including the target QoS flow is created on the target data plane node;
the execution flow table management unit 104 is configured to issue a flow execution flow table including the speed limit requirement for the target QoS flow on a QoS bridge Br-QoS bridge in the target data plane node when the request type is newly added;
each data plane node 101 is configured to execute, for each traffic execution flow table issued to a first service quality bridge Br-qos bridge pre-introduced inside: establishing a speed limit queue corresponding to the QoS flow aimed at by the current flow execution flow table according to the speed limit requirement included by the current flow execution flow table; respectively forwarding QoS (quality of service) flows, which are issued to each flow diversion flow table on a first integrated bridge Br-int bridge, to the first quality of service bridge Br-QoS bridge through the first integrated bridge Br-int bridge integrated inside; executing, by the first quality of service bridge Br-QoS bridge, for each QoS traffic forwarded by the first integrated bridge Br-int bridge: and the current QoS flow is injected into a speed limit queue corresponding to the current QoS flow so as to carry out speed limit treatment.
The embodiment of the invention provides a QoS flow rate limiting system, which comprises at least one data plane node, a policy management unit, a drainage flow table management unit and an execution flow table management unit. The method comprises the steps that a policy management unit receives an external configuration request comprising a request type and a speed limit requirement aiming at QoS flow; when the request type is newly increased, the flow guide flow table management unit sends a corresponding flow guide flow table down on an integrated bridge Br-int bridge of a corresponding data plane node, and the execution flow table management unit sends a corresponding flow execution flow table including speed limit requirements down on a service quality bridge Br-qos bridge of the corresponding data plane node; each data plane node establishes a corresponding speed limit queue aiming at the speed limit requirement of each flow execution flow table on the service quality bridge Br-qos bridge; forwarding the QoS flow for each flow diversion flow table to a service quality bridge Br-QoS bridge through an integrated bridge Br-int bridge; and (4) pumping each QoS flow into a corresponding speed limit queue through a service quality bridge Br-QoS bridge to carry out speed limit treatment. The embodiment of the invention can support QoS flow rate limit with different granularities and has high flexibility of rate limit.
Referring to fig. 2, vm (virtual Machine) in fig. 2 may be represented as a virtual Machine, and queue may be represented as a queue. Please refer to the arrow direction shown in fig. 2 for the speed-limiting path of the traffic to be limited from the virtual machine.
In an embodiment of the present invention, the data platform node may be a computing node, a network node, or the like in a data center. The data plane node is usually created with a plurality of virtual machines, and the embodiment of the invention can limit the speed of any QoS traffic involved by each virtual machine.
In detail, the external part can input a configuration request for any QoS flow to request the flow rate limiting of the QoS flow. This configuration request typically includes, among other things, the request type and the speed limit requirements. For example, the request type may be any one of addition, deletion, modification and check, and the speed limit requirement may relate to a speed limit threshold, a speed limit type and the like.
In an embodiment of the present invention, a user may add a QoS rule through a front-end configuration interface operation, and send the operation request to the policy management unit through a REST (Representational State Transfer) interface call or other manners.
The following description is directed to the addition, deletion, and modification:
aiming at new addition:
based on the above, in an embodiment of the present invention, when the request type is newly added, the speed limit requirement includes: a speed limit threshold and a speed limit type;
the speed limit type comprises the following steps: and any one or more of a source media access control layer Mac, a destination media access control layer Mac, a virtual local area network Vlan number, a source port, a source IP, a destination port, a destination IP and a transport layer protocol type of the virtual machine including the target QoS flow.
Therefore, it can be clear which QoS flow is to be speed-limited, the speed-limiting value is determined, and what the speed-limiting type is. For example, some of the methods only limit the speed of a source port, some of the methods only limit the speed of a destination port, some of the methods limit the speed of a combination of a source IP and a destination IP, and so on.
Taking IP as an example, for example, if a virtual machine requests news in new wave, the IP of the physical machine where the virtual machine is located is the source IP, and the IP of the server providing news in new wave is the destination IP. Thus, the speed of the north-south flow is limited.
Certainly, when the virtual machine on one node interacts with the virtual machine information on other nodes, the IP of the node is the source IP, and the IP of the other nodes is the destination IP. Thus, the east-west flow is limited.
For deletion:
based on the above, in an embodiment of the present invention, when the request type is deletion, the speed limit requirement may be null.
For the modification:
based on the above, in an embodiment of the present invention, when the request type is modification, the speed limit requirement may include: a speed limit threshold and/or a speed limit type;
the speed limit type comprises the following steps: and any one or more of a source media access control layer Mac, a destination media access control layer Mac, a virtual local area network Vlan number, a source port, a source IP, a destination port, a destination IP and a transport layer protocol type of the virtual machine including the target QoS flow.
For viewing:
based on the above, in an embodiment of the present invention, when the request type is deletion, the speed limit requirement may be null.
In general, the number of configuration requests is usually large, so that each configuration request inputted from the outside can be stored.
Therefore, in an embodiment of the present invention, referring to fig. 3, the flow rate limiting system further includes: a policy database 105;
the policy management unit 102 is further configured to log the configuration request into the policy database 105, and respectively trigger the drainage flow table management unit 103 and the execution flow table management unit 104 to process the configuration request;
the flow guide flow table management unit 103 is further configured to delete the flow guide flow table for the target QoS flow, which is issued to the br-int bridge of the integrated bridge in the target data plane node, when the request type is deletion; when the request type is modification or viewing, ending;
the execution flow table management unit 104 is further configured to delete the flow execution flow table for the target QoS flow, which is issued to the service quality bridge Br-QoS bridge in the target data plane node, when the request type is deletion; when the request type is modified, updating a flow execution flow table aiming at the target QoS flow and issued to a service quality bridge Br-QoS bridge in the target data plane node according to the speed limit requirement; and ending when the request type is checking.
It can be seen that when the request types of the configuration requests are different, the processing of the drainage flow table management unit and the execution flow table management unit is slightly different.
Based on the above content, the flow guide flow table management unit mainly issues the flow guide flow table of each flow to be rate-limited to the integrated bridge Br-int bridge in the corresponding data plane node, so that the integrated bridge Br-int bridge forwards each flow to be rate-limited to the service quality bridge Br-qos bridge, and the service quality bridge Br-qos bridge executes rate-limiting according to the requirement of rate-limiting. The execution flow table management unit issues flow execution flow tables including speed limit requirements of various to-be-limited flows to corresponding service quality bridge Br-qos bridges.
Thus, the drainage flow table management unit is related to the request type, but is not related to the speed limit requirement much. Therefore, when the request type is deletion, the drainage flow table management unit can delete the issued corresponding flow drainage flow table so as to avoid limiting the speed of the corresponding flow. When the request type is modification or viewing, the drainage flow table management unit may not have corresponding operation.
Correspondingly, the execution flow table management unit may not have a corresponding operation only when the request type is view.
Therefore, in summary, the speed limit processing of the corresponding flow can be realized for various speed limit requirements required by each user.
In an embodiment of the present invention, the drainage flow table management unit 103 is further configured to generate a path correction flow table for the target QoS flow according to an original path flow table for the target QoS flow, which is shorted by the flow drainage flow table for the target QoS flow, where a match field uses a virtual local area network Vlan number, and an action field uses an action field and a metadata field of the original path flow table;
each data plane node 101 is further configured to send the current QoS traffic after rate limiting back to the first integrated bridge Br-int bridge through the first QoS bridge Br-QoS bridge after performing rate limiting on the current QoS traffic; and correcting the flow table according to the path aiming at the current QoS flow through the first integrated bridge br-int bridge, and reintroducing the current QoS flow after speed limiting into the normal forwarding flow of the data plane node 101.
In detail, in order to distinguish the rate-limiting traffic from different ports on the br-int bridge of the integrated bridge, an action of a push virtual local area network Vlan can be added to the action of the flow-directing table, and the port number is injected into the virtual local area network Vlan field; because the flow table on the integrated bridge Br-int is generally closely related to the ingress port, and the ingress port of the message returned from the qos bridge Br-qos has changed and cannot continue to go the original path, the flow table management unit may also supplement a flow table to reform the rate-limited traffic passing through the qos bridge Br-qos to the original flow table path, which is specifically done in the following manner: and taking out the original path flow table of the service flow short-circuited by the drainage flow table, and generating a path correction flow table, wherein the match field uses a virtual local area network Vlan number (port number), the action field uses the action field and the metadata field of the original path flow table.
Therefore, in the embodiment of the present invention, a message, or called a traffic to be rate-limited, enters the integrated bridge Br-int bridge after coming out of the virtual machine, the message is guided to the qos bridge Br-qos bridge by a flow guide table on the integrated bridge Br-int bridge, the rate-limited flow table is matched after entering the qos bridge Br-qos, the rate-limited flow table is sent back to the integrated bridge Br-int after entering the rate-limited queue, and the message returned to the integrated bridge Br-int for the second time corrects the flow table for the matched path, so that the subsequent path (e.g., overlay) is forwarded continuously, and thus the data plane forwarding flow of the normal virtual network is not affected.
In an embodiment of the present invention, referring to fig. 3, the flow rate limiting system further includes: a setting unit 106, configured to perform, for each of the data plane nodes 101: introducing a service quality bridge Br-qos bridge into the current data plane node by using a qos queue mechanism of the open virtual switch Ovs; and connecting the introduced service quality bridge Br-QoS bridge with the integrated bridge Br-int bridge integrated in the current data plane node through two pairs of virtual Ethernet Veth ports so as to control the access of QoS flow between the service quality bridge Br-QoS bridge and the integrated bridge Br-int bridge based on the two pairs of virtual Ethernet Veth ports.
In the embodiment of the invention, a QoS management bridge service quality bridge Br-QoS is introduced to each data plane node, and then a QoS queue mechanism of an open virtual switch Ovs is utilized to realize a flexible speed limit strategy on the QoS bridge Br-QoS, so that the speed limit at a traffic flow (flow) level can be accurately achieved. The service quality bridge Br-QoS bridge can be connected with the integrated bridge Br-int bridge through two pairs of virtual Ethernet Veth ports to respectively realize the entrance and exit of QoS flow. And sending a QoS flow forwarding table on the integrated bridge Br-int, so that the flow needing speed limitation can be forwarded to the service quality bridge Br-QoS bridge. And issuing a rate-limiting flow table of QoS flow on a service quality bridge Br-QoS bridge so as to limit the flow needing rate limiting according to the requirement. The service quality bridge Br-qos bridge can also send the traffic after speed limiting back to the integrated bridge Br-int bridge.
In one embodiment of the invention, for QoS module initialization, a QoS bridge quality of service bridge Br-QoS is created by the open virtual switch Ovsdb protocol or manually and the bridge is associated to the control plane of the virtual Network, such as SDN (Software Defined Network) controller or openstack's neutron server.
In summary, compared with the existing speed limiting mode of global speed limiting for the ports of the virtual machines, the speed limiting requirement in the embodiment of the invention can be set as required, so the granularity of the speed limiting can be large or small, and the speed can be limited for both north-south traffic and east-west traffic, so that the flexibility of the speed limiting is greatly improved.
In addition, compared with the south-north speed limit for the Floating IP, that is, a TC (traffic control) speed limit function is started on the qrouter of the virtual network to limit the bandwidth of the specified Floating IP, this existing speed limit mode is that the TC speed limit on the qrouter can only limit the south-north flow, which is a centralized speed limit scheme and cannot limit the east-west flow. Correspondingly, the embodiment of the invention can avoid the problems that the south-north throughput capacity of the whole network is influenced by the over-concentration of the speed limit points possibly caused by the general location of the qrouter on the network node, so that the service quality of the whole data center is influenced, and the like.
Based on the above, in an embodiment of the present invention, the flow rate limiting system may include a plurality of data plane nodes, a policy management unit, a drainage flow table management unit, an execution flow table management unit, and a policy database.
In detail, the policy management unit may be responsible for configuration management of user QoS policy data, receive QoS configuration requests from users (e.g., add, delete, modify, view QoS configuration, etc.), and may post user QoS configuration data in the policy database.
In detail, the policy database may be responsible for recording QoS policy configuration information of a user, and providing a data source for the drainage flow table management unit and the execution flow table management unit.
In detail, the execution flow table management unit may mainly include two functions, that is, translating the QoS policy of the user into an execution flow table of the QoS data plane, and managing the flow table of the QoS data plane, that is, managing a specific QoS execution flow table on the quality of service bridge Br-QoS bridge.
In detail, the flow-guiding table management unit may be responsible for managing a QoS flow-guiding table on the integrated bridge Br-int bridge of the data plane node, adding or deleting the flow table on the integrated bridge Br-int bridge according to user configuration, and guiding the traffic that needs to execute QoS speed limiting to the QoS execution bridge Br-QoS. In addition, a path recovery flow table can be added to reintroduce the traffic limited by the Br-qos bridge of the qos bridge into the normal forwarding flow of the data plane, such as the data plane forwarding processing flow of the vxlan virtual network.
In detail, each data plane node may include a QoS enforcement unit. The QoS execution unit can be responsible for executing a specific QoS strategy, the main body of the QoS execution unit is a quality of service bridge Br-QoS bridge, different speed limit queues are established on the quality of service bridge Br-QoS, and different service flows are injected into different queues to realize speed limit processing of service flows.
The embodiment of the invention provides a general distributed QoS implementation mode based on an open virtual switch Ovs, which has more flexible QoS strategy management capability and can support QoS speed limit of various granularities, thereby improving the flexibility of QoS speed limit and realizing the refinement of QoS management granularity; the QoS speed limiting processing flow of the east-west flow and the south-north flow is unified, and the east-west flow QoS function and the south-north flow QoS function do not need to be independently developed, so that the unified QoS speed limitation of the east-west flow and the south-north flow is realized; the integrated full-distributed QoS implementation architecture has no performance bottleneck point, and has good expansibility and robustness.
As shown in fig. 4, an embodiment of the present invention provides a flow rate limiting method based on any one of the flow rate limiting systems described above, which may include the following steps:
step 401: receiving, by the policy management unit, an externally input configuration request for a target QoS traffic, where the configuration request includes a request type and a speed limit requirement.
Step 402: when the request type is newly increased, issuing a flow diversion flow table aiming at the target QoS flow on an integrated bridge br-int bridge in a target data plane node through the diversion flow table management unit, wherein a virtual machine comprising the target QoS flow is established on the target data plane node; and issuing a flow execution flow table which aims at the target QoS flow and comprises the speed limit requirement on a service quality bridge Br-QoS bridge in the target data plane node through the execution flow table management unit.
Step 403: executing, by each data plane node, each flow execution flow table issued to a first service quality bridge Br-qos bridge pre-introduced inside: establishing a speed limit queue corresponding to the QoS flow aimed at by the current flow execution flow table according to the speed limit requirement included by the current flow execution flow table; respectively forwarding QoS (quality of service) flows, which are issued to each flow diversion flow table on a first integrated bridge Br-int bridge, to the first quality of service bridge Br-QoS bridge through the first integrated bridge Br-int bridge integrated inside; executing, by the first quality of service bridge Br-QoS bridge, for each QoS traffic forwarded by the first integrated bridge Br-int bridge: and the current QoS flow is injected into a speed limit queue corresponding to the current QoS flow so as to carry out speed limit treatment.
In an embodiment of the present invention, when the request type is newly added, the speed limit requirement includes: a speed limit threshold and a speed limit type;
the speed limit type comprises the following steps: and any one or more of a source media access control layer Mac, a destination media access control layer Mac, a virtual local area network Vlan number, a source port, a source IP, a destination port, a destination IP and a transport layer protocol type of the virtual machine including the target QoS flow.
In one embodiment of the present invention, the method may further comprise:
a1: the configuration request is recorded into a policy database which is also included in the flow rate limiting system through the policy management unit;
a2: when the request type is deletion, deleting the flow guide flow table which is issued to the integrated bridge br-int bridge in the target data plane node and aims at the target QoS flow through the guide flow table management unit; deleting a flow execution flow table aiming at the target QoS flow, which is issued to a service quality bridge Br-QoS bridge in the target data plane node, by the execution flow table management unit;
a3: when the request type is modified, updating a flow execution flow table aiming at the target QoS flow, which is issued to a service quality bridge Br-QoS bridge in the target data plane node, by the execution flow table management unit according to the speed limit requirement;
a4: and ending when the request type is checking.
In detail, a1 may be performed after step 401, before steps 402 and a2, A3, a 4.
In addition, in step 401, when the request type of the externally input configuration request is newly added, step 402 is executed; when deleting, executing A2; when the modification is performed, A3 is executed; to view, A4 is executed.
In one embodiment of the present invention, the method may further comprise:
b1: generating, by the drainage flow table management unit, a path correction flow table for the target QoS flow according to an original path flow table for the target QoS flow, which is short-circuited by a flow drainage flow table for the target QoS flow, wherein a virtual local area network Vlan number is used for a match domain, and an action domain and a metadata domain of the original path flow table are used for an action domain;
b2: after the current QoS flow is subjected to speed limiting processing through each data plane node, the current QoS flow subjected to speed limiting is sent back to the first integrated bridge Br-int bridge through the first service quality bridge Br-QoS bridge; and correcting a flow table according to the path aiming at the current QoS flow through the first integrated bridge br-int bridge, and reintroducing the current QoS flow after speed limiting into the normal forwarding flow of the data plane node.
In detail, B1 is performed after step 401 and before B2. And B2 is performed after step 403.
In one embodiment of the present invention, the method may further comprise: through a setting unit included in the flow rate limiting system, for each data plane node, executing: introducing a service quality bridge Br-qos bridge into the current data plane node by using a qos queue mechanism of the open virtual switch Ovs; and connecting the introduced service quality bridge Br-QoS bridge with the integrated bridge Br-int bridge integrated in the current data plane node through two pairs of virtual Ethernet Veth ports so as to control the access of QoS flow between the service quality bridge Br-QoS bridge and the integrated bridge Br-int bridge based on the two pairs of virtual Ethernet Veth ports.
In detail, this newly adding step may be performed in advance before step 401.
Because the contents of information interaction, execution process and the like among the devices, components, modules and units in the method are based on the same conception as the system embodiment of the invention, the specific contents can be referred to the description in the system embodiment of the invention, and are not described again here.
In summary, the embodiments of the present invention have at least the following advantages:
1. in the embodiment of the invention, the QoS flow rate limiting system comprises at least one data plane node, a policy management unit, a drainage flow table management unit and an execution flow table management unit. The method comprises the steps that a policy management unit receives an external configuration request comprising a request type and a speed limit requirement aiming at QoS flow; when the request type is newly increased, the flow guide flow table management unit sends a corresponding flow guide flow table down on an integrated bridge Br-int bridge of a corresponding data plane node, and the execution flow table management unit sends a corresponding flow execution flow table including speed limit requirements down on a service quality bridge Br-qos bridge of the corresponding data plane node; each data plane node establishes a corresponding speed limit queue aiming at the speed limit requirement of each flow execution flow table on the service quality bridge Br-qos bridge; forwarding the QoS flow for each flow diversion flow table to a service quality bridge Br-QoS bridge through an integrated bridge Br-int bridge; and (4) pumping each QoS flow into a corresponding speed limit queue through a service quality bridge Br-QoS bridge to carry out speed limit treatment. The embodiment of the invention can support QoS flow rate limit with different granularities and has high flexibility of rate limit.
2. In the embodiment of the invention, a universal distributed QoS implementation mode based on an open virtual switch Ovs has more flexible QoS strategy management capability and can support QoS speed limit of various granularities, thereby improving the flexibility of QoS speed limit and realizing the refinement of QoS management granularity; the QoS speed limiting processing flow of the east-west flow and the south-north flow is unified, and the east-west flow QoS function and the south-north flow QoS function do not need to be independently developed, so that the unified QoS speed limitation of the east-west flow and the south-north flow is realized; the integrated full-distributed QoS implementation architecture has no performance bottleneck point, and has good expansibility and robustness.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of other similar elements in a process, method, article, or apparatus that comprises the element.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it is to be noted that: the above description is only a preferred embodiment of the present invention, and is only used to illustrate the technical solutions of the present invention, and not to limit the protection scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (2)
1. A flow rate limiting system, comprising:
the system comprises at least one data plane node, a policy management unit, a drainage flow table management unit and an execution flow table management unit;
at least one virtual machine is correspondingly established on each data plane node, and each virtual machine comprises at least one QoS flow;
the policy management unit is used for receiving an externally input configuration request aiming at target QoS flow, and the configuration request comprises a request type and a speed limit requirement;
the flow guide flow table management unit is used for issuing a flow guide flow table aiming at the target QoS flow on an integrated bridge br-int bridge in a target data plane node when the request type is newly added, wherein a virtual machine comprising the target QoS flow is established on the target data plane node;
the execution flow table management unit is used for issuing a flow execution flow table which aims at the target QoS flow and comprises the speed limit requirement on a service quality bridge Br-QoS bridge in the target data plane node when the request type is newly increased;
each data plane node is configured to execute, for each flow execution flow table issued to a first service quality bridge Br-qos bridge pre-introduced inside: establishing a speed limit queue corresponding to the QoS flow aimed at by the current flow execution flow table according to the speed limit requirement included by the current flow execution flow table; respectively forwarding QoS (quality of service) flows, which are issued to each flow diversion flow table on a first integrated bridge Br-int bridge, to the first quality of service bridge Br-QoS bridge through the first integrated bridge Br-int bridge integrated inside; executing, by the first quality of service bridge Br-QoS bridge, for each QoS traffic forwarded by the first integrated bridge Br-int bridge: the current QoS flow is injected into a speed limit queue corresponding to the current QoS flow so as to carry out speed limit treatment;
when the request type is newly increased, the speed limit requirement comprises the following steps: a speed limit threshold and a speed limit type;
the speed limit type comprises the following steps: any one or more of a source media access control layer Mac, a destination media access control layer Mac, a virtual local area network Vlan number, a source port, a source IP, a destination port, a destination IP and a transport layer protocol type of the virtual machine including the target QoS flow;
further comprising: a policy database;
the policy management unit is further configured to log the configuration request into the policy database, and respectively trigger the drainage flow table management unit and the execution flow table management unit to process the configuration request;
the flow guide flow table management unit is further configured to delete the flow guide flow table, which is issued to the br-int bridge of the integrated network bridge in the target data plane node and is directed to the target QoS flow, when the request type is deletion; when the request type is modification or viewing, ending;
the execution flow table management unit is further configured to delete the flow execution flow table for the target QoS flow, which is issued to the service quality bridge Br-QoS bridge in the target data plane node, when the request type is deletion; when the request type is modified, updating a flow execution flow table aiming at the target QoS flow and issued to a service quality bridge Br-QoS bridge in the target data plane node according to the speed limit requirement; when the request type is checking, ending;
the flow guide table management unit is further configured to generate a path correction flow table for the target QoS traffic according to an original path flow table for the target QoS traffic, which is short-circuited by the flow guide flow table for the target QoS traffic, where a virtual local area network Vlan number is used in a match domain, and an action domain and a metadata domain of the original path flow table are used in an action domain;
each data plane node is further configured to send the current QoS traffic after rate limiting back to the first integrated bridge Br-int bridge through the first QoS bridge Br-QoS bridge after performing rate limiting on the current QoS traffic; correcting a flow table according to the path aiming at the current QoS flow through the first integrated bridge br-int bridge, and reintroducing the current QoS flow after speed limiting into a normal forwarding flow of a data plane node;
further comprising: a setting unit configured to execute, for each of the data plane nodes: introducing a service quality bridge Br-qos bridge into the current data plane node by using a qos queue mechanism of the open virtual switch Ovs; and connecting the introduced service quality bridge Br-QoS bridge with the integrated bridge Br-int bridge integrated in the current data plane node through two pairs of virtual Ethernet Veth ports so as to control the access of QoS flow between the service quality bridge Br-QoS bridge and the integrated bridge Br-int bridge based on the two pairs of virtual Ethernet Veth ports.
2. A flow rate limiting method based on the flow rate limiting system according to claim 1, comprising:
receiving a configuration request aiming at target QoS (quality of service) flow input from the outside through the policy management unit, wherein the configuration request comprises a request type and a speed limit requirement;
when the request type is newly increased, issuing a flow diversion flow table aiming at the target QoS flow on an integrated bridge br-int bridge in a target data plane node through the diversion flow table management unit, wherein a virtual machine comprising the target QoS flow is established on the target data plane node; issuing a flow execution flow table which aims at the target QoS flow and comprises the speed limit requirement on a service quality bridge Br-QoS bridge in the target data plane node through the execution flow table management unit;
executing, by each data plane node, each flow execution flow table issued to a first service quality bridge Br-qos bridge pre-introduced inside: establishing a speed limit queue corresponding to the QoS flow aimed at by the current flow execution flow table according to the speed limit requirement included by the current flow execution flow table; respectively forwarding QoS (quality of service) flows, which are issued to each flow diversion flow table on a first integrated bridge Br-int bridge, to the first quality of service bridge Br-QoS bridge through the first integrated bridge Br-int bridge integrated inside; executing, by the first quality of service bridge Br-QoS bridge, for each QoS traffic forwarded by the first integrated bridge Br-int bridge: the current QoS flow is injected into a speed limit queue corresponding to the current QoS flow so as to carry out speed limit treatment;
when the request type is newly increased, the speed limit requirement comprises the following steps: a speed limit threshold and a speed limit type;
the speed limit type comprises the following steps: any one or more of a source media access control layer Mac, a destination media access control layer Mac, a virtual local area network Vlan number, a source port, a source IP, a destination port, a destination IP and a transport layer protocol type of the virtual machine including the target QoS flow;
further comprising: the configuration request is recorded into a policy database which is also included in the flow rate limiting system through the policy management unit;
when the request type is deletion, deleting the flow guide flow table which is issued to the integrated bridge br-int bridge in the target data plane node and aims at the target QoS flow through the guide flow table management unit; deleting a flow execution flow table aiming at the target QoS flow, which is issued to a service quality bridge Br-QoS bridge in the target data plane node, by the execution flow table management unit;
when the request type is modified, updating a flow execution flow table aiming at the target QoS flow, which is issued to a service quality bridge Br-QoS bridge in the target data plane node, by the execution flow table management unit according to the speed limit requirement;
when the request type is checking, the execution flow table management unit ends the operation;
when the request type is modification or checking, the drainage flow table management unit ends the operation;
further comprising: generating, by the drainage flow table management unit, a path correction flow table for the target QoS flow according to an original path flow table for the target QoS flow, which is short-circuited by a flow drainage flow table for the target QoS flow, wherein a virtual local area network Vlan number is used for a match domain, and an action domain and a metadata domain of the original path flow table are used for an action domain;
after the current QoS flow is subjected to speed limiting processing through each data plane node, the current QoS flow subjected to speed limiting is sent back to the first integrated bridge Br-int bridge through the first service quality bridge Br-QoS bridge; correcting a flow table according to the path aiming at the current QoS flow through the first integrated bridge br-int bridge, and reintroducing the current QoS flow after speed limiting into a normal forwarding flow of a data plane node;
further comprising: through a setting unit included in the flow rate limiting system, for each data plane node, executing: introducing a service quality bridge Br-qos bridge into the current data plane node by using a qos queue mechanism of the open virtual switch Ovs; and connecting the introduced service quality bridge Br-QoS bridge with the integrated bridge Br-int bridge integrated in the current data plane node through two pairs of virtual Ethernet Veth ports so as to control the access of QoS flow between the service quality bridge Br-QoS bridge and the integrated bridge Br-int bridge based on the two pairs of virtual Ethernet Veth ports.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811024132.2A CN109120454B (en) | 2018-09-04 | 2018-09-04 | QoS flow rate limiting system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811024132.2A CN109120454B (en) | 2018-09-04 | 2018-09-04 | QoS flow rate limiting system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109120454A CN109120454A (en) | 2019-01-01 |
| CN109120454B true CN109120454B (en) | 2021-06-15 |
Family
ID=64861904
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811024132.2A Active CN109120454B (en) | 2018-09-04 | 2018-09-04 | QoS flow rate limiting system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109120454B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112838936B (en) * | 2019-11-22 | 2023-05-26 | 北京金山云网络技术有限公司 | Resource speed limiting method, device, user terminal and server |
| CN111294291B (en) * | 2020-01-16 | 2022-07-12 | 新华三信息安全技术有限公司 | Protocol message processing method and device |
| CN111918340A (en) * | 2020-07-08 | 2020-11-10 | 河北百亚信息科技有限公司 | WiFi wireless network bandwidth allocation method |
| CN114338496B (en) * | 2020-10-12 | 2023-09-05 | 中移(苏州)软件技术有限公司 | Resource forwarding method, device, terminal and computer storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102611630B (en) * | 2012-04-12 | 2015-10-07 | 迈普通信技术股份有限公司 | A kind of message acceptance control method and system |
| CN105897611B (en) * | 2016-06-24 | 2019-03-29 | 武汉绿色网络信息服务有限责任公司 | A kind of system and method that business identification and flow scheduling is realized using DPI technology based on SDN |
| CN106789865B (en) * | 2016-07-14 | 2020-06-02 | 深圳市永达电子信息股份有限公司 | Network security protection method based on GRE network combined with SDN technology and honeypot technology |
| CN107800645B (en) * | 2016-09-06 | 2020-05-01 | 北京京东尚科信息技术有限公司 | Cloud host data sending, receiving and transmitting method and device |
| CN107864061A (en) * | 2017-11-15 | 2018-03-30 | 北京易讯通信息技术股份有限公司 | A kind of method of virtual machine port speed constraint and mirror image in private clound |
-
2018
- 2018-09-04 CN CN201811024132.2A patent/CN109120454B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109120454A (en) | 2019-01-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109120454B (en) | QoS flow rate limiting system and method | |
| US10419550B2 (en) | Automatic service function validation in a virtual network environment | |
| US9197563B2 (en) | Bypassing congestion points in a converged enhanced ethernet fabric | |
| US9417901B2 (en) | Switch and method for guaranteeing quality of service of multi-tenant cloud service and system having the same switch | |
| US9571417B2 (en) | Processing resource access request in network | |
| US9699063B2 (en) | Transitioning a routing switch device between network protocols | |
| US9014005B2 (en) | Low-latency lossless switch fabric for use in a data center | |
| US11102135B2 (en) | Latency reduction in service function paths | |
| US9590922B2 (en) | Programmable and high performance switch for data center networks | |
| US10296332B2 (en) | Systems and methods for selectively enabling and disabling hardware features | |
| CN107276916B (en) | Switch flow table management method based on protocol non-perception forwarding technology | |
| US10880220B1 (en) | Credit mechanisms for packet policing | |
| US10554513B2 (en) | Technologies for filtering network packets on ingress | |
| RU2584471C1 (en) | DEVICE FOR RECEIVING AND TRANSMITTING DATA WITH THE POSSIBILITY OF INTERACTION WITH OpenFlow CONTROLLER | |
| US9374308B2 (en) | Openflow switch mode transition processing | |
| WO2020119317A1 (en) | Message forwarding method and apparatus, storage medium, and electronic apparatus | |
| US10909122B2 (en) | Using modified blockchain concepts to guarantee no tuple loss and/or no tuple corruption | |
| US20160094443A1 (en) | Protocol independent multicast (pim) multicast route entry synchronization | |
| US20210012439A1 (en) | Systems and methods for adaptable location-based path optimization and entity selection | |
| US10951517B2 (en) | DCB-based path selection systems and methods for san traffic in lag topologies | |
| CN111107142A (en) | Service access method and device | |
| US11777973B2 (en) | Service plane optimizations with learning-enabled flow identification | |
| US20230138389A1 (en) | Forwarding table validation | |
| US11888751B2 (en) | Enhanced virtual channel switching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Applicant after: Inspur cloud Information Technology Co.,Ltd. Address before: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Applicant before: Tidal Cloud Information Technology Co.,Ltd. Address after: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Applicant after: SHANDONG INSPUR CLOUD INFORMATION TECHNOLOGY Co.,Ltd. Address before: 250100 S06 tower, 1036, Chao Lu Road, hi tech Zone, Ji'nan, Shandong. Applicant before: SHANDONG LANGCHAO YUNTOU INFORMATION TECHNOLOGY Co.,Ltd. Address after: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Applicant after: Tidal Cloud Information Technology Co.,Ltd. Address before: 250100 No. 1036 Tidal Road, Jinan High-tech Zone, Shandong Province, S01 Building, Tidal Science Park Applicant before: SHANDONG INSPUR CLOUD INFORMATION TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |