CN114675972A

CN114675972A - Method and system for flexibly scheduling cloud network resources based on integral algorithm

Info

Publication number: CN114675972A
Application number: CN202210381458.0A
Authority: CN
Inventors: 陈文智; 魏成坤; 陈义全; 徐天宇; 蒋骁翀; 张紫徽; 祝顺民; 李星; 陈子康; 杨博文
Original assignee: Zhejiang University ZJU
Current assignee: Zhejiang University ZJU
Priority date: 2022-04-12
Filing date: 2022-04-12
Publication date: 2022-06-28

Abstract

The invention discloses a cloud network resource flexible scheduling system and method based on an integral algorithm, which comprises a virtual switch module, a data acquisition module, an integral calculation module and a resource scheduling module; the virtual switch module is used as a core module for bearing the network function of the VM and is used for providing network service for the VM according to the CPU cycle ratio; the data acquisition module is used for collecting the actual CPU clock period number and the period ratio of the virtual switch consumed by each VM in each working time slice from the virtual switch module and transmitting the actual CPU clock period number and the period ratio to the integral calculation module; the integral calculation module is used for updating the integral value of each VM according to the input actual CPU clock periodicity and the period duty ratio and transmitting the updated integral value to the resource scheduling module; and the resource scheduling module is used for dynamically limiting the number of cycles and the cycle proportion of the CPU of the virtual switch consumed by each VM in the next working time slice according to the input integral value. The method ensures that a plurality of VM networks tend to have reasonable level of resource utilization.

Description

Cloud network resource flexible scheduling method and system based on integral algorithm

Technical Field

The invention belongs to the technical field of cloud data center networks, and particularly relates to a virtualized cloud network resource flexible scheduling method and system based on an integral algorithm.

Background

With the development of information technology, the trend of informatization of industrial enterprises is increasingly obvious, and the development of a plurality of industrial enterprises gradually tends to intelligent operation, namely the enterprises are in cloud. The cloud-on-enterprise refers to that the enterprise connects social resources, a shared platform, working contents and the like through an internet technology and a cloud computing technology, so that the whole process of applying information management infrastructure construction, a management method, a business process and the like is developed. As more and more enterprises dispute and cloud the business, the cloud service provider bears more and more operation pressure. On one hand, the total network flow of the cloud service provider is gradually increased; on the other hand, the service quality grades sold by cloud service providers are complicated. In order to ensure that enterprise users can obtain good service experience, a cloud service provider must reasonably schedule network resources according to traffic distribution characteristics and user preset service quality, otherwise, normal performance requirements of users cannot be met, and huge operation accidents that local network hot spots cause cloud server CPU overload and downtime can be caused.

Currently, mainstream cloud service providers all employ virtual large two-layer network technologies. The cloud service provider virtualizes a physical server (Host) into a plurality of virtual logical Servers (VMs) through a virtualization technology. For convenience of service management and deployment, cloud service providers need to incorporate a large number of VMs into the same two-layer broadcast domain. Because the traditional VLAN two-layer technology cannot support the number of hosts at tens of thousands or even hundreds of thousands of levels in a cloud data center, network protocols such as VXLAN, NVGRE, STT and the like are developed by cloud service providers to meet the requirements of crossing regions and crossing a center large two-layer. The core component implementing these protocols is the virtual switch. A virtual switch is a software application that allows network communication between VMs, and is typically deployed in a Host's system, working in conjunction with a virtual machine monitor (Hypervisor). Because the virtual switch is communicated with the virtual network cards of the multiple VMs through the software abstraction layer, the virtual switch is a direct interaction component for the VMs to perform network communication externally, and has a significant influence on the overall network performance of the VM cluster, the elastic resource guarantee mechanism of the virtual switch is one of important technologies for realizing service quality and performance isolation.

The flexible resource guarantee mechanism is a computing resource consumed by the virtual switch in processing each VM network message in a dynamic and reasonable scheduling manner. Currently, virtual switches face several challenges: firstly, since VMs deployed in the same Host often have different service specification indexes, a virtual switch is required to limit the speed of each VM in multiple dimensions, such as bit rate (BPS) and packet rate (PPS); secondly, the virtual switch needs to ensure isolation, that is, when one of the VMs generates abnormal traffic, the network service quality of the other VMs on the same Host is not affected; finally, the virtual switch needs to improve the overall utilization rate of Host computing resources as much as possible, and avoid that idle resources cannot be effectively utilized.

There are many algorithms and techniques related to flexible resource provisioning in the industry. For example, FairCloud and NetShare based on the Fair sharing model; elastic switch and Silo based on the Hose model, and so on. In addition, Google also proposes a PicNIC scheme for allocating virtual switch CPU cycles based on SLAs corresponding to VMs. Although the above schemes improve resource contention and quality of service to varying degrees, these methods still lack fine-grained scheduling means. One core problem is how to dynamically switch seamlessly between two different conditions, low load when idle and high load when busy.

Disclosure of Invention

In view of the foregoing, an object of the present invention is to provide a method and a system for flexibly scheduling cloud network resources based on an integral algorithm, so as to ensure that resource utilization of multiple VM networks tends to a reasonable level.

In order to achieve the above object, an embodiment of the present invention provides a cloud network resource flexible scheduling system based on an integral algorithm, including a virtual switch module, a data acquisition module, an integral calculation module, and a resource scheduling module;

the virtual switch module is used as a core module for bearing the network function of the VM and is used for providing network service for the VM according to the CPU cycle ratio;

the data acquisition module is used for collecting the actual CPU clock period number and the period ratio of the virtual switch consumed by each VM in each working time slice from the virtual switch module and transmitting the actual CPU clock period number and the period ratio to the integral calculation module;

the integral calculation module is used for updating the integral value of each VM according to the input actual CPU clock period number and the period ratio and transmitting the updated integral value to the resource scheduling module;

and the resource scheduling module is used for dynamically limiting the number of cycles and the proportion of cycles of the CPU of the virtual switch consumed by each VM in the next working time slice according to the input integral value, so as to realize resource allocation.

In one embodiment, the virtual switch module comprises a virtual network card interface module and a Netframe forwarding module;

the virtual network card interface module is used for providing an interface for carrying out network data communication with the VM;

the Netframe forwarding module is a DPDK-based user mode network protocol stack component, serves as a core module for data forwarding, and is used for realizing two-layer MAC address forwarding, namely, the two-layer MAC address forwarding is provided for the virtual network card interface module in a Netfilter Hook mode.

In one embodiment, the data collection module collects the number of virtual switch CPU clock cycles consumed by each VM in each working time slice and the proportion of the number of the virtual switch CPU clock cycles to the total number of the virtual switch CPU clock cycles from the Netframe forwarding module.

In the integral calculation module of one embodiment, the process of updating the integral value of each VM includes:

presetting integral parameters BASE, MAX and MIN for each VM, wherein BASE is the basic consumption CPU period ratio, and MAX and MIN are the maximum and minimum consumption CPU period ratios respectively;

for each VM, comparing whether the consumed CPU cycle ratio is smaller than a BASE value, and when the CPU cycle ratio is smaller than the BASE value, increasing the integral by X1; when the CPU cycle ratio is larger than the BASE value, the integral is decreased by X2 to obtain an integral updating result, wherein X1 and X2 are preset increasing proportion and decreasing proportion, wherein X1 is smaller than X2, which indicates that the accumulation speed of the integral is slower than the consumption speed of the integral.

In the integration calculation module of an embodiment, when the integration value is 0, updating of the integration is not performed, and the integration value is kept to be 0.

In the resource scheduling module of an embodiment, if the input integral value is 0, the maximum CPU cycle occupied ratio consumed by the VM in the next second is set to be BASE; if the integral value is larger than 0, setting the maximum CPU occupation ratio consumed by the VM in the next second to be MAX; meanwhile, the minimum CPU cycle consumed by the VM in the next second must be guaranteed to be MIN.

In one embodiment, the resource scheduling module dynamically controls the CPU cycle proportion of the virtual switch consumed by each VM according to the input integral value and transmits the CPU cycle proportion to the virtual switch module;

and the virtual switch module provides network service with corresponding specification for the VM according to the received CPU cycle ratio.

In the integral calculation module of an embodiment, an initial integral value preset for each VM is specified according to a service index of a user, and a value of the initial integral value is 100-500.

In order to achieve the above object, an embodiment of the present invention provides a cloud network resource flexible scheduling method based on an integral algorithm, where the method employs the cloud network resource flexible scheduling system, and the scheduling method includes the following steps:

step 1, a virtual switch module is used for providing network service for each VM according to the CPU cycle ratio;

step 2, collecting the actual CPU clock period number and the period ratio of the virtual switch consumed by each VM in each working time slice from the virtual switch module by using a data acquisition module, and transmitting the actual CPU clock period number and the period ratio to an integral calculation module;

step 3, updating the integral value of each VM by using the actual CPU clock period number and the period ratio input by the integral calculation module, and transmitting the updated integral value to the resource scheduling module;

and 4, dynamically calculating the CPU cycle number and the cycle proportion of the virtual switch consumed by each VM in the next working time slice by using the resource scheduling module according to the input integral value, and transmitting the cycle number and the cycle proportion to the virtual switch module.

Compared with the prior art, the invention has the beneficial effects that at least:

the binding of the CPU consumption period ratio of the VM and the VM specification is realized through an integral algorithm, the problem that the Host resource is contended and robbed due to the concurrence of various index pressures is avoided, and the overall utilization rate of the Host resource is effectively improved;

the dynamic adjustment of the CPU consumption period ratio of each VM through an integral algorithm realizes the network burst transmission capability with unified dimensionality and can realize better performance isolation under the situation.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

Fig. 1 is a schematic structural diagram of a cloud network resource flexible scheduling system based on an integration algorithm according to an embodiment;

FIG. 2 is a schematic diagram of an integration updating process and a CPU cycle duty ratio control process based on the integration updating value according to an embodiment;

fig. 3 is a schematic diagram of mode condition switching of a flexible scheduling system for cloud network resources according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the detailed description and specific examples, while indicating the scope of the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.

Through observation of a cloud data center network flow mode and based on a scheduling guiding idea of 'over selling in idle time and low protection in busy time', the embodiment provides a cloud network resource flexible scheduling system and method based on an integral algorithm.

Fig. 1 is a schematic structural diagram of a cloud network resource flexible scheduling system based on an integration algorithm according to an embodiment. As shown in fig. 1, the cloud network resource flexible scheduling system based on the integral algorithm provided in the embodiment includes a virtual switch module, a data acquisition module, an integral calculation module, and a resource scheduling module, and dynamically allocates bandwidth resources of a virtual logic server (VM) on a physical server (Host) through the cooperation of these modules.

The virtual switch module is used as a core module for bearing the network function of the VM and is used for providing network services with corresponding specifications for the VM on the Host according to the CPU cycle ratio. Specifically, the virtual switch module comprises a virtual network card interface module and a Netframe forwarding module, wherein the virtual network card interface module is used for providing an interface for network data communication with the VM; the Netframe forwarding module is a DPDK-based user mode network protocol stack component, serves as a core module for data forwarding, and is used for realizing two-layer MAC address forwarding, namely, the two-layer MAC address forwarding is provided for the virtual network card interface module in a Netfilter Hook mode. The virtual network card interface module and the Netframe forwarding module both belong to software architecture.

The data acquisition module is used for collecting the actual CPU clock period number and the period ratio of the virtual switch consumed by each VM in each working time slice from the Netframe forwarding module included by the virtual switch module, and transmitting the actual CPU clock period number and the period ratio to the integral calculation module. The working time slice is a period of working time, and can be 1 second.

The integral calculation module is used for updating the integral value of each VM according to the input actual CPU clock period number and the period ratio, and transmitting the updated integral value to the resource scheduling module. Specifically, the process of updating the integration value of each VM includes:

presetting integral parameters BASE, MAX and MIN for each VM, wherein BASE is the basic cycle proportion of the VM consumed by the CPU of the virtual switch under the default service quality, for example, the BASE can be set to be 0.1, and the BASE integral at the moment corresponds to 1 Gbps; MAX is the maximum period proportion of the VM consuming the CPU of the virtual switch during the burst transmission, for example, MAX may be set to 0.3, and the MAX integral at this time corresponds to 3Gbps, which represents the upper limit of the network performance of the VM when the integral is available; MIN is the minimum cycle proportion of the virtual switch CPU that the VM can consume in case of high load or point exhaustion, for example MIN can be set to 0.02, corresponding to 200Mbps, which represents the minimum quality of service of the VM. It should be noted that the specific value of BASE is related to the hardware specification of the specific physical server, and needs to be calibrated after the benchmark test.

The point calculation module simultaneously maintains each served VM point, an initial point of the point is determined as INIT, and the INIT is made according to a service index of a user and is generally 100-500. As shown in fig. 2, for each VM, whether the CPU cycle proportion consumed is smaller than the BASE value is compared, and when the CPU cycle proportion is smaller than the BASE value, the integral is increased by X1; when the CPU cycle occupation ratio is larger than the BASE value, the integral is decreased by X2 to obtain an integral updating result, wherein X1 and X2 are preset increasing proportion and decreasing proportion, wherein X1 is smaller than X2, the value of X1 is generally set to 0.1, the value of X2 is set to 1.0, and the accumulation speed of the integral is slower than the consumption speed of the integral.

It should be specially noted that, in the integral calculation module, when the integral obtained in each working time slice is 0, no processing is performed, and the integral is kept at 0.

The resource scheduling module is configured to dynamically limit, according to an input integral value, a CPU cycle number and a cycle ratio of a virtual switch consumed by each VM in a next working time slice, as shown in fig. 2, the specific process includes: if the input integral value is 0, setting the maximum CPU cycle occupation ratio consumed by the VM in the next second as BASE; if the integral value is larger than 0, setting the maximum CPU occupation ratio consumed by the VM in the next second to be MAX; meanwhile, the minimum CPU cycle consumed by the VM in the next second must be guaranteed to be MIN. And transmitting the number of the CPU cycles of the virtual switch and the cycle ratio consumed by each VM in the next working time slice under dynamic control to the virtual switch module, and providing network services with corresponding specifications for the VMs by the virtual switch module according to the received CPU cycle ratio.

The embodiment also provides a cloud network resource flexible scheduling method adopting the cloud network resource flexible scheduling system, which comprises the following steps:

step 1, a virtual switch module is used for providing network service for each VM according to the CPU cycle ratio.

In the embodiment, after the program is loaded, each module completes parameter initialization, and the integral corresponding to each VM is set as an initial value; and starting to execute the program process and providing network services for the VM.

And 2, collecting the actual CPU clock period number and the period ratio of the virtual switch consumed by each VM in each working time slice from the virtual switch module by using the data acquisition module, and transmitting the actual CPU clock period number and the period ratio to the integral calculation module.

In an embodiment, at the end of one working time slice (one second), a query request is issued to the Netframe forwarding module; the Netframe forwarding module counts the number of CPU cycles consumed by each VM in the last second, counts the total number of CPU cycles (including idle CPU cycles) in the past second, and returns the result to the data acquisition module.

And 3, updating the integral value of each VM by using the actual CPU clock period number and the period ratio input by the integral calculation module, and transmitting the updated integral value to the resource scheduling module.

In the embodiment, integral parameters BASE, MAX and MIN are preset for each VM, whether the consumed CPU cycle ratio is smaller than a BASE value or not is compared for each VM, and when the CPU cycle ratio is smaller than the BASE value, the integral is increased by X1; when the CPU cycle ratio is larger than the BASE value, the integral is decreased by X2 to obtain an integral updating result, wherein X1 and X2 are preset increasing proportion and decreasing proportion, wherein X1 is smaller than X2, which indicates that the accumulation speed of the integral is slower than the consumption speed of the integral.

In the embodiment, receiving an integral corresponding to each VM, and if the input integral is 0, setting the maximum CPU cycle occupied ratio consumed by the VM in the next second as BASE; if the integral value is larger than 0, setting the maximum CPU occupation ratio consumed by the VM in the next second to be MAX; meanwhile, the minimum CPU cycle consumed by the VM in the next second must be guaranteed to be MIN.

As shown in fig. 3, the cloud network resource flexible scheduling method based on the integral algorithm and the system design scheduling scenario: full light load, partial high load, full high load.

(one) completely light load

This condition corresponds to state 1 of fig. 3. Under this condition, the actual network overhead of all VMs reaches a BASE value, that is, the network rate of each VM is maintained at about a preset value, and after a period of time, each VM accumulates a certain number of integrals, and the VM can obtain network resources (but not exceeding MAX) that exceed the BASE limit. At this time, the whole system can tolerate moderate burst transmission and high-speed stream, so as to improve the whole resource utilization rate of the system.

Part two high load

This condition corresponds to state 2 of fig. 3. Under the working condition, the actual network overhead of part of the VMs reaches the BASE value, while the other part of the VMs are in burst transmission or high-speed flow, and the total network capacity of all the VMs is smaller than or close to the total capacity of the Host. For the VM which is carrying out burst transmission and has actual network cost larger than BASE, the previous accumulated points are consumed, so that the network speed of the VM is limited to be close to the BASE value by the virtual switch in a certain time, and the speed limit suppression function is realized. Meanwhile, since the credits correspond to the computational resources (CPU cycles) of the virtual switch, all VMs can obtain network resources proportional to the credits, so the VMs can still maintain performance isolation.

(III) completely high load

This condition corresponds to state 3 of fig. 3. Under this condition, the total network capacity of all VMs is higher than that of Host, so network resource competition among VMs must occur. At this time, the integration algorithm is no longer applicable, and the virtual switch will automatically allocate network resources according to the MIN value corresponding to each VM, so as to ensure the availability of VM network services. If the network hotspot disappears after a period of time, the system automatically returns to the two working conditions; if the network hot spot still exists in the system after a period of time, the partial VM on the Host is actively referred to be migrated to other hosts, so that the network fault caused by overheating of the CPU is avoided.

The above-mentioned embodiments are intended to illustrate the technical solutions and advantages of the present invention, and it should be understood that the above-mentioned embodiments are only the most preferred embodiments of the present invention, and are not intended to limit the present invention, and any modifications, additions, equivalents, etc. made within the scope of the principles of the present invention should be included in the scope of the present invention.

Claims

1. A cloud network resource flexible scheduling system based on an integral algorithm is characterized by comprising a virtual switch module, a data acquisition module, an integral calculation module and a resource scheduling module;

2. The cloud network resource elastic scheduling system based on the integral algorithm according to claim 1, wherein the virtual switch module comprises a virtual network card interface module, a Netframe forwarding module;

3. The system according to claim 2, wherein the data collection module collects the number of CPU clock cycles of the virtual switch consumed by each VM in each operating time slice from the Netframe forwarding module and the ratio of the number of CPU clock cycles to the total number of cycles.

4. The cloud network resource flexible scheduling system based on integration algorithm of claim 1, wherein the process of updating the integration value of each VM in the integration calculation module comprises:

5. The system according to claim 1, wherein in the integral calculation module, when the integral value is 0, the integral value is not updated and is kept to be 0.

6. The system according to claim 1, wherein if the input integral value is 0, the resource scheduling module sets a maximum CPU cycle occupied ratio consumed by the VM in the next second to be BASE; if the integral value is larger than 0, setting the maximum CPU occupation ratio consumed by the VM in the next second to be MAX; meanwhile, the minimum CPU cycle consumed by the VM in the next second must be guaranteed to be MIN.

7. The cloud network resource flexible scheduling system based on the integral algorithm of claim 1, wherein the resource scheduling module dynamically controls the ratio of the virtual switch CPU cycles consumed by each VM according to the input integral value to be transmitted to the virtual switch module;

8. The cloud network resource flexible scheduling system based on the integral algorithm as claimed in claim 1, wherein in the integral calculation module, an initial integral value preset for each VM is specified according to a service index of a user and has a value of 100-500.

9. A cloud network resource flexible scheduling method based on an integral algorithm, wherein the method adopts the cloud network resource flexible scheduling system of any one of claims 1 to 8, and the scheduling method comprises the following steps: