TW201523238A - Method and system for regulating monitor data of cloud platform - Google Patents

Abstract

The present invention discloses a method for regulating monitoring data of cloud platform. The method includes the following steps: a monitoring status collector collects each monitoring data in every computing process of a plurality of virtual machines. The monitoring status collector samples the monitoring datum according to a reference matrix and currently available network bandwidth and gets a plurality of monitoring coefficient datum. The monitoring status collector transmits the monitoring coefficient datum to a master. The master processes the monitoring coefficient datum according to the reference matrix and reconstructs a plurality of monitoring reconstruction datum. The master determines whether there exists abnormal computing process in the computing processes of the virtual machines or not according to the reconstructed monitoring datum.

Description

Monitoring method and system for monitoring data of cloud platform

The invention relates to a cloud platform computing method, in particular to a monitoring data monitoring method for a cloud platform.

In recent years, the demand for multimedia applications and video surveillance has increased, resulting in a huge increase in the amount of video data. Therefore, the demand for using a computer to process a large amount of video data is increasing. For example, (1) for video surveillance video, license plate recognition or flow counting, etc., to assist in statistical analysis; (2) quality enhancement processing for video images to improve (4) video enrichment for video footage to reduce the time spent on viewing the video; (4) time and place to retrieve a particular object from a large number of videos; or (5) for high resolution video Content is transcoded in a compressed format to reduce the amount of data for applications such as storage transfers. The above-mentioned applications for identification, statistics, enhancement, enrichment, retrieval or transcoding of video content are generally referred to as Video Processing (VP). With the continuous improvement of video resolution, the traditional stand-alone computing architecture has been unable to load the computational demands of video processing. Therefore, a cloud platform with computing resource scalability is needed to meet these large and ever-increasing computing needs.

The cloud platform will use multiple virtual computing machines to generate a large number of virtual computing machines. Cloud platform and based on user needs and application services The computing needs, the virtual machines (CPU, memory, storage space, network bandwidth) of different resources are configured to perform various video processing. The conventional cloud video processing cuts a single movie into multiple sets of clips and evenly distributes them to the cloud processing program for processing. The overall computing time depends on the slowest processing procedure. However, video processing has many causes (virtual machine computing resources, video processing complexity, and processed video content differences), and the real-time processing status of the processing program must be constantly monitored to dynamically allocate computing resources.

Different video because of the complexity of various video processing operations The computing resource requirements for processing functions are also different. Even the same video processing function may be due to the fact that the video source changes over time. At different times, the number of objects appearing in the camera screen at the same time is different, and the required computing resources also change.

How to monitor various video processing functions simultaneously on a virtual machine The processing state to find out the processing of insufficient resources. Or how to find out the worst performing processor set in the system based on the results of monitoring the processing status, and allocate more resources to these programs to optimize the overall computing performance, which has become an important issue. Therefore, the present invention provides a monitoring method for monitoring data of a cloud platform. For a large number of data processing applications, the operating resources are effectively utilized to improve the overall computing performance of the system while taking into account the available transmission bandwidth and the ability to find the most abnormal processing procedures.

An embodiment of the present invention provides a monitoring data of a cloud platform Control methods. The control method includes the following steps: collecting, by a monitoring status collector, each monitoring data of each computing program in a plurality of virtual machines. The monitoring status collector, based on a reference matrix and currently available network bandwidth, the monitoring data Sampling is performed to obtain a plurality of monitoring coefficient data. The monitoring status collector transmits the monitoring coefficient data to a main control terminal. The main control unit processes the monitoring coefficient data according to the reference matrix to reconstruct the complex monitoring and reorganizing data. Based on the monitoring of the reorganization data, the main control unit determines whether an arithmetic program capable of being abnormal in each of the arithmetic programs in the virtual machines is determined.

Embodiments of the present invention provide a monitoring data monitoring system for a cloud platform. The control system includes: a plurality of virtual machines, a monitoring state collector, a first register, a master, and a second register. The monitoring status collector is coupled to the virtual machines to collect various monitoring data of each computing program in the virtual machines. The monitoring status collector samples the monitoring data according to a reference matrix and the currently available network bandwidth to obtain a plurality of monitoring coefficient data. The first register is coupled to the monitoring state collector for storing the reference matrix. The master receives the monitoring coefficient data from the monitoring status collector. The main control unit processes the monitoring coefficient data according to the reference matrix to reconstruct the complex monitoring and reorganizing data. Based on the monitoring of the reorganization data, the main control unit determines whether an arithmetic program capable of being abnormal in each of the arithmetic programs in the virtual machines is determined. The second register is coupled to the master, and receives and stores the reference matrix from the monitoring state collector.

10‧‧‧Cloud Platform

100‧‧‧Virtual Machine Group

101‧‧‧Monitor Status Collector

102‧‧‧First register

103, 104, 105, 106‧‧‧ virtual machines

110‧‧‧Master

111‧‧‧Second register

Figure 1 shows a cloud platform 10 in accordance with one embodiment of the present invention.

2A and 2B are flowcharts illustrating a method for monitoring monitoring data of the cloud platform 10.

Figure 3 is a flow chart illustrating how the master 110 responds to the above The operation program that determines the processing state abnormality performs the resource allocation operation.

Figure 4 shows the network bandwidth required to use the monitoring data control method of the present invention at different compression rates for the monitoring data.

Figure 1 shows a cloud platform 10 in accordance with one embodiment of the present invention. As shown in FIG. 1 , the cloud platform 10 of the present invention includes at least one virtual machine group 100 and one host 110. The virtual machine group 100 and the host terminal 110 transmit data through a physical network, and the bandwidth of the physical network is limited. The virtual machine group 100 includes a monitoring status collector 101, a first temporary register 102, and a plurality of virtual machines 103-106. The monitoring status collector 101 is coupled to the virtual machines 103-106, wherein each virtual machine can execute one to several computing programs. The first register 102 is located in the monitoring state collector 101 for storing data from the monitoring state collector 101. The second register 111 is located in the host 110 for storing the data of the host 110 and the data from the monitoring status collector 101.

It should be noted that the present invention is not limited thereto. For example, the cloud platform 10 can also include a plurality of virtual machine groups, wherein each virtual machine group has a corresponding one of a monitoring status collector, a first temporary register, and a plurality of virtual machines; the first temporary storage The device 102 and the second register 111 can also be externally connected to the monitoring state collector 101 and the master terminal 110, respectively.

2A and 2B are flowcharts illustrating a method for monitoring monitoring data of the cloud platform 10. In step S210, the monitoring status collector 101 periodically collects each monitoring data of each computing program in the virtual machines 103-106, and records a first number N of all the computing programs (or the monitoring data). Lift For example, assuming that one, three, one, and two arithmetic programs (a total of seven arithmetic programs) are executed in the virtual machines 103-106, the monitoring state collector 101 periodically collects seven monitoring materials. At this time, the first number N is 7.

In step S220, the monitoring state collector 101 is N by one N reference matrix L _NxN according to the first number N construction dimension; then, the monitoring state collector 101 stores the reference matrix L _NxN in the first temporary The reference matrix L _{NxN is} transmitted to the second temporary storage unit 111 in the main control terminal 110 through the physical network. Therefore, the monitoring state collector 101 has the same reference matrix L _NxN as the master terminal 110. The reference matrix L _NxN is a structured random matrix (Structured Random Matrix), which is used as a reference for sparse matrix operations. The reference matrix L _NxN may be generated by a Random Partial Fourier Matrix or a Gaussian Distribution in a random manner, but is not limited thereto.

In step S230, the monitoring state collector 101 samples the N monitoring data x ₁ ~ x _N according to the reference matrix L _NxN and the network bandwidth currently available to the physical network to obtain M monitoring coefficient data y ₁ ~y _M , where M is the number of sampling coefficients and M is not greater than N. A more detailed implementation of step S230 can be found in steps S231-S233 of the embodiment of FIG. 2B as follows: in step S231, the monitoring state collector 101 first collects the N monitoring data x ₁ ~ x _N The conversion is converted to N monitored values x' ₁ ~ x' _N having sparse characteristics, wherein the sparse characteristic represents that the N monitored values x' ₁ ~ x' _N contain only a small number of non-zero values. This is because the number of abnormal operation programs in the cloud platform 10 is relatively small, so the N monitoring data can be normalized into N monitoring values x' ₁ with sparse characteristics by using a sparse coding (Sparse Coding) algorithm. ~x' _N . Related algorithms are currently known as Linear Generative Model, Feature-Sign, Least Angle Regression, Grafting, and QP Solver. For example, the corresponding one of the transformation matrices W _NxN can be calculated using the Linear Generative Model. The monitoring state collector 101 further normalizes the N monitoring data x ₁ ~x _N using the conversion matrix W _NxN to obtain the N monitored values x' ₁ ~x' _N , wherein the normalized conversion The expression is X' _Nx1 = W _NxN ^T X _Nx1 = [x' ₁ , x' ₂ , ..., x' _N ] ^T .

In step S232, the monitoring state collector 101 determines the number M of sampling coefficients to be sampled according to the network bandwidth currently available to the physical network. The smaller the value of M, the higher the compression ratio of such monitoring data; and vice versa. At this time, the B _{MAX is} defined as the network bandwidth limit that the cloud platform 10 configures to transmit the monitoring data to the virtual machine group 100 (which may be a fixed ratio of the network bandwidth, for example, 1% of the 1 GB network bandwidth). ), and B _FREE is the currently available network residual bandwidth, which can be obtained by known estimation techniques, where B _MAX and B _FREE are in bytes/sec. Under the period of transmitting monitoring data once per second, the amount of data of each monitoring data is D bytes, then the number of sampling coefficients M can be calculated according to the following two conditions: (1) When B _{FREE is} less than B _MAX , M=╚ B _FREE /D╛; (b) Otherwise, M=╚B _MAX /D╛, where ╚A╛ indicates that the logarithm A is unconditionally rounded off to get one of the integers. Therefore, embodiments of the present invention can dynamically adjust the compression ratio of the monitoring data by the network bandwidth currently available to the physical network.

In step S233, the monitoring state collector 101 extracts the first M columns of the reference matrix L _NxN according to the number M of sampling coefficients to form a sub-reference matrix L _MxN . Then, the monitoring state collector 101 obtains M monitoring coefficient data y ₁ ~ y _M according to the sub-reference matrix L _MxN and the N monitored values x' ₁ ~ x' _N , wherein the operation process is as follows: Y _Mx1 ( [y ₁ , y ₂ ,...,y _M ] ^T )=A _MxN X' _Nx1 .

In step S240, the monitoring status collector 101 transmits the M monitoring coefficient data y ₁ ~ y _M to the main control terminal 110 through the physical network.

In step 250, the master terminal 110 knows the magnitude of the sampling coefficient M of the sampling state collector 101 according to the received M monitoring coefficient data y' ₁ y' _M , and takes out the reference. The first M columns of the matrix L _NxN form the same sub-reference matrix A _MxN . The master 110 further reconstructs N monitored recombination data x ^~ ₁ ~ x ^~ _N according to the sub-reference matrix A _MxN and the received M monitoring coefficient data y' ₁ ~ y' _M , wherein the reconstruction process is According to the following formula:

Although there are infinitely many sets of solutions x ^~ ₁ ~ x ^~ _N satisfying the above-mentioned constraint condition Y _Mx1 = A _MxN X ^~ _Nx1 , the N monitored values x' ₁ ~ x' _N having the normalized conversion described above are sparse The characteristic of (Sparse) is that most of the elements have a value of 0 or close to zero. Theoretically, the sum of the absolute values of the N monitored values x' ₁ ~ x' _N is small. Therefore, in order to obtain a solution close to or equal to x' ₁ ~x' _N , the solution with the smallest sum of absolute values among all possible solutions of the above formula can be selected as the N monitored recombination data x ^~ ₁ ~ x ^~ _N . In the present embodiment, since the minimum of solving a linear optimization problem belongs to ₍₁ 1 -Minimization) may be utilized in the correlation algorithm (e.g. IRLS (Iteratively Re-Weighted Least Squares ) algorithm) to obtain the N monitoring recombination data _x ^~ ₁ ~ _x ^~ _N satisfies X ^~ _Nx1 = A ^T _NxM Y _Mx1 , where X ^~ _Nx1 = [x ^~ ₁ , x ^~ ₂ , ..., x ^~ _N ] ^T .

In step S260, the master terminal 110 determines whether the operation programs are abnormal according to the N monitored reorganization data x ^~ ₁ ~ x ^~ _N. Since the monitoring data corresponding to one of the processing states P _i (i=1~N) is converted by the above normalization, the monitored value x' _i will be close to 0, and the monitored value x' _i is sampled again. And after being transmitted to the master 110 for reorganization, it will also be close to zero. By using the above characteristics, it can be determined that if the monitoring recombination data value | x ^~ _i | corresponding to the operation program P _{i is} less than a threshold value ε, it can be determined that the processing state of the operation program P _i is normal; An abnormal multiplicity operation program, wherein the threshold value ε can be adjusted according to the characteristics of the monitoring data.

In addition, since the data x' ₁ ~x' _N after the normalization of the monitoring data has the characteristics of sparse (Sparse), the greater the absolute value of the processing data, the larger the absolute value of the monitoring recombination data | x ^~ _i | In addition, the accuracy of the data reconstruction process described in step S250 depends on the size of the sampling coefficient number M, and the larger the M, the more accurate the reconstruction result. When there is only a small amount of M, the monitored value of one of the N monitored values x' ₁ ~ x' _N will be reconstructed. As the number M of sampling coefficients increases, the remaining monitored values are reconstructed one by one from large to small.

Therefore, even if the number M of sampling coefficients is small, the sparsity characteristic can be reflected in the N monitored recombination data values x ^~ ₁ ~ x ^~ _N , and the abnormality of the processing state is monitored by the arithmetic program absolute value | x ^~ _i | The higher the probability, the greater the probability. Based on this principle, although the use of less network bandwidth transmission allows the host 110 to obtain only a small amount of sampled data, the operating program with the most abnormal processing status can be found, and the cloud platform 10 can be improved. Operational efficiency.

In step S270, the master terminal 110 performs an operation resource allocation operation on the operation program for determining the abnormality of the processing state. The more detailed operation resource allocation process can be found in the embodiment of FIG. 3 below.

Fig. 3 is a flow chart for explaining how the master terminal 110 performs resource allocation for the above-described arithmetic program for determining the abnormality of the processing state. In step S310, the master terminal 110 sorts the N operation programs according to the N monitored recombination data absolute values |x ^~ ₁ |~|x ^~ _N | size. In step S320, the master terminal 110 determines whether the N arithmetic programs have been processed. If yes, the process ends; if no, the process proceeds to step S330. In step S330, the master terminal 110 selects an unprocessed and abnormally processed operating program (i.e., one of the highest values of the absolute values | x ^~ _i | of the monitored recombined data) from all the arithmetic programs. In step S340, the master terminal 110 determines whether the arithmetic program whose processing state is abnormal needs to be processed. If yes, go to step S350; if no, go to step S320.

In step S350, the master terminal 110 determines that the processing state is abnormal. Has the algorithm been stopped? If yes, go to step S360; if no, go to step S370. In step S360, the master terminal 110 notifies the virtual machine in which the processing state abnormality is located to restart the arithmetic program of the processing state abnormality, and returns to step S320. In step S370, the host 110 checks whether the arithmetic program whose processing state is abnormal has sufficient computing resources. If yes, go back to step S320; if no, go to step S380. In step S380, the master terminal 110 allocates a large number of computing resources to the arithmetic program of the processing state abnormality.

In an embodiment of the present invention, the monitoring state collector 101 collects monitoring data of five operating programs of the virtual machines 103-106, respectively X _5x1 =[ _x1 , x ₂ , x ₃ , x ₄ , x ₅ ] ^T = [30, 29, 30, 28, 6] ^T , wherein the value of each monitoring data represents the picture data of the frame that can be processed by the operation program per second, but is not limited thereto. From the characteristics of the monitoring data, it can be known that the monitoring data values of the first four processing states with normal processing states are close, and the monitoring data value of the fifth processing state abnormal computing program is 6. As in step S231, the monitoring state collector 101 normalizes the above five monitoring data into having a Sparse Coding algorithm (for example, performing the normalization conversion using the conversion matrix W _NxN obtained by the Linear Generative Model). 5 monitored values for sparse characteristics. As in the above step S231, the preferred embodiment of the present invention performs the normalization operation using the conversion matrix W _NxN of the sparse coding. However, in order to make the present invention easier to understand, the present embodiment performs a normalization operation by replacing the conversion matrix W _NxN with a common conversion matrix P _5x5 . The monitoring state collector 101 obtains 5 monitored values X ^' _5x1 = P _5x5 X _5x1 = [5.4, 4.4, 5.4, 3.4, -18.6] ^T , where P _5x5 = [0.8, 0.8, 0.8, 0.8, 0.8; 0.8 , 0.8, 0.8, 0.8, 0.8; 0.8, 0.8, 0.8, 0.8, 0.8; 0.8, 0.8, 0.8, 0.8, 0.8; 0.8, 0.8, 0.8, 0.8, 0.8].

Then, the monitoring state collector 101 randomly generates a reference matrix L _5x5 = [0.2428, 0.1958, 0.0593, 0.0911, 0.4024; 0.0267, 0.1253, 0.2437, 0.2435, 0.0974; 0.2943, 0.4406, 0.3702, 0.3913, 0.0344; 0.3979, 0.2722 , 0.3743, 0.0541, 0.2121; 0.0490, 0.1135, 0.3727, 0.3830, 0.3734], and the reference matrix L _{5x5 is} transmitted to the second register 111 in the main control terminal 110 for storage. The monitoring state collector 101 determines that the number of sampling coefficients M is 1 according to the current network bandwidth, and takes the first column of the reference matrix L _{5x5 to} obtain a sub-reference matrix A _1x5 = [0.2428, 0.1958, 0.0593, 0.0911, 0.4024]. The monitoring state collector 101 further calculates a monitoring coefficient data Y _1x1 = A _1x5 X ^' _5x1 = [y ₁ ] = [-4.6818] according to the sub-reference matrix and the monitored values, and transmits the same to the main control. End 110. Since the original monitoring data has five values and only one value remains after sampling, the compression ratio is (NM)/N=4/5=0.8.

Finally, after receiving the monitoring coefficient data Y _1x1 , the main control terminal 110 reconstructs five monitoring and recombining data X ^~ _5x1 = [0, 0, 0, 0, -11.6348] ^T by using the above step S250. In the same step S260 as above, the master _{terminal 110} determines the absolute value of the first four monitored recombinations in the reconstructed monitored reconstructed data X ^~ _{5x1 to} be less than 0, and determines the threshold of the first four operations. The processing state is normal, and the master terminal 110 monitors the absolute value of the recombined data (|x ^~ ₅ |=11.6348) by the fifth operation program, which is greater than a threshold value ε of the definition, and determines the fifth The arithmetic program is an arithmetic program that handles state abnormalities. Therefore, the monitoring coefficient data Y _1x1 is correctly reconstructed to determine the fifth arithmetic program in which the processing state is abnormal.

Figure 4 shows the use of the monitoring data control method of the present invention in The network bandwidth required for monitoring data at different compression rates. In this embodiment, the monitoring state collector 101 samples the monitoring data according to six different compression ratios (NM)/N, and determines which compression ratio monitoring coefficient data to transmit according to the current bandwidth limitation. To the master terminal 110, wherein the monitoring state collector 101 transmits the monitoring coefficient data with a compression ratio of 0.89, the master terminal 110 can still find one of the most abnormal processing states of the processing state. As can be seen from Figure 4, the network bandwidth required to transmit uncompressed monitoring data is 320 Mb/sec. After applying the monitoring data monitoring method of the present invention, the network bandwidth required for transmitting the compressed monitoring data is reduced to 35.2 to 92.8 Mb/sec. As a result, the network bandwidth requirement for transmitting the monitoring data by the cloud platform 10 is greatly reduced. In addition, The cloud platform 10 can determine different compression ratios (ie, different number of sampling coefficients M) of the post-sampling monitoring coefficient data according to the currently available network bandwidth.

The present invention has been described above in terms of preferred embodiments, so that those skilled in the art can understand the present invention more clearly. However, those of ordinary skill in the art will appreciate that they can be readily based on the present invention, designing or modifying the processes and using the different cloud platforms for the same purpose and/or achieving the same advantages of the embodiments described herein. Therefore, the scope of the invention is defined by the scope of the appended claims.

Claims (12)

A monitoring method for monitoring data of a cloud platform, comprising: collecting, by a monitoring state collector, each monitoring data of each computing program in a plurality of virtual machines; and the monitoring state collector, according to a reference matrix and currently available network bandwidth Sampling the monitoring data to obtain a plurality of monitoring coefficient data; the monitoring state collector transmits the monitoring coefficient data to a main control terminal through a physical network; and the main control terminal processes the reference matrix according to the reference matrix The monitoring coefficient data is used to reconstruct the complex monitoring and reorganizing data; and the main controller determines whether the operating program in the virtual machine is abnormal or not according to the monitoring of the reorganization data. The method for monitoring monitoring data according to claim 1, wherein the monitoring state collector further constructs the reference matrix according to the first number of the operating procedures, and transmits the reference matrix to the main control terminal. The method for monitoring monitoring data according to claim 2, wherein the monitoring status collector further includes converting the monitoring data into a plurality of monitoring values having sparse characteristics, wherein the sparse characteristic represents These monitored values contain only a few non-zero values. For example, in the monitoring data monitoring method described in claim 3, wherein the monitoring state collector samples the monitoring data, and determines the second number of the monitoring coefficient data according to the currently available network bandwidth. And determining a sub-reference matrix of the reference matrix according to the second number, and the monitoring state collector further converts the monitoring values having sparse characteristics according to the sub-reference matrix And changing to the monitoring coefficient data, wherein the second number is less than the first number. For example, in the method for controlling monitoring data according to item 4 of the patent application scope, wherein the main control terminal reconstructs the monitoring and reorganizing data, the sub-reference matrix of the reference matrix is further determined according to the second number of receiving the monitoring coefficient data. The master further reconstructs the monitored reorganization data according to the sub-reference matrix. The method for monitoring monitoring data according to claim 1, wherein the main controller performs a resource allocation operation on the performance program of the performance abnormality. A cloud platform system includes: a plurality of virtual machines; a monitoring state collector coupled to the virtual machines to collect various monitoring data of each computing program in the virtual machines, the monitoring state collector according to a reference matrix and current The network bandwidth is used to sample the monitoring data to obtain a plurality of monitoring coefficient data; a first temporary register coupled to the monitoring state collector for storing the reference matrix; and a master terminal The physical network receives the monitoring coefficient data from the monitoring status collector; the main control unit processes the monitoring coefficient data according to the reference matrix to reconstruct a plurality of monitoring and recombining data; the main control unit monitors the reorganization data according to the monitoring Determining whether an arithmetic program capable of being abnormal in each of the operating programs in the virtual machine; and a second register coupled to the master to receive and store the reference matrix from the monitoring state collector. The cloud platform system of claim 7, wherein the monitoring state collector further constructs the reference matrix according to the number of operation programs, and transmits the reference matrix to the second temporary register. The cloud platform system of claim 8, wherein the monitoring status collector further includes converting the monitoring data into a plurality of monitoring values having sparse characteristics, wherein the sparse characteristic represents the The monitored values contain only a few non-zero values. The cloud platform system of claim 9, wherein the monitoring status collector determines the second number of the monitoring coefficient data according to the currently available network bandwidth when sampling the monitoring data. And determining, according to the second number, a sub-reference matrix of the reference matrix, wherein the monitoring state collector further converts the monitored values having the sparse characteristics into the monitoring coefficient data according to the sub-reference matrix, wherein the second number is less than The first number. The cloud platform system of claim 10, wherein when the master rebuilds the monitoring and reorganizing data, determining the sub-reference matrix of the reference matrix according to the second number of receiving the monitoring coefficient data, The master further reconstructs the monitored reorganization data according to the sub-reference matrix. The cloud platform system according to claim 7, wherein the master terminal performs a resource allocation action on the performance program of the performance abnormality.