WO2018199372A1 - Monitoring apparatus for monitoring high-speed streaming data processing system, and method therefor - Google Patents

Abstract

An embodiment of the present invention provides a monitoring apparatus for monitoring a high-speed streaming data processing system, the monitoring apparatus comprising: a database for storing, at unit time intervals, processing speed information and fault occurrence information of data processed until a first time point by the high-speed streaming data processing system; an underspeed information identifying unit for identifying, from the processing speed information, underspeed information at a time point when data is processed at a lower speed than a first reference speed; a relationship information calculating unit for calculating correlation information between the processing speed information and the fault occurrence information; a second reference speed calculating unit for receiving ratio information between the underspeed information and the fault occurrence information and calculating a second reference speed at a second time point after the first time point on the basis of the calculated correlation information and the received ratio information; and an alert issuing unit for, when a data processing speed at the second time point is lower than the second reference speed, outputting a performance anomaly occurrence alert.

Description

Monitoring device and method for monitoring streaming data processing system

The present invention relates to a monitoring apparatus and a method for monitoring a streaming data high-speed processing system, and more particularly, a threshold value that is variable to accurately detect performance degradation and failure occurrence of a system that collects and processes streaming data at high speed. It relates to a monitoring device for calculating the and a monitoring method of the device.

The amount of transmission and reception of video data on the Internet has increased dramatically, and users who use video data prefer to watch the video data one-time via streaming rather than permanently storing and playing the video data. In addition, a system that collects and processes streaming data must have a stable, high-speed processing capability for streaming data.

In order to stably maintain high-speed processing performance, the streaming data processing system is generally operated in conjunction with a monitoring device that monitors the occurrence of failure or failure of the system, and the system administrator (system administrator) uses the streaming data processing system through the monitoring device. Can be monitored efficiently.

In order for the monitoring device to determine whether the streaming data processing system is operating normally, a standard is required.If the system administrator assigns and manages the standard value every time, the monitoring device can effectively cope with changes in streaming data throughput or system failure. It is difficult and the burden of the system administrator is excessive.

In order to solve the above problems, the monitoring device actively adjusts the reference value for determining the normal operation of the streaming data processing system according to the passage of time and the system situation, the monitoring device that can minimize unnecessary intervention of the system administrator Is needed.

The technical problem to be solved by the present invention is to calculate the reference value for determining the performance degradation of the processing system for processing the streaming data through the accumulated performance indicators, the normal value of the processing system without human intervention through the calculated reference value The present invention provides a monitoring device and method for accurately determining whether or not to operate.

Monitoring device according to an embodiment of the present invention for monitoring the technical problem, in the monitoring device for monitoring the streaming data high-speed processing system, processing speed information of the data processed by the streaming data high-speed processing system to the first time point And a database storing failure occurrence information for each unit time. A speed shortening information detecting unit for grasping speed shortening information at a time when data is processed slower than a first reference speed in the processing speed information; A relationship information calculation unit for calculating correlation information between the processing speed information and the failure occurrence information; Receiving ratio information between the speed shortage information and the failure occurrence information and calculating a second reference speed at a second time after a first time based on the calculated correlation information and the received rate information; 2 standard speed calculation unit; And an alarm output unit configured to output a performance abnormality alarm when the data processing speed at the second time point is slower than the second reference speed.

Monitoring method according to another embodiment of the present invention for solving the technical problem, in the monitoring method for monitoring a streaming data high-speed processing system, the processing speed information and failure occurrence information of the data processed up to the first time point unit A speed shortening information identifying step of identifying speed shortening information at a time when data is processed slower than a first reference speed in the processing speed information by referring to a database stored for each time; Calculating relationship information between the processing speed information and the failure occurrence information; Receiving ratio information between the speed shortage information and the failure occurrence information and calculating a second reference speed at a second time after a first time based on the calculated correlation information and the received rate information; 2 standard speed calculation step; And an alarm output step of outputting a performance abnormality alarm if the data processing speed at the second time point is slower than the second reference speed.

According to the present invention, a second reference speed, which is a threshold value compared with a processing result of a streaming data processing system that processes big streaming data, is actively calculated and applied without the need for an administrator to intervene with a change in time. As a result, the performance change of the streaming data processing system can be monitored while minimizing the intervention of the system administrator, thereby reducing the burden on the system administrator.

In addition, when the monitoring device according to the present invention is applied to the streaming data processing system, it is possible to achieve the same or more system monitoring effect as before even with less manpower and time.

1 is a view schematically showing the overall configuration of a monitoring system according to the present invention.

2 is a block diagram of an example of a monitoring apparatus according to the present invention.

3 is a diagram schematically showing a second reference speed calculated according to the present invention.

4 shows an example of implementing the second reference speed calculation method according to the present invention using pseudo-code.

5 is a flowchart illustrating an example of a method for monitoring a streaming data high speed processing system according to the present invention.

Monitoring device according to an embodiment of the present invention for monitoring the technical problem, in the monitoring device for monitoring the streaming data high-speed processing system, processing speed information of the data processed by the streaming data high-speed processing system to the first time point And a database storing failure occurrence information for each unit time. A speed shortening information detecting unit for grasping speed shortening information at a time when data is processed slower than a first reference speed in the processing speed information; A relationship information calculation unit for calculating correlation information between the processing speed information and the failure occurrence information; Receiving ratio information between the speed shortage information and the failure occurrence information and calculating a second reference speed at a second time after a first time based on the calculated correlation information and the received rate information; 2 standard speed calculation unit; And an alarm output unit configured to output a performance abnormality alarm when the data processing speed at the second time point is slower than the second reference speed.

In the apparatus, the first reference speed may be set differently for each unit time.

In the apparatus, the failure occurrence information may include reference failure information when a preset reference failure code occurs and non-reference failure information at a time except when the reference failure code occurs.

In the apparatus, the received ratio information may be information on a ratio between the speed shortage information and the reference failure information.

In the apparatus, the relationship information calculation unit may calculate the correlation information by using a maximum likelihood method.

In the apparatus, the relationship information calculation unit may be configured to use logit transformation in calculating the correlation information.

In the apparatus, the underspeed information detecting unit uses a binary variable to distinguish the speed achievement information at the time when data is processed faster than the first reference speed in the processing speed information from the determined underspeed information. You can do

Monitoring method according to another embodiment of the present invention for solving the technical problem, in the monitoring method for monitoring a streaming data high-speed processing system, the processing speed information and failure occurrence information of the data processed up to the first time point unit A speed shortening information identifying step of identifying speed shortening information at a time when data is processed slower than a first reference speed in the processing speed information by referring to a database stored for each time; Calculating relationship information between the processing speed information and the failure occurrence information; Receiving ratio information between the speed shortage information and the failure occurrence information and calculating a second reference speed at a second time after a first time based on the calculated correlation information and the received rate information; 2 standard speed calculation step; And an alarm output step of outputting a performance abnormality alarm if the data processing speed at the second time point is slower than the second reference speed.

In the method, the first reference speed may be set differently for each unit time.

In the above method, the failure occurrence information may include reference failure information when a preset reference failure code occurs and non-reference failure information at a time except when the reference failure code occurs.

In the above method, the received ratio information may be information on a ratio between the speed shortage information and the reference failure information.

In the method, the calculating of the relationship information may include calculating the correlation information by using a maximum likelihood method.

In the method, the calculating of the relationship information may include using logit transformation to calculate the correlation information.

In the method, the step of identifying the underspeed information may include using a binary variable to distinguish the speed achievement information at the time when data is processed faster than the first reference speed in the processing speed information from the determined underspeed information. It can be characterized.

The present invention can provide a computer readable recording medium having recorded thereon a program for executing a method for solving the above technical problem.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. Effects and features of the present invention, and methods of achieving them will be apparent with reference to the embodiments described below in detail with reference to the drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various forms.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, and the same or corresponding components will be denoted by the same reference numerals, and redundant description thereof will be omitted. .

In the following embodiments, the terms first, second, etc. are used for the purpose of distinguishing one component from other components rather than a restrictive meaning.

In the following embodiments, the singular forms “a”, “an” and “the” include plural forms unless the context clearly indicates otherwise.

In the following embodiments, terms such as include or have means that the features or components described in the specification are present, and does not preclude the possibility of adding one or more other features or components in advance.

When a certain embodiment can be implemented differently, a specific process order may be performed differently from the described order. For example, two processes described in succession may be performed substantially simultaneously or in the reverse order of the described order.

1 is a view schematically showing the overall configuration of a monitoring system according to the present invention.

Referring to FIG. 1, the monitoring system 10 of the streaming data processing system according to the present invention includes a streaming data transmission system 110, a streaming data processing system 130, and a monitoring device 200. It can be seen that the transmission system 110 and the streaming data processing system 130 are connected through the communication network 150 to transmit and receive various data.

First, the streaming data transmission system 110 stores streaming data. When the streaming data transmission system 110 receives a request to transmit streaming data from the streaming data processing system 130, the streaming data transmission system 110 streams the streaming data to the streaming data processing system 130 through the communication network 150. Send the data. Here, the streaming data refers to various video and audio data that can be transmitted by dividing only a part of the entire data, and can play a part of the streaming data on the side receiving the part of the streaming data.

The streaming data processing system 130 transmits a streaming data transmission request to the streaming data transmission system 110 and receives and processes the streaming data. The streaming data processing system 130 is a storage device capable of semi-permanently storing the streaming data received from the streaming data transmission system 110, and a graphics control device for processing the streaming data and outputting it through an output device, audio control. The device may include all kinds of processors. In addition, the streaming data processing system 130 may record and store a history of processing the streaming data in a log for each unit time or event.

The monitoring device 200 according to the present invention performs a function of monitoring whether the performance of the streaming data processing system 130 or an error code is generated. The monitoring apparatus 200 directly receives the data processing result from the streaming data processing system 130 or analyzes the log of the streaming data processing system 130 to generate a performance degradation or failure code in the streaming data processing system 130. To detect that.

When the streaming data processing system 130 detects that a performance degradation or a failure code has occurred, the detected result may be visually output. The monitoring device 200 may be connected to the streaming data processing system 130 by wire or may be physically or logically included in the streaming data processing system 130 to operate.

The streaming data transmission system 110 and the streaming data processing system 130 transmit and receive various data through the communication network 150, where the communication network 150 includes various wired and wireless communication networks such as a general telephone network, a data network, and a mobile communication network. do.

2 is a block diagram of an example of a monitoring apparatus according to the present invention.

Referring to FIG. 2, the monitoring device 200 according to the present invention includes a database 210, an underspeed information detecting unit 230, a relationship information calculating unit 250, a second reference speed calculating unit 270, and an alarm output. It can be seen that the unit 290 is included, and for convenience of description, it will be described with reference to FIG. 1.

The database 210 stores processing speed information and failure occurrence information of data processed up to a first time point for each unit time. Here, the first time point refers to a time zone closest to the present of the past time points when the streaming data processing system 130 processes the streaming data. For example, if the streaming data processing system 130 processes the streaming data from 2 pm to 3 pm, the first time point may be 3 pm.

The processing speed information refers to information on a value of the speed at which the streaming data processing system 130 processes data. The processing speed information may be generated based on information recorded for each unit time in a log of the streaming data processing system 130. The data processing speed of the streaming data processing system 130 depends on the amount of resources held by the system when processing data, a failure state, the capacity of the streaming data, the type of streaming data (meaning the data format, extension, etc.), and the like. For example, the processing speed information is information in which various information including the various data processing speeds are recorded by unit time. In general, slowing down the data throughput of a system is interpreted as a performance degradation of the system.

The failure occurrence information refers to information on various failures that occur while the streaming data processing system 130 processes the data. Here, the failure occurrence information includes not only a state in which a preset failure code is generated in the streaming data processing system 130, but also all information on a state in which a failure code is not preset.

Even when the streaming data processing system 130 outputs a failure code, the data processing speed of the streaming data processing system 130 may not be changed at all. In other words, failure and performance degradation in the streaming data processing system 130 are separate problems. For example, if the data processing result of the streaming data processing system 130 violates the SLO (Service Level Objective) preset in the streaming data processing system 130, the streaming data processing system 130 may output a failure code. have. In this case, the SLO means a predetermined criterion required for the processing result of the streaming data processing system 130, and the criterion may be a criterion irrelevant to the data processing speed of the streaming data processing system 130.

As an optional embodiment, the failure occurrence information may include reference failure information when a predetermined reference failure code is generated and non-reference failure information when a point other than the time when the reference failure code is generated. A time point other than the time point of occurrence of the reference failure code includes both a time point of occurrence of a failure code other than the reference failure code and a time of no failure code generation.

That is, the failure occurrence information is information output from the streaming data processing system 130 at a unit time, but a failure corresponding to a failure code preset in the streaming data processing system 130 is not generated every time. Reference failure information and non-reference failure information are included. In this optional embodiment, the reference failure information may be expressed as 1 and the non-reference failure information may be expressed as 0.

The database 210 may store in advance a comparison code for distinguishing whether a failure code output from the system is a reference failure code. The database 210 analyzes the data processing result output from the streaming data processing system 130 to determine whether there is a reference failure code that matches the comparison code, and controls to store the reference failure information or non-reference failure information. It may also include a processor. For example, if fault code F6009 is output from the streaming data processing system 130 at time t1 and F6009 is included in the comparison code, the failure occurrence information at time t1 stored in the database 210 includes reference failure information. Done.

Equation 1 shows an example of failure occurrence information from 0 second to t seconds. As shown in Equation 1, fault occurrence information from 0 second to t seconds may be expressed as a vector of t + 1 dimension.

Both the processing speed information and the failure occurrence information are recorded in unit time. Here, since the unit time is not limited to a specific time, it may be a unit of 10 seconds, 1 minute, 10 minutes, one hour, or the like. Since both the processing speed information and the failure occurrence information are recorded for each unit time, the database 210 may map and store the processing speed information and the failure occurrence information. For example, assuming that the unit time is 1 minute and streaming data continues to be processed from 12:00 pm on January 1, the database 210 displays processing speed information and failure occurrence information at 12:01 pm on January 1st. Each may be stored correspondingly. When the database 210 is searched at 12:01 pm on January 1, the database 210 may output processing speed information and failure occurrence information accordingly.

Subsequently, the speed underspeed information detecting unit 230 grasps the speed underspeed information at the time of processing the data slower than the first reference speed from the processing speed information stored in the database 210.

The first reference speed is a speed value previously stored in the underspeed information detecting unit 230 or stored in the database 210 and then transmitted to the underspeed information detecting unit 230. The first reference speed is compared with the data processing speed for each unit time included in the processing speed information. If the data processing speed of the processing speed information at a specific point in time is slower than the first reference speed, the speed undershooting information detecting unit 230 recognizes the processing speed information as the speed falling information. The processing speed information includes all the information on the processing time, processing data, and processing speed of the streaming data processing system 130. If the data processing speed of the processing speed information is slower than the first reference speed, the processing is performed. All information included in the speed information becomes information included in the speed underspeed information. That is, the first reference speed is a criterion for the underspeed information detecting unit 230 to determine how much the speed of processing the data by the streaming data processing system 130 occurs due to a decrease in performance in the streaming data processing system 130. to be.

Optionally, the first reference speed may be set differently for each unit time. For example, the first reference speed from 0 to t may be expressed as Equation 2.

According to the present exemplary embodiment, since the first reference speed is variable as shown in Equation 2 according to the timing at which the streaming data is processed by the streaming data processing system 130, the processing speed of the streaming data different for each unit time is different. With respect to the monitoring device 200 can minimize the determination that the performance is uniformly reduced.

The processing speed of the data of the streaming data processing system 130 may vary with time, and even if the processing speed becomes slow at a specific time point, it may be only a temporary delay caused by factors other than performance degradation. According to the monitoring device 200 according to the present invention, the system administrator can determine whether or not the performance of the system in consideration of such factors.

Hereinafter, in order to distinguish the speed underspeed information, information which is not identified as the speed underspeed information among the processing speed information will be referred to as speed achievement information. The speed shortening information detecting unit 230 can distinguish the speed speed information from the speed short information and the speed short information by using the above-described process, and based on the speed short information and the speed achievement information, Information on underspeed can be calculated for each unit time.

Equation 3 shows an example of information on whether the speed falls from 0 to t seconds. For example, in Equation 3, y (1), which is speed inferiority information at one second when data is processed slower than the first reference speed, is 1, at two second when data is processed faster than the first reference speed. Y (2), which is information on underspeed of, becomes 0. In this case, 0 and 1 may be reversely applied according to a preset value in the speed undertaking detecting unit 230.

Hereinafter, FIG. 2 will be described subsequently.

The relationship information calculation unit 250 calculates correlation information between the processing speed information and the failure occurrence information.

Correlation information refers to information that expresses a correlation between processing speed information and failure occurrence information. Since the streaming data processing system 130 processes the data until the first point of time, there are many results, and the failure occurrence information has binary variable characteristics (whether or not a preset failure code has occurred). It is possible to approximate the correlation with fault occurrence information with one formula.

	Fault code output	Fault code not output
Below 1st standard speed	x	y
Achieved first reference speed	u	v

Table 1 is a table showing the ratio of the data processing result output from the streaming data processing system 130. Since the data processing result output from the streaming data processing system 130 may be any one of the four types in Table 1, the sum of x, y, u, and v results in 1.

Information according to Equations 4 to 7 may be defined using x, y, u, and v of Table 1.

First, in the equation 4, PPV (Positive Predictive Value) is a time point when processing the data slower than the first reference speed among the data processing results output from the streaming data processing system 130 from 0 to t, and outputs the fault code Means the proportion of information. In Equation 5, cPPV (complementary PPV) is a data processing result that is slower than the first reference speed among the data processing results output from the streaming data processing system 130 from 0 to t, but the error code does not output information Means the ratio.

In Equation 6, NPV (Negative Predictive Value) is a data processing result output from the streaming data processing system 130 from 0 to t, the data processing faster than the first reference speed, and does not output the fault code The ratio of information. In Equation 7, cNPV (complementary NPV) is a data processing result output from the streaming data processing system 130 from 0 to t, processing the data faster than the first reference speed, and the information of the time when the error code is output Means percentage.

After sufficient time has elapsed, when the system manager calculates the PPV and the NPV from the data processing result outputted by the streaming data processing system 130, the exemplary PPV is such that the calculated PPV and the NPV become the specific values desired by the system administrator. And NPV are set in advance. In the following, exemplary PPV and NPV set by the system administrator will be referred to as alpha (alpha) and beta (beta), respectively.

When the system administrator sets alpha and beta in the streaming data processing system 130, the PPV and NPV calculated from the data processing result of the streaming data processing system 130 converge to alpha and beta, respectively, as time passes. That is, when the PPV and NPV of the data processing result output from the streaming data processing system 130 become alpha and beta, respectively, the system administrator operates the streaming data processing system 130 for a sufficiently long time and lasts for a long time in a stable state. It can be seen that the system has a monotonicity that becomes. The present invention proposes a method for calculating a second reference speed that can be compared with the data processing speed of the streaming data processing system 130 at a time after the forging is established as described above, and further description of the second reference speed will be described later. Let's do it.

Subsequently, in order to calculate correlation information between the processing speed information and the failure occurrence information, the relationship information calculation unit 250 may normalize the processing speed information to a relational expression based on the failure occurrence information and the first reference speed.

Equation 8 shows a linear relation based on the processing speed information on the occurrence information of the failure and the first reference speed. In Equation 8, Y (t) is a vector for processing speed information, Γ (t) is a vector for first reference speed, A (t) is a vector for failure occurrence information, and x (t) is a processing. Refers to a composite vector of velocity information and fault occurrence information.

At this time, the processing speed information, more specifically, means under speed information. The underspeed information is information on whether data processing speeds of different streaming data processing systems 130 are faster or slower than the first reference speed for each unit time, and can be represented by binary variables. Here, the speed under whether information is included in the processing speed information, or according to the embodiment, after the relationship information calculation unit 250 receives the processing speed information, based on the processing speed information and the first reference speed, the speed under Whether or not information can also be calculated.

As described above, the failure occurrence information may also be represented as a binary variable. Although the first reference speed may have a fixed constant value, it has been described through Equation 2 that the first reference speed may have a different value every time according to an embodiment.

When regression modeling is applied to a linear equation such as Equation 8, the unknown values of b1, b2, and c can be found. However, in the present invention, since Y (t) is a binary variable, Equation 8 and The same general linear equation is not available.

The first reason why Equation 8 cannot be used is that when X (t) has a sufficiently large value in Equation 8, Y (t) may exceed one. The second reason why Equation 8 cannot be used is that Y (t) does not satisfy the precondition for using linear regression because Y (t) is only 0 or 1. A prerequisite for using a linear regression equation is that the residuals, which are the tests of significance, must follow a normal distribution.

Also, finally, A (t) is also a binary variable of 0 or 1, and Γ (t) is a variable that can be various values, so in the present invention, a linear regression equation such as Equation 8 cannot be used, and logistic regression ( Logistic Regression should be used.

Equation 9 shows a vector p (x) for using a regression equation of logit transformation. In Equation 9, p (x) is defined as the probability that the data processing speed of the streaming data processing system 130 is slower than the first reference speed when the vector x is defined as the first reference speed and the failure occurrence information.

Equation 10 is a regression equation calculated by applying a logit transform to Equation 9. Since each term is defined as a vector of the same dimension, the regression coefficients b1, b2, and c can be obtained using the Maximum Likelihood Estimation. The maximum likelihood estimation method is an effective estimation method when a nonlinear statistical model is analyzed based on binary data when the number of samples is sufficiently secured, and since the maximum likelihood estimation method is widely known, the maximum likelihood estimation method is used. The calculation process will be omitted.

Time	A (t)	Γ (t)	Y (t)
One	0	58	0
2	One	109	One
3	One	126	0
4	0	60	One
5	One	120	One
6	0	69	0
7	One	147	One
8	0	85	One
9	0	58	0
10	One	116	0
11	One	131	0
12	One	104	One
13	One	149	0
14	0	60	One
15	0	64	One

Table 2 shows Y (t), A (t) and Γ (t) from 1 second to 15 seconds. By arranging the values shown in Table 2 as vectors of the same dimension, substituting the arranged vectors into Equation 10, and applying the maximum likelihood estimation method, b1 is 0.8445968, b2 is -0.01878321, and c is 1.507379. Since Table 2 is an exemplary value, when Y (t), A (t), and Γ (t) for each unit time are different from those in Table 2, b1, b2, and c may also vary.

The second reference speed calculator 270 receives the ratio information between the speed underspeed information and the failure occurrence information, and based on the correlation information calculated by the ratio information and the relationship information calculator 250 after the first time point. The second reference speed at the second time point is calculated.

Here, the ratio information between the speed underspeed information and the failure occurrence information refers to the ratio information of a value representing the speed underspeed information and a value representing the failure occurrence information. For example, through Table 1, α may be ratio information between speed underspeed information and failure occurrence information.

α is a data processing speed of the streaming data processing system 130 is slower than the first reference speed among the data processing results outputted by the streaming data processing system 130 when time t has elapsed, and outputs a preset fault code. This is because it is defined as information on the ratio of the time. By the same logic as above, β, PPV, cPPV, NPV, cNPV at a specific time point may also be the ratio information between the underspeed information and the failure occurrence information. The second reference speed calculator 270 may receive a ratio information between the speed underspeed information and the failure occurrence information from the system manager or use a value previously stored in the database 210.

In addition, since the correlation information received from the relationship information calculation unit 250 means information indicating the relationship between the processing speed information and the failure occurrence information, the correlation information b1, b2, and c are not only correlation information. Can be

Hereinafter, an example in which the second reference speed calculation unit 270 calculates the second reference speed based on the regression coefficients b1, b2, c, which are examples of correlation information, and α and β, which are examples of ratio information, will be described. Do it.

First, Equation 10 is expressed as Equation 11 with respect to the vector p (x).

At this time, referring to Equation 8, by replacing the vector x by the combination of γ (t) and a (t), the same result as in Equation 12 can be obtained.

In Equation 12, a (t + 1) is 0 or 1 as a binary variable. That is, a (t + 1) is 1 when a preset fault code is output, and a (t + 1) is 0 when a preset fault code is output.

Equation 13 is a probability at time t + 1 at which a preset failure code is output, and Equation 14 is a probability at time t + 1 at which a preset failure code is not output. As described above, since the time t + 1 has already been assumed that the streaming data processing system 130 has started to show monotonism after a sufficiently long time, the result of Equation 13 is expressed as 1-β, Equation 14 The result is the same as α.

Equation 15 shows the result of combining Equations 13 and 14 together. That is, the second reference speed compared with the data processing speed of the streaming data processing system 130 at the time t + 1 may be calculated if all values of α, β, and regression coefficients b1, b2, and c are present. Computing the second reference speed by deriving the equation 15 from Equations 11 to 14 is an example of a method of calculating the second reference speed, and thus, calculates the second reference speed based on the ratio information and the correlation information. If it is calculated, it may be included in the scope of the present invention even if the same equation as in Equations 11 to 15 is not used.

3 is a diagram schematically showing a second reference speed calculated according to the present invention.

More specifically, FIG. 3 represents a first reference speed, speed failure information, and fault occurrence information according to Table 2 as vectors, respectively, and is calculated by substituting Equation 15 together with ratio information (α and β). The second reference speed at the time point is shown in graph form.

In this case, it is assumed that the values of b1, b2 and c in Equation 15 are used in b1, b2 and c according to Table 2, and α is 0.95 and β is 0.9. When the maximum likelihood estimation method is applied to the data according to Table 2, b1 is 0.8445968, b2 is -0.01878321, and c is 1.507379.

That is, it is assumed that the streaming data processing system 130 has a monotony at the time t is 15, and the second reference speed is 165.6873, about 166, at the time t is 16. The streaming data processing system 130 may be interpreted by the system manager as having no performance degradation when the data processing speed is faster than 166 at the time t is 16.

Referring to FIG. 3, the first reference speed continues to change with time, and the second reference speed at the time t is 16 is the cumulative first reference speed at the time t from 0 to 15. It can be understood that the value is calculated by the characteristic. If the reference speed at time t is 17 is assumed to be the third reference speed, the third reference speed is the first reference speed at time t from 0 to 15 and the second reference speed at time t is 16. It is calculated in consideration of all the characteristics, and by repeating the above method, the reference speed of the time after the point t + 2 can also be calculated.

4 shows an example of implementing the second reference speed calculation method according to the present invention using pseudo-code.

Referring to FIG. 4, α and β are set to 0.95 and 0.9, respectively, and it can be seen that the first reference speed at the time t is 0 is also assigned a preset value. In FIG. 4, μ (t) denotes a data processing speed of the streaming data processing system 130, and other remaining variables are the same as those used in the above description.

According to the water code of FIG. 4, the second reference speed calculation unit 270 repeats the calculation until the PPV and the NPV at the present time have sufficiently monotonous time, and is the next time point after the time t. It can be seen that the second reference speed for the time t + 1 is calculated. In this case, 'isViolated' is assumed to be a function of receiving an SLO value as an input and returning a value of 0 or 1, and has already been described while explaining the database 210 that the SLO value is an example of failure information. . In the pseudo code of FIG. 4, the final equation means (15). The second reference speed at time t + 1 calculated through Equation 15 becomes the first reference speed when calculating the second reference speed at time t + 2.

The alarm output unit 290 outputs a performance abnormality alarm when the data processing speed at the second time point of the streaming data processing system 130 is slower than the second reference speed. Here, the second time point refers to a time point (time) after the first time point, and refers to a time point after the streaming data processing system 130 shows monotony after sufficient time has elapsed. That is, the second time may be immediately after the unit time of one unit has elapsed after the first time. For example, if data of the data processed by the streaming data processing system 130 for 60 seconds is stored in the database 210 and the unit time is 1 second, the 61 second time point may be the second time point.

Performance abnormality alarm is information that the monitoring device 200 according to the present invention directly to the system administrator that the performance degradation occurred in the streaming data processing system 130, the system administrator checks the performance abnormality alarm, If necessary, the streaming data processing system 130 may be checked.

5 is a flowchart illustrating an example of a method for monitoring a streaming data high speed processing system according to the present invention.

Since the method according to FIG. 5 may be implemented by the monitoring apparatus 200 for monitoring the streaming data high speed processing system according to FIG. 2, it will be described with reference to FIG. 2, and a redundant description will be omitted. Let's do it.

First, the speed shortening information detecting unit 230 refers to the database 210 to grasp the speed shortening information at the time when data is processed slower than the first reference speed in the processing speed information (S510). Here, the database 210 stores processing speed information and failure occurrence information of the data processed at the first time point for each unit time.

As an optional embodiment of step S510, the first reference speed may be set differently for each unit time. In addition, the failure occurrence information stored in the database 210 in step S510 may include reference failure information at the time when the preset reference failure code occurs and non-reference failure information at the time when only the failure code other than the reference failure code occurs.

Subsequently, the relationship information calculation unit 250 calculates correlation information between the processing speed information and the failure occurrence information (S530).

As an optional embodiment of step S530, the relationship information calculation unit 250 may calculate correlation information by using a maximum likelihood method.

In another optional embodiment of step S530, the relationship information calculator 250 may calculate correlation information by using logit transformation.

The second reference speed calculator 270 receives the ratio information between the speed underspeed information and the failure occurrence information and calculates a second reference speed at a second time after the first time based on the correlation information and the rate information. (S550).

According to an exemplary embodiment of step S550, the ratio information received by the second reference speed calculator 270 may be information about a ratio between the speed fail information and the reference failure information.

The alarm output unit 290 compares the data processing speed of the streaming data processing system 130 with the second reference speed at the second time point (S570). If the data processing speed of the streaming data processing system 130 at the second time is slower than the second reference speed, the alarm output unit 290 outputs a performance abnormality alarm (S590). In operation S590, the abnormality occurrence alarm output by the alarm output unit 290 is information that directly informs the system administrator that the performance decrease has occurred in the streaming data processing system 130, and the system administrator reports the abnormality occurrence alarm. The reporting streaming data processing system 130 may be checked.

According to the present invention, a second reference speed, which is a threshold value compared with a processing result of a streaming data processing system that processes big streaming data, is actively calculated and applied without the need for an administrator to intervene with a change in time. As a result, the performance change state of the streaming data processing system can be monitored with minimal intervention by the system administrator, thereby reducing the burden on the system administrator.

In addition, when the monitoring device according to the present invention is applied to the streaming data processing system, it is possible to achieve the same or more system monitoring effect as before even with less manpower and time.

Embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, such a computer program may be recorded in a computer-readable medium. At this time, the media may be magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROMs. Hardware devices specifically configured to store and execute program instructions, such as memory, RAM, flash memory, and the like.

On the other hand, the computer program may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of computer programs may include not only machine code generated by a compiler, but also high-level language code executable by a computer using an interpreter or the like.

Particular implementations described in the present invention are embodiments and do not limit the scope of the present invention in any way. For brevity of description, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings by way of example shows a functional connection and / or physical or circuit connections, in the actual device replaceable or additional various functional connections, physical It may be represented as a connection, or circuit connections. In addition, unless specifically mentioned, such as "essential", "important" may not be a necessary component for the application of the present invention.

In the specification (particularly in the claims) of the present invention, the use of the term “above” and similar indicating terminology may correspond to both the singular and the plural. In addition, in the present invention, when the range is described, it includes the invention to which the individual values belonging to the range are applied (if not stated to the contrary), and each individual value constituting the range is described in the detailed description of the invention. Same as Finally, if there is no explicit order or contrary to the steps constituting the method according to the invention, the steps may be performed in a suitable order. The present invention is not necessarily limited to the description order of the above steps. The use of all examples or exemplary terms (eg, etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is limited by the examples or exemplary terms unless defined by the claims. It doesn't happen. In addition, one of ordinary skill in the art appreciates that various modifications, combinations and changes can be made depending on design conditions and factors within the scope of the appended claims or equivalents thereof.

The above-described embodiments of the present invention can be applied to implementing an apparatus for monitoring a system for processing streaming data at high speed and detecting an abnormality.

Claims (15)

1. In the monitoring device for monitoring a streaming data high-speed processing system,

   A database storing processing speed information and failure occurrence information of data processed by the streaming data high speed processing system up to a first time point by unit time;

   A speed shortening information detecting unit for grasping speed shortening information at a time when data is processed slower than a first reference speed in the processing speed information;

   A relationship information calculation unit for calculating correlation information between the processing speed information and the failure occurrence information;

   Receiving ratio information between the speed shortage information and the failure occurrence information and calculating a second reference speed at a second time after a first time based on the calculated correlation information and the received rate information; 2 standard speed calculation unit; And

   And an alarm output unit for outputting a performance abnormality alarm when the data processing speed at the second time point is slower than the second reference speed.
2. The method of claim 1,

   The first reference speed is,

   Monitoring device for monitoring the streaming data high-speed processing system, characterized in that differently set by the unit time.
3. The method of claim 1,

   The failure occurrence information,

   And a reference failure information at a time point when a predetermined reference failure code occurs and non-reference failure information at a time except when the reference failure code occurs.
4. The method of claim 3,

   The received ratio information is,

   And monitoring the streaming data high speed processing system, wherein the information is on the ratio between the speed shortage information and the reference failure information.
5. The method of claim 1,

   And the relationship information calculation unit calculates the correlation information by using a maximum likelihood method.
6. The method of claim 1,

   The relationship information calculation unit,

   The monitoring device for monitoring the streaming data high-speed processing system, characterized in that using the Logit Transformation in calculating the correlation information.
7. The method of claim 1,

   The underspeed information detecting unit,

   Monitoring device for monitoring the streaming data high-speed processing system, characterized in that for using the binary variable to distinguish the speed achievement information at the time the data is processed faster than the first reference speed in the processing speed information with the identified underspeed information .
8. In the monitoring method for monitoring a streaming data high-speed processing system,

   Determining the speed below information at the time when the data is processed slower than the first reference speed in the processing speed information by referring to a database storing processing speed information and failure occurrence information of data processed up to the first time point by unit time Grasping underspeed information information;

   Calculating relationship information between the processing speed information and the failure occurrence information;

   Receiving ratio information between the speed shortage information and the failure occurrence information and calculating a second reference speed at a second time after a first time based on the calculated correlation information and the received rate information; 2 standard speed calculation step; And

   And an alarm output step of outputting an alarm for occurrence of a performance abnormality if the data processing speed at the second time point is slower than the second reference speed.
9. The method of claim 8,

   The first reference speed is,

   Monitoring method for monitoring a streaming data high-speed processing system, characterized in that differently set for each unit time.
10. The method of claim 8,

    The failure occurrence information,

    A monitoring method for monitoring a streaming data high-speed processing system, characterized in that it comprises the reference failure information of the time point when the predetermined reference failure code occurs and the non-reference failure information of the time except for the occurrence of the reference failure code.
11. The method of claim 10,

    The received ratio information is,

    And monitoring the streaming data high speed processing system, wherein the rate information is information on a ratio between the speed short information and the reference failure information.
12. The method of claim 8,

    The relationship information calculation step is a monitoring method for monitoring a streaming data high-speed processing system, characterized in that for calculating the correlation information using the Maximum Likelihood Method (Maximum Likelihood Method).
13. The method of claim 8,

    The relation information calculation step,

    The method for monitoring the streaming data high-speed processing system, characterized in that using the Logit Transformation in calculating the correlation information.
14. The method of claim 8,

    The underspeed information identifying step,

    Monitoring method for monitoring the streaming data high-speed processing system, characterized in that using the binary variable to distinguish the speed achievement information at the time the data is processed faster than the first reference speed in the processing speed information with the identified speed under the information. .
15. A computer-readable recording medium having recorded thereon a program for executing the method according to claim 8.

Images