KR20100088399A - Apparatus and method for software faults prediction using metrics - Google Patents

Abstract

PURPOSE: An apparatus and a method for software faults prediction using metrics are provided to change a measured metric value in other system, thereby applying fault prediction model to the other system. CONSTITUTION: Based on a normalized metric value of a fault prediction model and a normalization module(20), a fault prediction module predicts fault per class of test target. Based on fault information of a bug tracking system, a fault information collecting module(40) collects information about actual fault of a test objects. Based on information of the class having a predicted result value of the fault prediction model and a collected actual fault of the fault information collection module, a production force evaluation module(50) evaluates the forecastability of the fault prediction model.

Description

Apparatus and Method for Software Faults Prediction using Metrics

The present invention relates to a calculation apparatus and a calculation method for improving the predictive power in order to predict the defect of software or to universally apply the existing defect prediction model.

In the face of intense competition, companies need to develop software products that are superior to their competitors in a timely manner so that they do not get out of the market. Due to the influence of such a development environment, things are often delivered to consumers with defects in the developed software. In the case of cell phones, users often experience cases where the phone suddenly turns off. When this happens, customers generally have questions about the reliability of the product and, in serious cases, lead to distrust in the company.

Therefore, it is important to detect software defects during the product development stage and before release. Popular methods for finding defects include static analysis, code inspection, and testing. Static analysis is a technique that analyzes the source code automatically and informs you where the defect will occur. Code inspection is the detection of defects by analyzing the source code directly by a person other than the developer. Testing is specific to development software. It is a method to check whether outputting expected result is output after passing input value. However, since these methods can be executed later in the development stage, that is, the source code is in development, even if a defect is found, it is expensive to fix it.

To solve this problem, studies have been conducted to predict software defects early in the development phase over the last decade. In these studies, metrics were measured for the outputs of the design phase to find relationships with defects. At this time, due to the increase of software development using the object-oriented paradigm, most of the metrics used in the research were object-oriented metrics, and the size and complexity of the software were also used. The researchers also used logistic regression mainly to build predictive models. Logistic regression can be used to classify classes that are prone to defects and those that do not.

In particular, Gyimothy, Zhou and Olague have used logistic regression models to predict defects in large systems over the last three years. First, Gyimothy developed a defect prediction model for Mozilla, a large open source software. As a technique for developing a defect prediction model, an attempt was made to increase the prediction rate by applying the machine learning technique together with logistic regression. As a result, the predictive model developed by logistic regression analysis was higher in accuracy than the model developed by machine learning. Zhou also developed predictive models for defect severity for software developed by NASA. As a result, the predictive model produced by logistic regression had higher prediction rate for defects of lower severity than those of higher severity. In addition, Olague created defect prediction models using CK and MOOD metric collections for systems developed using iterative or agile development processes. It was higher than the prediction rate of the defect prediction model.

In order for these findings to be widely used in the development site, the defect prediction model must be universal. However, the existing studies have established a defect prediction model based on defect and metric data collected from specific software, and then applied this model to the same software to verify the generality of the model. Therefore, in order to be widely used in the development site, an experiment is needed to determine whether the connection defect prediction model has universality.

An object of the present invention is to apply a conventionally defined defect prediction model to other systems in general. There are two major benefits to versatility. First of all, the cost of developing software is reduced. Because new development of the defect prediction model to the development system requires a separate management system for the management of defect information. In addition, the statistical work procedures and tools required for the development of the defect prediction model are required, so using the defect prediction model of other systems does not incur unnecessary development system cost. Next, when developing a new system with different software characteristics, existing defect information does not exist. In this case, there is an advantage in efficiently determining whether there is a defect using the existing defect prediction model.

Accordingly, the present invention has been made to solve the above problems, the software defect prediction calculation device and calculation using the metric that the existing defect prediction model can have suitable prediction power even in a large system through the conversion process of the measured metric value The purpose is to provide a method.

Another object of the present invention is to provide an apparatus and method for calculating a software defect prediction using a metric capable of efficiently managing development costs by intensively managing software development resources in a module that is expected to cause a defect by increasing the predictive power of the defect prediction model. .

A feature of the software defect prediction calculation apparatus using the metric according to the present invention for achieving the above object is a metric measurement value for parsing the source file as an experiment and removing unnecessary information and then measuring the metric measurement value to detect the refined metric measurement value. A normalization module for performing a normalization process on the metric measurement values detected by the metric measurement input module, or performing a standard normalization process to calculate a normalized metric value, an implicit defect prediction model, and the normalization module Defect prediction module that derives the result of estimating the presence or absence of defects of each subject based on the normalized metric value in, and based on the defect information present in the bug tracking system. The actual name of the subject using the registered class name A defect prediction model that collects information about the presence and absence of a defect, and a defect prediction model based on the result value predicted by the defect prediction model in the defect prediction module and the information of the actual defective class collected in the defect information collection module. It includes a predictive power evaluation module for evaluating the predictive power of.

Preferably, the predictive power evaluation criteria of the defect prediction model of the predictive power evaluation module include precision representing a ratio of a class correctly classified by the prediction model among all classes, and actually among all classes classified as defects by the prediction model. It is characterized by using the accuracy (Correctness) indicating the proportion of the class which is a defect, and the completeness (Completeness) indicating the ratio of defects existing in the class classified as defects in the prediction model among the total defects actually present.

In order to achieve the above object, the software defect prediction calculation method using the metric according to the present invention is characterized by (a) collecting a source file that is an experiment target and measuring the metric to detect a refined metric measurement value, (b A metric normalized by performing a normalization process of dividing the detected metric measurement into a predetermined size and unifying it to make a normal distribution, or performing a standard normalization process of making the normal distributions in a standard form. Calculating a value, (c) deriving a result of predicting the presence or absence of a defect for each class of a test subject based on the normalized metric value and the threshold value of an implicit defect prediction model; ) Independent of the steps (a) to (c), collecting information about the actual defects of the test subject, and (e) in step (c) Based on the actual fault class information collected at the predicted class information and, (d) the step may comprises a step of evaluating the predictive power of the model fault prediction.

에서 클래스의 결함 발생 확률(

)을 계산하는 단계와, 상기 계산된 결함 발생 확률과 내제된 결함 예측 모형의 임계값을 비교하는 단계와, 상기 비교결과, 임계값이 결함 확률(

)과 같거나 큰 경우 결함이 있는 것으로 판단하고, 임계값보다 결함 확률(

) 이 작으면 결함이 없는 것으로 판단하는 단계를 포함하며, 이때,

는 독립변수인 객체지향 메트릭의 측정값,

은

의 회귀계수이며,

값의 범의는 0부터 1까지인 것을 특징으로 한다.Preferably step (c) is a formula

Probability of defects in a class in

), Comparing the calculated defect occurrence probability with a threshold value of an implicit defect prediction model, and as a result of the comparison, the threshold is a defect probability (

Is greater than or equal to), it is determined to be defective, and

) Is determined to be free from defects, wherein

Is a measure of an object-oriented metric that is an independent variable,

silver

Is the regression coefficient of,

The range of values is characterized by being from 0 to 1.

Preferably, the logistic value used in the above equation may be used by applying the logistic value expressed in the logistic regression model according to the defect prediction model used.

Preferably, the step (d) includes collecting defect information from bugzilla provided by the Eclipse website and files provided by NASA and PROMISE, and using the collected defect information to detect defects in the source content. Extracting the name of the class registered as being present, increasing the number of defects by one each time the name of the class is extracted, and extracting the name of a class or invalid class that does not exist in the metric measurement If it is characterized in that it comprises the step of removing the name of the class from the actual defect information.

Preferably, the criterion for evaluating predictive power in step (e) uses precision, correctness, and completeness, and uses a tradeoff between accuracy and completeness as a threshold. To determine the accuracy and completeness of the model.

Software defect prediction calculation apparatus and calculation method using the metric according to the present invention as described above has the following effects.

First, the process of converting the metric values measured in other systems is increased, so that the defect prediction model can be applied to other systems in general.

Second, through the conversion process and evaluation tool of the measured metric value, it is possible to increase the predictive power of the existing defect prediction model.

Third, as the predictive power is increased, software development resources can be centrally managed in modules that are expected to cause defects more accurately, thereby enabling effective management of development costs.

Other objects, features and advantages of the present invention will become apparent from the following detailed description of embodiments with reference to the accompanying drawings.

Referring to the accompanying drawings, a preferred embodiment of a software defect prediction calculation apparatus and a calculation method using the metric according to the present invention will be described. However, the present invention is not limited to the embodiments disclosed below, but can be embodied in various forms, and only the present embodiments are intended to complete the disclosure of the present invention and to those skilled in the art to fully understand the scope of the invention. It is provided to inform you. Therefore, the embodiments described in the specification and the drawings shown in the drawings are only the most preferred embodiment of the present invention and do not represent all of the technical idea of the present invention, various modifications that can be replaced at the time of the present application It should be understood that there may be equivalents and variations.

1 is a block diagram illustrating a structure of a software defect prediction calculation apparatus using metrics according to an embodiment of the present invention.

As shown in FIG. 1, the software defect prediction calculator includes a metric measurement value input module 10, a normalization module 20, a defect prediction module 30, a defect information collection module 40, and a predictive power evaluation module 50. ).

The metric measurement input module 10 parses the source file, which is a subject of experiment, removes unnecessary information, measures the measured CK metric, detects the purified metric measurement, and then transfers the refined metric measurement to the normalization module 20, which is the next module. In this case, the metric measurement uses Together 2007, and the measured value is output in the form of an XML file.

The normalization module 20 divides the CK metric measurement value detected by the metric measurement input module 10 into a predetermined size and performs a normalization process to create a normal distribution, or the normal distributions are in one standard form. A normalized CK metric value is produced by performing a standard normalization process that produces a standard normal distribution of.

The defect prediction module 30 derives a result value of predicting the presence or absence of a defect for each class of the test subject based on the implicit defect prediction model and the CK metric value normalized by the normalization module 20. The defect prediction models are Olague, Yuming and Gyimothy models, and each model has a threshold value for determining a class defect.

The defect information collecting module 40 collects information on the actual defect presence or absence of a test subject by using a name of a class registered as a defect among source contents based on defect information existing in a bug tracking system. do. For reference, the defect information is information collected from bugzilla, a bug tracking system provided by the Eclipse website, and files provided by NASA and PROMISE.

The predictive power evaluation module 50 is based on the result predicted by the defect prediction model in the defect prediction module 20 and the defect prediction model based on the information of the actual defective class collected by the defect information collection module 30 Evaluate the predictive power of At this time, as the criterion for evaluating the predictive power of the defect prediction model, the precision representing the ratio of the class correctly classified by the prediction model among the entire classes, and the class that is actually the defect among all the classes classified as the defect by the prediction model are occupied. We use the correctness of the ratio, and the completeness of the proportion of defects in the class classified as defects in the predictive model.

The calculation method of the software defect prediction calculation apparatus using the metric according to the present invention configured as described above will be described in detail with reference to the accompanying drawings. The same reference numerals as in FIG. 1 refer to the same members performing the same function.

2 is a flowchart illustrating a software defect prediction calculation method using metrics according to an embodiment of the present invention.

Referring to FIG. 2, first, the metric measurement input module 10 collects basic data from an Eclipse 3.3 source file, which is an experiment target, measures the metric, and detects purified metric measurement (S10). The metric measurement was set to Together 2007. However, other existing model studies use self-developed metric measuring tools or commercial tools. As described above, it should be noted that the measuring tool used for measuring the metric can be various embodiments within the scope of the technical idea of the present invention. Eclipse, the open source software selected for the experiment, is a Java-based open source project supported by IBM, Nokia, and others. It is an IDE (Integrated Development Evironment) tool for JAVA and C ++ development.

Subsequently, the normalization module 20 performs a normalization process of generating a normal distribution by dividing the detected metric measurement into a predetermined size and unifying it, or a standard normalization process of creating the standard normal distribution of one standard form. In operation S20, the normalized metric value is calculated.

The defect prediction module 30 derives a result value of predicting the presence or absence of a defect for each class of the test subject based on the calculated normalized metric value and the threshold value of the implicit defect prediction model (S30). Prediction models used are Olague, Yuming and Gyimothy models, and there are thresholds for determining the class defects for each defect prediction model.

)을 계산한다. 즉, 결함 예측 모형은 개별 클래스에서 임계값이 결함 확률(

)과 같거나 큰 경우 결함이 있는 것으로 예측하고, 임계값보다 결함 확률(

) 이 작으면 결함이 없는 것으로 예측한다.At this time, the presence or absence of a defect in the test target class is the probability of occurrence of a defect of the class

). In other words, the defect prediction model shows that the threshold for each class

Is equal to or greater than), and it is assumed to be defective,

Smaller) predicts that there are no defects.

는 독립변수인 객체지향 메트릭의 측정값이며, 종속변수인

는 클래스에서 결함이 발견될 확률을 의미하며, 메트릭 측정값을 사용하여 계산된 확률이 임계값 이상이면 해당 클래스에는 결함이 있다고 간주한다.

값의 범의는 0부터 1까지이다.

은

의 회귀계수를 위미하며,

값이 크면 클수록 해당 독립변수가 클래스에서 결함을 발견할 확률에 영향력이 높다.At this time,

Is a measure of the object-oriented metric, which is an independent variable,

Is the probability of finding a defect in a class. If the probability calculated using a metric measure is above the threshold, the class is considered to have a defect.

The range of values is 0 through 1.

silver

The regression coefficient of,

The larger the value, the higher the probability that the independent variable will find a defect in the class.

)을 상기 수학식 1에 적용하여 클래스의 결함 발생 확률로 계산된다.In addition, the lodge values used in Equation 1 are each defined according to the predictive model used, and Equation 2 represents an Olague model represented by a logistic regression model, and uses the Olague model to determine a class defect. In this case, the logistic value used in the prediction model of Equation 2

) Is calculated as a probability of occurrence of a defect of a class by applying to Equation 1 above.

On the other hand, the defect information collection module 40 collects information on the presence or absence of actual defects of the test subject, independently of the steps "S10" to "S30". It collects defect information that exists in Bugzilla, a bug tracking system provided by the Eclipse website through a URL. The defect information provided by the URL collects data from an HTML source. Other defect information is obtained from files provided by NASA and PROMISE. Using the collected information, the name of the class registered as defective is extracted from the source contents, and the number of defects is increased by 1 for each occurrence of the name. And the name of the class or wrong class that does not exist in the metric measurement value removes the name of the class from the actual defect information (S40).

Then, the predictive power evaluation module 50 may determine the class of the defect prediction model based on the class information predicted by the defect prediction model in the defect prediction module 20 and the actual defective class information collected by the defect information collection module 30. The predictive power is evaluated, and the predictive power evaluation, the metric value of the class belonging to each area, and the presence or absence of a defect are output to a file (S50). Precise, correctness and completeness are used for evaluating predictive power.

The precision is a ratio occupied by a class correctly classified by a prediction model among all classes, and may be expressed by Equation 2 below.

Precision = (A1 + A4) / (A1 + A2 + A3 + A4)

The accuracy is a ratio of a class that is actually a defect among all classes classified as defects by the prediction model, and may be expressed as in Equation 3 below.

Accuracy = A4 / (A2 + A4)

In addition, the completeness is a ratio of defects existing in the class classified as defects in the prediction model among the total defects actually present, and can be expressed as Equation 4 below.

Integrity = S2 / (B1 + B2)

At this time, the areas A1 to A4 are the number of classified Java classes, and B1 and B2 are the actual number of defects of the source files classified in the corresponding area.

For example, if the precision of the defect prediction model is calculated as 60%, it indicates that 60 out of 100 classes are predicted as defective or missing classes. Next, when the accuracy is calculated at 60%, it indicates that 60 classes are predicted to be defective in 100 defective classes. Finally, when completeness is calculated at 60%, it indicates that 60 of 100 defects were predicted. For reference, when evaluating a defect prediction model using precision, accuracy, and completeness, there is a trade-off between accuracy and completeness. Therefore, as shown in FIG. Calculate accuracy and completeness.

It can be seen from the experimental results below that the calculation device and the calculation method can show the result that the existing defect prediction model can have suitable prediction power even in a large-scale system.

Figure 4a is a graph showing the distribution of defect occurrence probability calculated by the denormalized metric value, Figure 4b is a graph showing the distribution of defect occurrence probability calculated by the Gyimothy model after the standard normalized metric value.

Unlike the graph showing the distribution of defect occurrence probability calculated with non-normalized metric values as shown in FIG. 4A, the graph of FIG. 4B may be evenly distributed from side to side based on the average of defect occurrence probability. Therefore, it can be expected that the variability of the predictive power of the model changes smoothly according to the change of the threshold value.

FIG. 5 is a graph showing a result of predicting defects using a Gyimothy model using standard normalized metric values of the NASA KCI system.

As shown in FIG. 5, the predictive power result using the non-normalized metric value was not predicted as a defect in the experiment, but in the prediction using the normalized metric value, the accuracy of the defect prediction was 70.7% and the perfection was 70.5%. This result was similar to the main predictive power in the defect prediction model that Yuming built based on NASA KCI data.

Table 1 also shows the results of applying the defect prediction model to the test system after normalizing the metric values. In six of eight experiments, the results of the defect prediction model could be interpreted. The most notable model is the Gyimothy model. In experiments predicted using non-normalized metric values, all three systems failed to predict defects, but normalized metric values were used to predict defects in all three systems. In Yuming's model, we could explain the result of defect prediction in Eclipse. Therefore, when applying the defect prediction model to other systems, it is first necessary to examine the distribution of probability of occurrence of defects calculated by the defect prediction model, and if the distribution is not even, normalize the metric values and reapply them to the system. I think that can be improved.

Fault prediction model Test Target System Precision accuracy completeness Compromise Olague Claimed predictive power - 82% - Eclipse 3.3 76.8% 21.7% 21.7% 0.105 NASA KCI - - - 0.044 jEdit 4.0 - - - 0.039 Yuming Claimed predictive power 69.7% 61.4% 74.6% Eclipse 3.3 38.3% 34.3% 34.3% 0.488 NASA KCI Do not experiment jEdit 4.0 60.4% 64.2% 64.2% 0.421 Gyimothy Claimed predictive power 69.6% 72.6% 65.2% Eclipse 3.3 82.4% 38.3% 38.3% 0.716 NASA KCI 71.7% 70.7% 70.5% 0.579 jEdit 4.0 59.9% 63.7% 63.7% 0.332

Although the technical spirit of the present invention described above has been described in detail in a preferred embodiment, it should be noted that the above-described embodiment is for the purpose of description and not of limitation. In addition, those skilled in the art will understand that various embodiments are possible within the scope of the technical idea of the present invention. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

1 is a block diagram showing the structure of a software defect prediction calculation apparatus using metrics according to an embodiment of the present invention.

2 is a flowchart illustrating a software defect prediction calculation method using metrics according to an embodiment of the present invention.

3 is a flowchart illustrating a cutoff relationship between accuracy and completeness in evaluating predictive power of a defect prediction model in software defect prediction calculation using metrics according to an embodiment of the present invention.

4A is a graph showing the distribution of defect occurrence probability calculated from nonnormalized metric values.

4B is a graph showing the distribution of defect occurrence probability calculated by the Gyimothy model after normalizing metric values.

5 is a graph showing the results of predicting defects using a Gyimothy model using standard normalized metric values of the NASA KCI system.

DESCRIPTION OF THE REFERENCE NUMERALS

10: metric measurement input module 20: normalization module

30: defect prediction module 40: defect information collection module

50: predictive evaluation module

Claims (9)

A metric measurement input module for parsing an experimental source file and measuring the metric to detect refined metric measurement; A normalization module configured to normalize the metric measurement value detected by the metric measurement value input module or to perform a standard normalization process to calculate a normalized metric value; A defect prediction module for deriving a result of predicting the presence or absence of defects for each class of the test subject based on the implicit defect prediction model and the metric value normalized in the normalization module, A defect information collecting module that collects information on the actual defects of the subject by using the name of a class registered as defective among source contents based on defect information existing in a bug tracking system; And a predictive power evaluation module for evaluating the predictive power of the defect prediction model based on the result predicted by the defect prediction model in the defect prediction module and the information of the actual defective class collected by the defect information collection module. Software defect prediction calculation device using the metric. The method of claim 1, The defect prediction model is any one of an Olague, Yuming and Gyimothy model, the software defect prediction calculation apparatus using a metric, characterized in that the threshold for determining the defect of the class is defined for each model. The method of claim 1, The defect information is a software defect prediction calculation device using a metric, characterized in that any one of the information collected from bugzilla (bugzilla) provided by the Eclipse website and files provided by NASA and PROMISE. According to claim 1, Predictive power evaluation criteria of the defect prediction model of the predictive power evaluation module Precision, which represents the proportion of the classes correctly classified by the prediction model among all classes, Correctness, which represents the proportion of the classes that are actually defects among all the classes classified as defects by the prediction model, Software defect prediction calculation apparatus using a metric, characterized in that the completeness that represents the percentage of the total defects actually present in the class classified as defects in the prediction model. (a) collecting a source file that is an experiment target and measuring the measured metric to detect purified metric measurements; (b) normalize by normalizing by dividing the detected metric measurement into a predetermined size and unifying to make a normal distribution; Calculating a calculated metric value, (c) deriving a result of predicting the presence or absence of defects for each class of the test subject based on the calculated normalized metric value and the threshold value of the implicit defect prediction model; (d) collecting information on the presence or absence of actual defects of the test subject, independently of steps (a) to (c); (e) evaluating the predictive power of the defect prediction model based on the class information predicted in step (c) and the actual defective class information collected in step (d). How to calculate software defect predictions. The method of claim 5, wherein step (c) Equation

Probability of defects in a class in

), Comparing the calculated defect occurrence probability with a threshold value of the implicit defect prediction model; As a result of the comparison, the threshold is the probability of defect (

Is greater than or equal to), it is determined to be defective, and

) Is determined to be free from defects, wherein

Is a measure of an object-oriented metric that is an independent variable,

silver

Is the regression coefficient of,

Software flaw prediction calculation method using a metric, characterized in that the range of the value is from 0 to 1. The method of claim 6, The logarithmic value used in the equation is a software defect prediction calculation method using a metric, characterized in that to use the logarithm value expressed in the logistic regression model according to the used defect prediction model. The method of claim 5, wherein step (d) Collecting defect information from bugzilla provided by the Eclipse website and files provided by NASA and PROMISE, Extracting a name of a class registered as defective from source contents using the collected defect information; Increasing the number of defects by one each time the name of the class is extracted; And extracting the name of the class that is not present in the metric measurement or the wrong class, removing the name of the class from the actual defect information. The method of claim 5, The criterion for evaluating predictive power in step (e) uses precision, correctness and completeness, and determines the tradeoff between the accuracy and completeness as a threshold. A method for calculating software defect prediction using metrics, which calculates the accuracy and completeness of the model.

Images