JPH05233369A - Software reliability model selection device - Google Patents

Abstract

PURPOSE:To enable anybody to select a reliability estimation model which has a degree of conviction larger than a specified value among plural models at all times. CONSTITUTION:The device which selects the best model among plural kind of models when certifying reliability by a model for reliability estimation by testing software whose reliability is to be found and using data on the number of obtained inconvenient parts is equipped with a clustering means 1 which classifies estimated total inconvenience part quantity data classified by various models, an evaluating means 2 which performs reliability evaluation by weighting processing by inference based upon the criterion of an experienced person for classified data sets, an arithmetic means 3 which calculates the degree of contradiction between the classes by using the reliability evaluation result, a selecting means 4 which selects a class which has high evaluated reliability among reliability evaluation results when the reliability is allowed as a result of the contradiction arithmetic, and a means which outputs the selected class and model information belonging to the class.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION The present invention relates to quality assurance of software systems, and selects an optimal model from various existing reliability estimation models (mathematical models) used for software reliability verification. The reliability data of the software system to be tested calculated by the reliability estimation model to select data with higher confidence based on the test progress information. The present invention relates to a software reliability model selection device capable of accurately knowing the releaseable time of a software system and the number of man-hours required until then.

[0002]

2. Description of the Related Art In recent years, a system using a computer has penetrated deeply into society and has been used in various forms, and is now indispensable. by the way,
In a system using a computer, most of the causes of system down are related to software, and many are caused by program errors and algorithm errors that occur during the development of huge and complicated software.

It is difficult to completely eliminate errors (defects) inherent in these software even if various tests are performed. Of course, as tests are repeated, many defects are generally exposed, so it is crushed by correcting the exposed defects, and defects with high occurrence frequency are exhausted and resolved, but occurrence frequency is extremely high. Small defects are hard to find. And how much the developed software is defective in the current state, how long it will take to discover and fix it, how reliable is it, when will it be released, etc. There is a need to make such predictions. Therefore, various software quality evaluation techniques for increasing productivity and reliability of software development have been studied.

As a technique for estimating the reliability of software, "Information Processing" Vo
l. 32, No11, Nov. Page 1189 of 1991-
There are various types as shown on page 1200.

Typical of these are a software reliability growth model and a software complexity model. The software reliability growth model considers that latent defects or failures in the software system are discovered according to a certain mathematical model over the test time, and as a result, the reliability of the software increases and the software failure occurs. It means that the time interval becomes longer.

On the other hand, the software complexity model estimates the number of defects and failures inherent in the software system based on the complexity of the software system and the complexity of the development process.

However, in order to actually use these reliability estimation techniques, first, various reliability estimation techniques are used over a long period of time, and the results are compared and examined. You have to choose a technology that is suitable for the department you are going to use and customize it further.

[0008] Such utilization know-how is not only required to be organized and systematized, but it is usually closed to the public, so it is difficult to develop a reliability estimation technique suitable for one's own department, and it is not widely used. Not not.

[0009]

As described above, various mathematical reliability estimation models have been used to estimate the reliability of the developed software. However, the reliability estimation model depends on the development environment and the characteristics of the system, and various models have been used for that purpose.
Therefore, the problem has emerged that software developers who use multiple types of reliability estimation models do not know which model is the correct estimation result. In particular, due to the depth of experience, it is difficult to make an appropriate choice, and even experienced people often find it uncertain whether the choice was correct. If the wrong reliability estimation model is selected in the department, the obtained estimation result will be unreliable, which is a serious problem.

Therefore, the object of the present invention is to
We provide a software reliability model selection device that allows anyone to always select an estimated value with a certain degree of reliability by adopting the method currently experienced by people, and to provide a more reliable software system. It is to be able to calculate the reliability.

[0011]

In order to achieve the above object, the present invention is configured as follows. In other words, when testing the software for which you want to certify the reliability and certifying the reliability by the reliability estimation model using the obtained data on the number of defects, the optimal reliability estimation from multiple reliability estimation models As a device for selecting a model, a clustering means for classifying the estimated total number of defect data by model obtained by various reliability estimation models, and an experienced person's judgment criteria for the classified data set Evaluating means for weighted reliability evaluation based on the above inference, operation means for calculating the inconsistency between classes using the reliability evaluation result, and reliability when the inconsistency operation is allowed. Among the evaluation results, output the selection means to select the class with high reliability evaluation, the selected class and the reliability estimation model information belonging to that class. Configure and an output unit.

[0012]

In the above structure, first, the estimated total number of defects data for each model is clustered, the data having similar values are collected and classified, and the data set that is classified is weighted regarding the reliability. To do. Then, input the test item exhaustion status, the development period exhaustion status, the test progress status, and the amount of error in the estimated data at that time, and compare the representative value of each class with the judgment criteria of the experienced person. Quantify with "weight, certainty" as a weight. Then, the overall inconsistency is calculated from the weight of the confidence of each class (the overall weight of the inconsistency is calculated by relativizing the weights representing the confidence of each class and performing eigenvalue analysis on those weights. If there is no problem, select the most probable class and inform the reliability estimation model belonging to that class. As a result, anyone can always select a reliability estimation model having a certainty value or more.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of the present invention. In FIG. 1, D1 to Dn are reliability data for each model obtained by using various models. Reference numeral 1 denotes a clustering means for classifying reliability data, which is to say that similar pieces of reliability measurement data are grouped together. Reference numeral 2 is a reliability evaluation means, which reads a given external condition, evaluates it by the rule evaluation part, and judges true or false by the action analysis part.

Reference numeral 3 denotes a contradiction degree calculating means, which obtains the contradiction degree and the relative credibility degree of each class based on the absolute credibility data of each class. Reference numeral 4 denotes an estimated value selecting means, which obtains the reliability of the system from data such as the relative certainty factor of each class, the absolute certainty factor of each class, the representative value of each class, and the number of discovered defects.

An embodiment of the software system of the present invention will be described below with reference to the reliability data (total number of defects) shown in FIG. The reliability data in Fig. 2 is the number of existing defects (number of failures, defects, errors, etc. for each reliability estimation model such as exponential model and delayed S-shaped model. , The number of detected defects), the total number of defects (the test value predicted by the model), and the amount of error (the difference between the actual value and the theoretical value of the usage model).

FIG. 3 is a flow chart showing the flow of processing when selecting the optimum reliability estimation model in this apparatus, and FIG. 4 shows the calculation efficiency required for reliability selection by customizing the processing flow of FIG. It is an improvement. 5 and 6 are partially enlarged views of FIG. 4, and FIG. 7 is a flowchart showing a clustering procedure. The clustering (classification) means in FIGS. 3 and 4 performs clustering according to the procedure shown in FIG.

The operation of this apparatus will be described. First, the basic concept will be explained. Which one should be used as the optimum reliability estimation model for estimating the reliability (confidence) for the target software system (hereinafter referred to as the target system)? This is done by selecting from among several types, and this is done as follows. First, various tests are first performed on the target system by a test program for a predetermined period, and error detection and correction are repeated.

As a result, the obtained defect count data and conditions (test program scale, number of test items, number of test runs, used processor speed, used test man-hours, test time and software failure occurrence count, test time and software error) The number of detected data, the type and content of detected software errors, the number of software error corrections, the number of statement modifications, software error correction time, test coverage, software development progress, various data such as complexity) Calculated using the estimated number of defects at each test time point obtained by various software reliability estimation models (mathematical models) used at this time, the current actual number of defects, and the total number of defects estimated by the reliability estimation model. And the reliability (confidence) of the target system
Find the optimal software reliability estimation model for testing.

In this apparatus, the reliability data of the estimation for the target system by various software reliability estimation methods (various software reliability estimation models) are processed as follows.

First, as a premise, various software reliability estimation models (mathematical models;
Called the reliability model), the estimated number of defects at each test point and the current number of actual defects are calculated, and the difference between the two is calculated as the amount of error. Then, for each reliability model desired to be selected, data of the total number of defects estimated by the model and data of the error amount are prepared and used for the following processing.

[First Step] The reliability estimation results are clustered (S1 in FIG. 3). Clustering of estimated reliability data (data of estimated total number of defects in this example) for the same target system obtained using various software reliability estimation methods (various reliability models) (data with similar values are The estimated reliability data is classified into a set of similar data.

However, the reliability data taken up in this embodiment refers to the total number of defects, the total number of failures, or the reliability itself for each model estimated by various software reliability models. The discussion below proceeds assuming that the reliability data is the total number of defects.

[Second Step] When the clustering is performed in this way, the clustered data set is then weighted with respect to the reliability (confidence) (S2 in FIG. 3).

For this, the digestion status of the test items at that time, the digestion status of the development period, the progress status of the test, the error amount of the data estimated by the reliability model, etc. are input, and the judgment of the experienced person (expert) The representative value of each class will be digitized with the weight "how likely and likely" against the criteria.

[Third stage] Next, the overall degree of contradiction is calculated from the weight of the certainty factor of each class (S3 in FIG. 3). In this method, the weight representing the certainty factor of each class is made relative, and the eigenvalue analysis is performed on these weights to calculate the overall discrepancy.

[Fourth Stage] The most probable estimation data is selected (S4 in FIG. 3). This calculates the centroid value of the class with the highest reliability, and based on each reliability data of the estimation in all reliability models belonging to that class and the total number of defects found up to that point, Calculate confidence of class reliability.

When the reliability of the target system is estimated, the class to which the reliability model to which the reliability data of the estimation having the certainty value or more is always assigned is automatically calculated by performing the arithmetic processing according to the above procedure. It is possible to make a selection, and it is possible to show how much it is as a certainty factor of reliability by a concrete numerical value. Also, since it is easy to output the type of the reliability model of the belonging clustered to the selected class at the same time, it is possible to pick up and display the type (name) of the belonging model to display the optimum reliability. Can inform the model concretely. Although the above steps are taken, a more concrete example in which the calculation efficiency is improved is shown in FIGS.
Will be described next.

(Processing 1): An error amount of estimated data for each reliability model is selected within a certain standard (step S1). This processing is performed by the clustering means 1 by giving the reliability data of the estimation (in this example, the total number of estimated defects) and the error amount obtained by various reliability models together with the number of the reliability model.

Assuming that the reliability data estimation result as shown in FIG. 2 is given, for example, the reliability data, the error amount, and the reliability model number used to obtain the data are used as the clustering means. By inputting 1 into 1, the clustering means 1 excludes data considered to be useless, and clusters the remaining data when the number of data is four or more.

That is, first, an estimated value in which the error amount is less than the reference value, that is, an error value of the reliability data obtained by estimation by the reliability model showing a value less than the reference value is extracted, Whether or not to perform clustering is determined depending on how many pieces of data there are (S11). In the case of this example, since there are three classifications, the process proceeds to step S12 and clustering is performed only when the number of extracted data is four or more. Therefore, if the result of determination in step S11 is that the number of extracted data is one or less, the process proceeds to step S18, and
When the number is one or more and three or less, the process proceeds to step S14.

For example, if the selection reference value of the error amount is less than 1.0, in the example of FIG. 2, the data of [Exponential model 2] indicated by number No. 2 in the reliability data estimation result has the error amount. Since it is 1.0 or more, it is excluded from the target of clustering. The number of data whose error amount is within the selection reference value is 1
Since there is one, the process moves to step S12 to move to (Process 2). (Processing 2): Cluster the remaining data.

For example, when fuzzy clustering is adopted as the clustering method and the clustering is performed so that the number of classes is 3 or less, Table 1 is obtained. Then, even if the numbers of members of the respective classes are compared with each other, there is no great difference (procedure 3).

In Table 1, the numbers 1, 2, 3, ... 12 in the item of data number are the numbers in FIG.
The corresponding number of the item No. is shown, and the center-of-gravity value is the average of the total number of defects of each mathematical model collected in the class. Therefore, the mathematical model whose number Nos. In FIG. 2 are 1, 3 and 5 are included in the class 1, and the center of gravity value is said to be the value obtained by summing and totalizing the total number of defects in these mathematical models. It will be.

Since clustering collects what is considered to be appropriate for each class, there is a case in which a certain mathematical model applies to a plurality of different classes. Case numbers 10 and 11 are examples, and in this case, they belong to class 2 and class 3.

[(Table 1) Clustering result] ------------------ [Class name] [Data No.] [Center of gravity] −−−−−− + −− + −− + −− + −− + −− + −− + −−−−−−− Class 1 1 3 5 − − − − 79 .25 −−−−−− + −− + −− + −− + −− + −− + −− + −−−−−−−− Class 2 5 4 6 11 10 −12.40 −−−−− − + −− + −− + −− + −− + −− + −− + −−−−−−− Class 3 11 10 8 12 7 9 10.53 −−−−−−−−−−−− −−−−−−−−−−−−−−−−−−−−

The above (Processing 1) and (Processing 2) will be described in detail with reference to a flowchart. Estimation reliability data (that is, error amount data) D1 to DN for each model obtained by using various mathematical models
, The clustering means 1 determines the number of data whose error amount is less than the reference value (step S in FIGS. 4 and 5).
11) As a result, if it is “2” or “3”, S14
If it is "4" or more, clustering is performed by the fuzzy c-means method or the like (step S12 in FIGS. 4 and 5). If "0" or "1", S18
Move on to. The amount of error is calculated from the difference between the estimated value under the current conditions and the actual value obtained by calculation for each model.

The details of the clustering process in step S12 are as shown in FIG. 7, parameters are initialized (S31), and data within the reference value (in this example,
Import all relevant error amount data) (S3
2) When classifying these data into three classes, the data are grouped into the class considered to be optimum while searching for the optimum area division (S33 to S35). Even if a piece of data spans multiple classes, it is allowed.

When the clustering processing in step S12 is completed, the number of elements of a class (the number of data items belonging to that class) is 1 which is relatively larger than the other classes.
It is checked whether or not one exists (step S1 in FIGS. 4 and 5).
3) If not present, move to S14, and if present, move to step S21.

In step S21, the representative value of the class is selected (that is, the centroid value belonging to the class is obtained in this example). Then, the process proceeds to step S17 (shift to (Process 5)).

If the result of the determination in S11 is "0" or "1", it is checked whether the error amount is less than the reference value (step S18 in FIGS. 4 and 5). As a result, if there is a value exceeding the reference value (No), the value estimated by the complexity model is selected (step S1 in FIGS. 4 and 5).
9) and then to S17 (shift to (Process 5)).

If the result of determination in step S18 is that there is no value exceeding the reference value (Yes), that value (error amount) is selected (step S20 in FIGS. 4 and 5) and the process moves to S17 ((process 5)). Transition to).

In step S20, the estimated total defect number is selected as it is (because the number of elements in the class is one). Then, the process proceeds to step S17 (shift to (Process 5)). (Processing 3): Weighting regarding the certainty factor with respect to the estimated value is performed (step S14 in FIGS. 4, 5, and 6). This is performed by the confidence measure evaluation means 2.

In step S14, the centroid value of the estimated total number of defects in each class clustered within three (when the number of elements in the class is plural) or the estimated total number of defects in each class (the number of elements in the class is 1 Two cases) as reliability data. Then, the reliability data for each class is weighted.

For example, using the type of test, the digestion rate (%) of the test item, the digestion rate (%) of the test period, the test coverage (%), etc. as input variables, the judgment conditions of the expert and the action at that time are set as IF. -It is described as a THEN rule, and weighting is performed based on the distance from the representative value (for example, the center of gravity value) of each class. As the IF-THEN rule, there is one as shown in FIG. 8, which allows processing corresponding to a human judgment standard.

In FIG. 8, I indicates a class name, K1 and K2 are coefficients, WEIGHT is a class weight, and DISTANCE is a function name, which is a function for obtaining a weight from the distance between the center of gravity and the number of defects. ..

Then, from the upper and middle IFs to END
Up to IF is a routine for obtaining weights according to test types, and from IF to ENDIF in the lower stage is an example of a routine for obtaining test time digestibility and model weights. The data in Table 3 can be obtained by adding the result of this processing and the data in Table 2.

In the case of the specific example, if the test type is a functional test and the digestion rate of the test items is 80%, and K1 is 0.1, the total number of defects is 100 × 10 ÷ 80 = 12.5.
Therefore, if this value is used as an experience value and the distance between it and the center of gravity of each class is calculated, the result is as follows. The experience value is obtained by preparing an optimal calculation formula according to various conditions such as the test period, the number of tests, and whether the target software is newly developed or upgraded.

[(Table 2) Weighting result for reliability data] ------------------------- [Class name] [Weight] Cardiac value] [Experimental value] [Distance] -------- + ---------- + -------- + ------- Class 1 79.25 12.50 66 .75 −−−−−− + −−−−−− + −−−−−−− + −−−−−−− Class 2 12.40 12.50 0.10 −−−−−−−− −−−−− + −−−−−−− + −−−−−−− Class 3 10.53 12.50 1.97 −−−−−−−−−−−−−−−−−− −−−−−−−−−−−

(Processing 4): The overall contradiction is calculated (step S15 in FIGS. 4 and 6). This is the contradiction calculation means 3
Done by Here, (process 3), that is, the certainty factor obtained in step S14 is made relative to create a pair comparison matrix. Then, eigenvalue analysis is performed based on the paired comparison matrix to calculate the degree of contradiction. Table 3 shows a pairwise comparison matrix, and Table 4 shows the certainty factor and the overall contradiction factor when each class is relativized using eigenvalue analysis.

[(Table 3) Paired Comparison Matrix] -------------------------- ************ [Class 1] [Class 2] [Class 3] −−−−−− + −−−−−− + −−−−−− + −−−−−− Class 1 ＊＊＊＊＊ 1/9 1/9 −−−− −− + −−−−−− + −−−−−− + −−−−−− Class 2 ＊＊＊＊＊＊＊＊＊＊＊ 3 −−−−−− + −−−−−−− + −−−−−− + −−−−−− Class 3 ＊＊＊＊＊＊＊＊＊＊＊＊＊＊ －−−−−−−−−−−−−−−−−−−−− However, in Table 3, rows are "similar" when 1 is compared to columns, "slightly good" when 3, "pretty good" when 5, and 7 when compared to columns. Indicates "very good", 9 means "much better", fraction means vice versa.

[(Table 4) Eigenvalue calculation results] ------------------------- [Class name] [Weight] -------- − + −−−−−−−− Class 1 0.05 −−−−−−−−−− + −−−−−−−− Class 2 0.64 −−−−−−−− −−− + −−−−−−−− Class 3 0.31 −−−−−−−−−−−−−−−−−−−− However, matching degree = 0.068, matching ratio = It is 0.12.

Here, the overall degree of contradiction is represented by the degree of matching and the matching ratio, and if either of them is 0.15 or more, there is an unacceptable contradiction in the weighting of reliability data. Notify the user to that effect. In the case of this example, it can be seen that it is sufficiently within the allowable range. (Processing 5) The reliability is calculated (steps S16 and S17 in FIGS. 4 and 6). This is performed by the estimated value selection means 4.

This is the absolute certainty factor obtained in S14 of (Process 3), that is, the distance and the representative value of the class having the highest relative certainty factor obtained in S15 of (Process 4) (reliability data). , For example, the center of gravity value is selected (step S16 in FIGS. 4 and 6).

Then, using the data selected in step S16 (or step S20 or step S21), the reliability (certainty) of the target system is calculated based on the following equation (steps in FIGS. 4 and 6). S17).

In the case of the specific example, since the certainty factor of [class 2] is the highest, the total number of defects of this target system is 12.4.
The reliability (certainty) is reliability (%) = (number of defects already found / total number of defects) * 100 (Equation 1)

Therefore, the reliability is (10.0 / 12.4) * 100 = 80.6 (%).

As a result, the current reliability of the target system could be estimated. Therefore, the class having this reliability (%) and the reliability model belonging to the class are output from the output means together with the estimated value of the reliability (%) to notify the software developer of the reliability to be used. You can objectively tell which is the degree model. Further, by outputting the representative value of the class with the highest certainty degree obtained in S16, it is possible to notify the number of remaining defects.

As described above, it is possible to select and inform the optimum model to be used for the verification from among the plurality of software system reliability estimation models. Even beginners will be able to select the most suitable model to be performed by a skilled person, regardless of experience. This makes it possible to perform high reliability management at an important final step in the software system development process. It is needless to say that the present invention is not limited to the embodiments described above and shown in the drawings, and can be appropriately modified and carried out within the scope of the invention.

[0060]

As described above in detail, according to the present invention, anyone can always select a reliability estimation model having a certainty value or more.

[Brief description of drawings]

FIG. 1 is a system block diagram showing an embodiment of the present invention.

FIG. 2 is a diagram showing an example of reliability data for each reliability model.

FIG. 3 is a flowchart showing the flow of processing when the optimum reliability model selection is carried out in the apparatus shown in FIG.

FIG. 4 is a flowchart in which the processing flow of FIG. 3 is customized to improve the calculation efficiency required for reliability selection.

5 is a partially enlarged view of FIG.

6 is a partially enlarged view of FIG.

FIG. 7 is a flowchart showing details of an example of clustering processing.

FIG. 8 is a diagram showing an example of an IF-THEN rule used in the device of the present invention.

[Explanation of symbols]

1 ... Clustering means, 2 ... Reliability evaluation means, 3 ... Contradiction degree calculation means, 4 ... Estimated value selection means.

Claims (1)

[Claims] 1. When testing the software for which the reliability is desired and certifying the reliability by the reliability estimation model using the obtained data of the number of defects, it is possible to select the optimum reliability estimation model from a plurality of types. As a device for selecting the reliability estimation model, a clustering means for classifying the estimated total number of defect data by model obtained by various reliability estimation models, and judgment of experienced persons for the classified data set Evaluation means for performing reliability evaluation weighted by inference based on criteria, operation means for calculating the inconsistency between classes using the reliability evaluation result, and the result of the inconsistency operation is acceptable. Occasionally, among the reliability evaluation results, a selection method for selecting a class with a high reliability evaluation and the selected class and reliability estimation model information belonging to that class are output. Software reliability model selection apparatus characterized by comprising an output means for.