JP4646248B2 - Program inspection item generation system and method, program test system and method, and program - Google Patents

Description

  The present invention relates to a technique for inspecting the quality state of a program used in a computer system by a programmed computer, and in particular, a program inspection technique suitable for efficiently and accurately analyzing the quality state of the program by a black box test. It is about.

  The black box test focuses on only the input and output, and confirms whether or not output according to the specifications can be obtained for various inputs.When performing program inspection in this black box test, Inspection items necessary for the inspection are created in advance. Conventionally, this search item is created based on an inspector's empirical rule or based on an inspection item of a similar program. For this reason, the inspection using such inspection items is an unreliable inspection because the evaluation is inconsistent depending on the ability of the inspector.

  In addition, for inspections that occur suddenly, it is necessary to understand the functions and prepare inspection items. Unless a limited inspector who understands the functions, it is necessary to prepare appropriate inspection items. It takes a huge amount of time and cannot be evaluated quickly. For this reason, it was a situation where only a limited number of inspectors could deal with an inspection that occurred suddenly.

  Furthermore, in addition to preparing a design document in advance for information to be input for inspection, information such as past inspection items is necessary. Without such input information, neither inspection item generation nor inspection evaluation is possible. Can not.

  As a technique for solving such a problem, for example, Patent Document 1 describes a technique for automatically generating an inspection item regardless of the skill of an inspector. However, with this technique, inspection data must be defined and input in advance, and furthermore, since this inspection is limited to a specific inspection object, inspection items cannot be created for general purposes. In addition, it is not possible to carry out an inspection focused on the evaluation target viewpoint (inspection purpose), and a specific inspection with a narrow scope cannot be performed because the entire inspection is performed. Therefore, it does not lead to further efficiency improvement.

  Patent Document 2 describes a technique for inputting a quality element and comparing it with a reference value in order to perform quality evaluation. However, with this technique, evaluation cannot be performed when no quality factor is set, and accurate evaluation may not be performed when the reference value is ambiguous. In this technique, the quality factor and the reference value are indispensable conditions as input information and cannot be evaluated alone. Note that the evaluation by itself means that a test is performed and an evaluation is made based on the result, not an evaluation based on information from a third party.

JP 2000-305811 A JP 09-265393 A

  The problem to be solved is that in the conventional technology, when performing black box test inspection of a program, it is necessary for an inspector to understand the function and create inspection items, and the inspection time becomes enormous. In addition, some inspectors cannot perform accurate tests due to omissions in inspection items due to lack of understanding, and the evaluation method is not uniform, resulting in variations in evaluation results among each inspector. is there.

  An object of the present invention is to solve these problems of the prior art, to reduce the inspection time in the black box test of the program, and to enable an accurate inspection that is not influenced by the skill of the inspector.

  In order to achieve the above object, in the present invention, a user (inspector) based on pre-registered general-purpose “inspection purpose (inspection viewpoint)” and “inspection item” information necessary for actual inspection. A GUI screen for selecting a viewpoint (purpose) to be evaluated is generated and displayed, and an inspection item is generated by selecting a viewpoint (purpose) that the user (inspector) wants to evaluate via this GUI screen. Thereby, digestion (inspection), analysis, and evaluation of items in a short time become possible. For example, when “I want to determine whether the input check of the target system is valid”, the inspector simply selects the inspection item from the viewpoint necessary for “input check” on the GUI screen, and only the selected inspection item is selected. As a result, the time required for digestion from an unnecessary viewpoint can be reduced, and any personnel can digest the same test item.

  According to the present invention, it is possible to reduce the time required for the preparation for the program inspection, and it is possible to uniformly perform the test inspection item determination and the test evaluation, and to carry out an efficient and accurate program inspection. It is possible.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. First, each figure will be described.

  FIG. 1 is a block diagram showing a configuration example of a program test system according to the present invention, and FIG. 2 is an explanatory diagram showing a configuration example of a viewpoint / item DB managed by the data storage unit of the server computer in FIG. FIG. 3 is an explanatory diagram showing a configuration example of the selection information DB managed by the data storage unit of the server computer in FIG. 1, and FIG. 4 is target information managed by the data storage unit of the server computer in FIG. FIG. 5 is an explanatory diagram illustrating a configuration example of the evaluation point DB managed by the data storage unit of the server computer in FIG. 1, and FIG. 6 is a data storage of the server computer in FIG. FIG. 7 is a flowchart showing a processing operation example according to the present invention of the program test system in FIG. It is a door.

  8A to 8F are explanatory diagrams showing an example of an evaluation viewpoint selection operation by the program test system in FIG. 1, and FIG. 8A is an example of a form output screen in the evaluation viewpoint selection operation by the program test system in FIG. FIG. 8B is an explanatory diagram showing an example of key acquisition operation in the evaluation viewpoint selection operation by the program test system in FIG. 1, and FIG. 8C is a viewpoint in the evaluation viewpoint selection operation by the program test system in FIG. FIG. 8D is an explanatory diagram showing an example of the item database search operation, FIG. 8D is an explanatory diagram showing an example of the selection information database registration operation in the selection operation of the evaluation viewpoint by the program test system in FIG. 1, and FIG. 8E is an evaluation by the program test system in FIG. Selection information database and view in selection action of viewpoint - Item explanatory diagram showing a search operation example of a database, Fig. 8F is an explanatory view showing a form output screen example displayed output in the evaluation point of view of selecting operation of the program test system in FIG.

  FIG. 9 is an explanatory diagram showing an example of evaluation processing operation by the program test system in FIG. 10A to 10D are explanatory diagrams illustrating a specific example of the evaluation processing operation in FIG. 9, and FIG. 10A is an explanatory diagram illustrating an example of the importance rank setting operation before the start of the examination in the evaluation processing operation in FIG. FIG. 10B is an explanatory diagram illustrating an example of setting an evaluation point in the evaluation processing operation of FIG. 9, FIG. 10C is an explanatory diagram illustrating an example of an evaluation determination operation after completion of the inspection in the evaluation processing operation of FIG. 9, and FIG. FIG. 10 is an explanatory diagram illustrating an example of an overall evaluation determination operation after completion of an inspection in the evaluation processing operation of FIG. 9.

  11A to 11C are explanatory diagrams illustrating a specific example of the overall evaluation determination operation after completion of the inspection in FIG. 10D. FIG. 11A illustrates the functional evaluation and the viewpoint evaluation in the overall evaluation determination operation of FIG. FIG. 11B is an explanatory diagram illustrating a specific operation example of the comprehensive evaluation, FIG. 11B is an explanatory diagram illustrating an operation example of the viewpoint-specific evaluation and the function-specific evaluation in FIG. 11A, and FIG. 11C is a function-specific evaluation in FIG. It is explanatory drawing which shows the operation example of comprehensive evaluation.

  12A is an explanatory diagram illustrating an example of individual view processing operation based on the operation results of evaluation by function, evaluation by viewpoint, and comprehensive evaluation in FIG. 11A, and FIG. 12B is a specific example of the individual view processing operation in FIG. 12A. It is explanatory drawing which shows an example.

  FIG. 13 is an explanatory diagram showing an example of an output processing operation for evaluation results by the program test system in FIG.

  The program test system in FIG. 1 performs program quality evaluation by a programmed computer, and includes a computer configuration including a CPU (Central Processing Unit), a main memory, a display device, an input device, and an external storage device. Each processing unit is installed by installing a program or data recorded in a storage medium such as a CD-ROM via an optical disk drive or the like in the external storage device, then reading the program or data from the external storage device into the main memory and processing it by the CPU. Perform the function.

  That is, as shown in FIG. 1, the program test system of the present example includes a client computer 100 and a server computer 105 connected via a network, and the program test system performs processing based on the programs of the client computer 100 and the server computer 105. The inspection item generation and automatic evaluation of the program according to the invention are performed.

  In this example, the inspection person in charge uses the client computer 100 to interactively generate and automatically inspect inspection items, and data (information) is accumulated in the server computer 105 to reduce the load.

  During work, the client computer 100 and the server computer 105 are connected via a LAN (Local Area Network) to exchange information.

  The client computer 100 includes an input / output unit 101, a form output unit (described as “form output” in the figure) 102, a data control unit 103, and a processing unit 104. The data control unit 103 receives information and instructions input by the person in charge of inspection from the input screen, and passes them to the processing unit 104. In the processing unit 104, whether the input process is a new input process, inspection item digestion, evaluation analysis (during completion) It is judged whether it is.

  The processing unit 104 includes a data selection distribution processing unit (described as “data selection distribution processing” in the drawing) 104 a and an evaluation analysis 1 processing (remaining estimation) unit (“evaluation analysis” in the drawing) as processing functions of the programmed computer. 104b, report edit processing section (described as “report edit processing” in the figure) 104c, information storage processing section (described as “information storage processing” in the figure) 104d, evaluation analysis 2 processing (result evaluation) section (described as “evaluation analysis 2 processing (result evaluation)” in the drawing) 104e, viewpoint item editing processing section (described as “viewpoint item editing processing” in the drawing) 104f, result registration processing section (shown in FIG. 104 g and “screen form display processing” (shown as “screen form display processing” in the figure) 104 h. The data is transferred to each processing unit according to the determination result. Acquire / edit.

  The server computer 105 includes a data control unit 106 and a data storage unit 107 as processing functions of the programmed computer.

  When the processing unit 104 acquires / edits data in each processing unit (104a to 104h) according to the determination result, information necessary for each processing unit (104a to 104h) is stored on the server computer 105 side. If it exists in the data storage unit 107, it is accessed via the data control units 103 and 106, and data acquisition (reading) and registration (writing) are performed.

  After data editing, the processing unit 104 displays the result on a display screen such as a CRT by the input / output unit 101 and prompts the person in charge of the inspection to input the next process. When the person inspecting the operator operates the input device such as a keyboard or a mouse to request a form output, the request information is input via the input / output unit 101, and the same transition as described above is performed. The result is output by a printer or the like via the form output unit 102.

  The data selection / distribution processing unit 104a in the processing unit 104 selects an inspection purpose (inspection viewpoint) and an item to create an inspection item, performs an evaluation point distribution process, and the like, and performs an evaluation analysis 1 process (remaining estimation). The unit 104b performs residual defect estimation processing, the report edit processing unit 104c performs report editing / printing processing, and the information storage processing unit 104d stores the input information (function, etc.). The evaluation analysis 2 process (result evaluation) unit 104e performs an evaluation analysis process using the inspection item digestion result, and the viewpoint item edit processing unit 104f adds / updates the inspection viewpoint and items. -The deletion process etc. are performed, the result registration processing unit 104g performs the storage processing of the examination item digest result and the evaluation result in the storage device, and the screen form display processing unit 104h displays the screen used in each processing Performing a display processing, and the like of the form.

  A data storage unit (described as “database” in the figure) 107 in the server computer 105 has a database function, and includes a viewpoint / item DB (described as “viewpoint / item” in the figure) 107 a and a selection information DB (in the figure). 107b, target information DB (denoted as “target information” in the figure) 107c, evaluation point DB (denoted as “evaluation point” in the figure) 107d, opinion information DB (denoted as “opinion information” in the figure) (Description) Each database information of 107e is managed.

  In the viewpoint / item DB 107a, all inspection viewpoints / items are stored with the configuration contents shown in FIG. 2, and in the selection information DB 107b, information evaluated and selected by the target system with the configuration contents shown in FIG. 3 is stored, and in the target information DB 107c. 4, the input information of the target system is stored with the configuration content shown in FIG. 4, the evaluation point DB 107 d stores the information used as the basis for various evaluations with the configuration content shown in FIG. 5, and the opinion information DB 107 e stores the information shown in FIG. 6. The wording of the opinion decided for each viewpoint / evaluation is stored in the structure shown.

  With such a configuration, in the program test system of this example, an inspection item used in a black box test of the program is generated by a programmed computer based on an interactive operation with a user (inspector).

  That is, in the server computer 105, each inspection item is divided into a plurality of groups according to the purpose of inspection and stored as viewpoint / item information in advance in the viewpoint / item DB 107 a of the data storage unit 107. GUI screen which displays the inspection item / item information from the viewpoint / item DB 107a by the data selection / distribution processing unit 104a of the processing unit 104, and displays the inspection items in a hierarchy by inspection purpose. Is generated and displayed via the input / output unit 101, and inspection item list information including inspection items selected by the inspector on the GUI screen is generated.

  Then, the processing unit 104 of the client computer 100 inputs the inspection item list information generated in this way by the evaluation analysis 2 processing unit 104e, and inspects the program based on the inspection item included in the inspection item list information. And analyze the inspection results to evaluate the quality of the program.

  At this time, the evaluation analysis 2 processing unit 104e reads out the weighting information for each inspection purpose and each inspection item stored in advance in the evaluation point DB 107d by the server computer 105 and refers to the inspection result of each inspection item. The weight is added to evaluate the quality of the program.

  Further, the processing unit 104 of the client computer 100 uses the evaluation analysis 1 processing unit 104b to predict the number of defects remaining in the program by a statistical method using the inspection result information during the program inspection by the evaluation analysis 2 processing unit 104e. Calculate the value. For example, the number of inspection digests, the number of defective extractions, and the risk rate are included as intermediate inspection result information, and the evaluation analysis 1 processing unit 104b remains in the program with a binomial probability using the number of inspection digestions, the number of defective extractions, and the risk rate. A predicted value of the number of defects to be calculated is calculated.

  Hereinafter, an example of the inspection item generation processing operation and the automatic evaluation processing operation by the program test system having the above configuration will be described with reference to FIG.

  FIG. 7 shows an example of processing procedures of the client computer 100 and the server computer 105.

(1) In the process of starting the inspection evaluation system (step S200), the inspection person starts the inspection item generation / automatic evaluation apparatus on the client computer.

(2) In the system information registration (step S201) process to be evaluated, basic information (system name, scale, function name, etc.) of the system (target system) to be evaluated this time is input from the initial screen.

(3) In the selection (viewpoint) (item) (steps S202 and S203) of the evaluation viewpoint (purpose of inspection), the inspection purpose (evaluation viewpoint) of the system to be evaluated is selected. There are two types of evaluation viewpoints: “select all viewpoints (step S204)” that targets the entire system and “select partial viewpoints (step S205)” that allow selection of individual viewpoints according to the evaluation purpose. .

  Since “selection of partial viewpoints” allows you to select multiple combinations of viewpoints, for example, when you want to perform an evaluation specialized for screen functions, you can narrow down verification (evaluation) by selecting individual viewpoints. .

  The selection of the evaluation viewpoint consists of a large item “point of view” and a small item “item”. By selecting the large item “point of view”, the detailed small item “item” is displayed, and from that, it further matches the evaluation. Since the selected item can be selected, evaluation analysis can be performed from a plurality of angles.

(4) The item automatic generation (step S206) processing is automatically performed after the evaluation viewpoint selection is confirmed. The role of this process is to generate the “inspection item” necessary for the inspection work based on the selection result of the viewpoint / item performed in “selection of evaluation viewpoint (steps S202 and S203)”.

  This “inspection item” includes items that become inspection conditions when verifying the evaluation target system, items that are confirmed to be valid, items that indicate when and for each function / item (the confirmation result is a problem) An item indicating whether or not there is included is included in the left column).

(5) In the item digest (step S207) process, verification (inspection) is actually started based on the item created in the item automatic generation process (step S206). Perform verification for each viewpoint / item and register the result.

(6) In the digestion completion (step S208) process, the process branches depending on whether or not the inspection item has been digested. The following (7) and (8) show processing after branching.

(7) The evaluation generation (remaining prediction type) (step S209) process is a process in the case where the digestion of the inspection item is started in the middle of the preprocess (step S208). The evaluation determination is processed as described above. Furthermore, in this process, a function is implemented that predicts (automatically) remaining defects in advance while checking the inspection items, and supports actions that prompt the developer to improve quality in a timely manner. The following two methods (a) and (b) are prepared as remaining defect prediction methods (details will be described with reference to FIG. 9).

(B) Prediction with binomial probability: An automated binomial probability paper used in statistics.

(B) Prediction from execution area: Based on the viewpoint, scale, and number of defects implemented to date, the rate of increase when all inspection items and functions are implemented is calculated, and the remaining defects are predicted in proportion to the number of defects.

(8) The automatic evaluation & view (step S210) process is a process that is activated regardless of whether or not all items have been digested in the preprocess (step S208). Based on the pass / fail of the inspection results described in the “Confirmation column” in the inspection item, it is calculated for each point of view / item. Separately, evaluation determination is performed, and further overall evaluation determination is performed by the same processing. The evaluation contents (results) are displayed (viewed) on the screen.

(9) In the report (step S211) process, the process branches depending on whether the evaluation result is generated as a report. The next (10) shows the processing when report generation is selected.

(10) The report generation (step S212) process is a process performed when report generation is started in the preprocessing (step S211). This process automatically creates a report so that the evaluation results can be submitted to a third party.

(11) In the evaluation system termination (step S213) process, when the processes such as digestion of test items and report creation are completed, the test evaluation system is terminated. There is no particular processing at the end.

  Next, with reference to FIGS. 2, 8A to 8F, FIG. 9, FIGS. 10A to 10D, FIGS. 11A to 11C, FIGS. 12A, 12B, and 13, an inspection item by the program test system of this example is used. A specific example of the generation process and the automatic evaluation process will be described.

  8A to 8F show specific processing operations when the evaluation viewpoint selection processing in steps S202 and S203 in FIG. 7 is performed.

  When the target system is inspected, it is necessary to determine an evaluation method (inspection viewpoint) according to the purpose of the inspection at the time of inspection planning. First, the person in charge of inspection receives the “perspective to be evaluated this time” and the “perspective of evaluation required by a third party”, and selects the pertinent viewpoint from the evaluation viewpoint selection screen 300 which is a GUI screen shown in FIG. 8A. .

  The evaluation viewpoint can be broadly divided into “system overall evaluation version” and “individual evaluation version” (major items). "System-wide evaluation version" refers to all aspects of the system to be evaluated during normal inspection (that the system operates as requested by the customer, that the operation is considered from a general viewpoint, and that the system operates normally. In addition, it is possible to evaluate the product as a product by setting a system that takes into account the irregular case.

  The “individual evaluation version” is intended to evaluate a product from a specific viewpoint. For example, it is assumed that there are many problems in the screen input / output system in the testing process during development, and that the third party is concerned about the quality of the part.

  In such a situation, when an inspection department is requested to quickly determine the quality of a specific location, the inspector selects “screen input / output emphasis” of the individual evaluation version on the evaluation viewpoint selection screen 300 in FIG. By selecting (item), the minimum necessary viewpoint can be automatically selected, and it is a mechanism that can realize efficient evaluation.

  Further, in the evaluation regarding “screen input / output”, it is possible to select a more detailed evaluation viewpoint, such as focusing on “input check” (small item).

  After the evaluation viewpoint is selected in this way, the following processing procedures 301 to 304 shown in FIGS. 8B to 8F are executed as internal logic to generate an inspection item 305.

(A) In accordance with the selected item, a work key used when acquiring information from the “viewpoint / item DB 107a” shown in detail in FIG. 2 is set (processing procedure 301 in FIG. 8B).

  “B” in the processing procedure 301 is a flag for indicating that it has been checked. When the middle item is checked, the “key flag 1” of the viewpoint / item DB 107a is referred to. When the check is made for each small item, the “key flag 2” of the viewpoint / item DB 107a is referred to. The key is set. Key flag 1 corresponds to NO1, and key flag 2 corresponds to NO2.

  (B) The “viewpoint / item DB 107a” is searched based on the acquired work key, and the matching information is acquired (processing procedure 302 in FIG. 8C).

  This processing procedure 302 is a process for determining which viewpoint has been selected. For example, when “screen input emphasis” and “input check” are checked, the key flag 1 of the viewpoint / item DB 107a = “ The information of NO1 corresponding to “B” is acquired, and the information of NO2 corresponding to the key flag 2 = “B” is acquired.

  (C) Compare / merge the “target information DB 107c” from the acquired information, and reflect the result in the “selection information DB 107b” (processing procedure 303 in FIG. 8D). Here, a viewpoint is set for each function, and the information is stored in the selection information DB 107b.

  (D) Information for screen display is acquired from the “selection information DB 107b” and the “viewpoint / item DB 107a” (processing procedure 304 in FIG. 8E).

  (E) The information acquired in the processing procedure 304 of (d) described above is generated and displayed as the inspection item 305 in FIG. 8F.

  The inspection item generation screen displayed as the inspection item 305 in FIG. 8F is a result of automatically generating “inspection items” necessary for the inspection from the result of selecting the viewpoint. Since this inspection item is output in, for example, a table in Excel (registered trademark) format, it is possible to bring it into an external device such as another computer.

  8F includes “item number”, “large item”, “medium item”, “confirmation date”, “confirmer”,..., “Confirmation date”, “confirmer”. “Item No.” is an ID for identifying an inspection item, “Large item” is a more detailed inspection viewpoint, “Middle item” is the confirmation content of the inspection item, “Confirmation date” is each inspection A field for entering the date on which the item was confirmed, “Confirmer” is a field for entering the name of the person in charge who confirmed each inspection item, and the “Date of confirmation” and “Confirmer” fields are entered for each function. Have taken.

  FIG. 9 shows a specific processing operation when the evaluation generation (remaining defective type) in step S209 of FIG. 7 is performed by the evaluation analysis 1 processing (remaining estimation) unit 104b in the processing unit 104 of the client computer 100 of FIG. It is explanatory drawing.

  The evaluation analysis 1 process (remaining estimation) unit 104b derives a remaining defect prediction based on the information 400 in FIG. In addition, when calculating information, the calculation is performed for the entire system / for each function. Further, in the calculation for each function, information is acquired using “function information” as a key.

  The information 400 in FIG. 9 includes the number of defects (n defects), the number of all items (m items of all items), the number of undigested items (number of undigested items x), the number of functions (number of functions y), and the risk rate. (Risk rate 5%) and scale (scale a step).

  The “number of defects” is an item whose check item confirmation date is “NG”, and is acquired from the “confirmation date” in the selection information DB 107b of FIG. 3 by SUM calculation of the number of “NG” items.

  “Number of all items” is all items of the inspection items generated from the selected viewpoint, and the number of rows in the selection information DB 107b of FIG. 3 is counted and acquired.

  “The number of undigested items” is an item for which the confirmation date of the examination item has not been input, and is obtained by SUM calculation of the NULL number from the “confirmation date” in the selection information DB 107b of FIG.

  The “number of functions” is the number of functions registered in the system information for the first time, and the total number of “function information” in the target information DB 107c of FIG. 4 is acquired.

  The “risk rate” is a statistical method used when the evaluation analysis 1 processing (remaining estimation) unit 104b in FIG. 1 calculates a predicted value of the number of defects remaining in the program based on the inspection result information during the program inspection. This information is necessary when calculating with the “binary probability”, and in this example, 5% of a general numerical value is set.

  The “scale” is the number of development steps of the program to be evaluated, and is obtained by calculating the SUM from the “scale” in the target information DB 107c of FIG.

  The residual failure prediction by the evaluation analysis 1 processing (remaining estimation) unit 104b in FIG. 1 can be calculated from two statistical methods of prediction based on “binary probability” and prediction based on “implemented area”. The person in charge (inspector) can select and use a prediction method suitable for the evaluation.

  The procedures of “prediction with binomial probabilities” and “prediction from implementation area” will be described below.

  First, in the calculation procedure 401 of “predict with binomial probability”, necessary information includes “total number of items”, “number of undigested items”, “number of defective items: n”, “risk rate: X”, “total number of items”. : M ”and substituting these necessary information into each calculation formula as follows to calculate the residual defect (for each system / function).

  (1) a = number of all items−number of undigested items

  (2) A ′ = a / number of defects n

  (3) A1 = A ′ + (Risk factor X × (SQRT (A ′ (1-A ′) / a)))

  (4) A2 = A ′ − (Risk factor X × (SQRT (A ′ (1-A ′) / a)))

  (5) Residual defect 1 = total number of items m × A1

  (6) Residual defect 2 = total number of items m × A2

  (7) “Residual defect 1” is the number of remaining defects. “Remaining defect 2” is a numerical value when the binomial probability sheet 403 shown in FIG. 9 is created.

  Next, in the calculation procedure 402 of “predict from the execution area”, as necessary information, “total number of items m1 for each function”, “number of undigested items for each function r1”, “number of defective cases n”, “total scale d” The necessary information is substituted into each calculation formula as described below to calculate a residual defect (for each system / function).

  (1) Scale of b = SUM (all items for each function (total number of items for each function m1−number of undigested items for each function r1> 0))

  (2) c = SUM (b × (1− (r1 / m1)))

  (3) B = number of defects n / b × total scale d

  (4) e = 1- (SUM (1- (r1 / m1)))

  (5) B1 = e × B

  (6) “B” obtained in this way is the number of remaining defects for the target function, and “B1” is the number of remaining defects in the entire system. An implementation area diagram 404 shown in FIG. 9 is created based on “B”, “B1”, and “c”.

  After performing the above-described calculation logic processing, for example, graph creation processing is performed using the graph function of Excel (registered trademark).

  Next, an evaluation generation (completion type) process in step S210 in FIG. 7 is performed by the evaluation analysis 2 process (result evaluation) unit 104e in the processing unit 104 of the client computer 100 in FIG. 1 using FIGS. 10A to 10D. A specific processing operation in this case will be described.

  Prior to the start of inspection, an importance rank is set for each target function. The importance rank setting criteria is set in consideration of the influence of the function on the customer as the importance rank information 500 shown in FIG. 10A.

  In addition, each evaluation point shown in the evaluation point information 501 in FIG. 10B is assigned in advance to each selected viewpoint / item, and a weight is automatically set for each viewpoint / item.

  After the inspection is completed, automatic determination is performed based on the inspection result (OK / NG for each inspection item) and the defect rate point information 502 shown in FIG. 10C, and the evaluation of each function and the entire system is performed. The evaluation processing procedure 503 is executed for each function in FIG. 10D and comprehensively so that the evaluation of each function and the entire system can be visually performed.

  In addition, this process is implemented irrespective of digestion of all the inspection items.

  The evaluation processing procedure 503 for each function and in FIG. 10D will be described with reference to FIGS. 11A to 11C and FIGS. 12A and 12B.

  “Premise”: Information that retains information as evaluation criteria is as follows.

  (I) Importance rank (manual input): 4 types of A to D.

  (Ii) Evaluation points (automatic setting): 5 types a to e.

  (Iii) Defect ratio point (automatic counting): 0 to 10 points are allocated based on the processes in “predict with binomial probability” and “predict from execution area”.

  (Iv) Percentage of defectives (automatic counting): range setting from 0% to 100%.

  (V) Rank weighting (automatic counting): 0% to 15%, and assigned to the rank of circle 1 in the evaluation table 601 of FIG. 11A.

  (Vi) Evaluation determination (automatic determination): 5 types such as “circle”, “triangle”, “x” and the like are determined for each function and the entire system.

  First, “Evaluation Method 1” will be described with reference to FIGS. 11A to 11C. In this “evaluation method 1”, tabulation is performed based on the above assumptions, evaluation determination is performed as follows, and an evaluation table 601 shown in FIG. 11A is generated.

  (1) The evaluation is calculated for each viewpoint and function according to the procedure shown in FIG. 11B.

  First, the evaluation points of the search item 602 shown in FIG. 11A selected as the inspection object are totaled for each viewpoint / function, and the defect ratio points related to the evaluation points (subtotals) thus tabulated are calculated. Multiply and obtain A point (subtotal / total) as shown in the inspection item total table 603 of FIG. 11B.

  Next, the evaluation points of the inspection items from which defects in the inspection item 602 are extracted are totaled for each viewpoint / function, and the evaluation points (subtotals) thus tabulated are multiplied by the related defect ratio points. Then, B points (subtotal / total) are obtained as shown in the defect count table 604 in FIG. 11B.

  Then, the ratio of B point (total) is obtained by using A point (total) as a parameter, and this ratio is compared with the defect ratio percentage that is internal data. As shown in the evaluation summary table 605 in FIG. The evaluation judgment is assigned to one viewpoint / one function.

  (2) When calculating evaluations by viewpoint, the same processing as that for calculating evaluations by viewpoint and function in (1) above is performed for all functions as shown in the evaluation summary table 606 by viewpoint in FIG. 11B. Do it.

  (3) When the evaluation is calculated for each function, it is calculated as follows according to the procedure shown in FIG. 11C.

  That is, the number of viewpoints of the functions selected as inspection targets is totaled for each function, and then the evaluation judgments distributed in the process (1) above are summed for each function / evaluation judgment, and are totaled as described above. Multiply the evaluation ratio (subtotal) by the related defect rate points to obtain C points (total). The result of this processing is shown in the tabulation table 607 by viewpoint in FIG. 11C.

  Furthermore, the defect percentage point (maximum value) is multiplied by the total number of viewpoints to obtain the Z point. The result of this processing is shown in the parameter point calculation table 608 in FIG. 11C.

  Then, the ratio of C points (total) is determined using the determined Z points as a parameter, and the calculation result is defined as a ratio X. The result of this processing is shown in the evaluation summary table 609 in FIG. 11C.

  As shown in the rank weighting condition table 610 in FIG. 11C, when the obtained ratio X is 0% or 50% or more, the rank weighting is not taken into consideration, but otherwise, the rank weighting table in FIG. 11C is used. As indicated by 611, the rank weight assigned to each function is added to the ratio X.

  The rank-weighted ratio X is compared with the defect ratio percentage that is internal data, and the evaluation determination of the corresponding range is distributed for each function. The result of this processing is shown in the evaluation summary table 612 in FIG. 11C.

  (4) In the process of calculating the comprehensive evaluation, the pessimistic value of the evaluation determination calculated in the function-based evaluation in (3) is used as shown in the comprehensive evaluation aggregation table 613 in FIG. 11C.

  Next, “Evaluation Method 2” will be described with reference to FIGS. 12A and 12B. In this “evaluation method 2”, based on the above “evaluation method 1”, an individual opinion is acquired by the following processing, and an evaluation table 620 in FIG. 12A is generated.

  (1) In the process of calculating the viewpoints according to the viewpoints, as shown in the viewpoint-specific viewpoint calculation processing procedure 621 in FIG.

  When there are six or more minimum evaluations, the minimum evaluation number is calculated for each function from the acquired viewpoint, and a larger number is adopted. Further, when the same number of processes is employed, the number of evaluation points is calculated and a larger number is employed.

  If the minimum evaluation is 4 or less, the acquired viewpoint is adopted, and the next lowest evaluation is acquired. This process is repeated until the total number of acquisitions reaches 5.

  Further, when the lowest evaluation is 0, a fixed character “no problem” is adopted, and the process of referring to the opinion information DB 107e in FIG. 6 in the next process is not performed.

  As the next process, based on the evaluation determination for each viewpoint acquired in the above process, the corresponding opinion content is acquired with reference to the opinion information DB 622 shown in FIG. 12B.

  (2) In the process of setting views by function, as shown in the function-by-function opinion calculation processing procedure 623 in FIG. 12B, first, the distribution is performed in the same process as the process for setting the viewpoints by viewpoint in (1) above. The five lowest evaluation top evaluations are acquired for each function, and then a fixed character is created based on the acquired function name.

  (3) In the process of calculating the opinion of the comprehensive evaluation, as shown in the comprehensive evaluation opinion calculation processing procedure 624 in FIG. 12B, the corresponding opinion with reference to the opinion information DB 625 in FIG. Get the contents.

  Next, with reference to FIG. 13, a specific example of the processing operation for automatically creating the report in step S <b> 212 in FIG. 7 by the report editing processing unit 104 c of the processing unit 104 in the client computer 100 in FIG. 1 will be described. .

  The information (number of remaining defects, evaluation judgment, opinion) calculated in the processing in FIG. 9, FIG. 10A to FIG. 10D, FIG. 11A to FIG. 11C, FIG. 12A, and FIG. Based on this, the contents are embedded in each area of the report format 700 generated in advance to create a report.

  That is, in the area A of the report format 700, the information (work area evaluation table) obtained in FIGS. 11A to 11C, 12A, and 12B is embedded (* the number of remaining defects is embedded from the information of FIG. 9). In area B (inspection viewpoint area), “selection NO1” in the selection information DB 107b is used as a key, and “NO1” in the viewpoint / item DB 107a is compared. If there is, perform UNIQ processing) and embed the result.

  Further, in area C (overall opinion area), the opinion content obtained in the comprehensive evaluation opinion calculation processing procedure 624 in FIG. 12B (“(3) Calculation of opinion of comprehensive evaluation”) is embedded, and area D (individual opinion area) Is embedded with the opinion content obtained in the function-based opinion calculation processing procedure 623 in FIG. 12B (“(2) Function-specific opinion setting”).

  In this process, report creation is optional, and whether or not the report is created is determined by the judgment of the user.

  As described above with reference to FIGS. 1 to 13, in this example, in the black box type program inspection performed without understanding the functional specifications, a unified inspection is realized regardless of the ability of the inspector. For this purpose, standard inspections are performed based on pre-registered general-purpose “inspection viewpoints” and information on “inspection items” necessary for actual item digestion. Information is registered in a storage device, and analysis / evaluation is automatically performed from an evaluation point set and registered in advance. At that time, even if the inspection is in progress, the number of remaining defects is predicted from the information registered so far. Then, a unified view of the evaluation results is derived and a report is automatically created.

  The program test system of this example is compatible with various interactive systems, and can be applied to program inspection of any system.

  In particular, in the system of this example, “viewpoints necessary for inspection” and “inspection items to be checked at the time of program inspection” accumulated in the past are registered in the storage device in advance, and necessary information (for evaluation) By selecting the desired viewpoint), the confirmation contents suitable for the system to be inspected can be automatically generated, and the program inspection can be realized if any system is interactive. Since each system can select how a third party wants to evaluate, several tens of evaluations can be realized depending on the selection method.

  Also, in this example, program inspection (actual machine verification) is performed based on inspection viewpoint / standard inspection items, and the result (OK / NG) is registered for each viewpoint and item, so that weighting points and program defects of the viewpoint are registered. A unique evaluation is automatically performed based on the importance of each item and the point according to function, and the number of defect extraction requests, the review viewpoint, etc. are judged, and a consistent quality evaluation can be performed.

  Moreover, in the program test system of this example, the progress from the viewpoint of inspection / standard inspection items to the present is calculated in the middle of the work, and the statistical method is used from the digestion status, the number of defective programs, and the risk rate. Residual defect prediction can be calculated, and timely quality assurance requirements can be realized.

  In addition, from the evaluation result, a report can be automatically created using a table and a graph with quantitative and qualitative information and respective views in a predetermined format.

  In this way, in this example, the viewpoint that the user (inspector) wants to evaluate based on the pre-registered general-purpose “inspection purpose (inspection viewpoint)” and “inspection item” information necessary for the actual inspection. A GUI screen for selecting (purpose) is generated and displayed, and an inspection item is generated by selecting a viewpoint (purpose) that the user (inspector) wants to evaluate via this GUI screen. Thereby, digestion (inspection), analysis, and evaluation of items in a short time become possible.

  For example, when “I want to determine whether the input check of the target system is valid”, the inspector simply selects the inspection item from the viewpoint necessary for “input check” on the GUI screen, and only the selected inspection item is selected. As a result, the time required for digestion from an unnecessary viewpoint can be reduced, and any personnel can digest the same test item. Note that not only a single evaluation but also a viewpoint evaluation of the entire system is possible.

  In addition, as the evaluation process, the evaluation can be automatically performed at any timing during the inspection and after the completion of the inspection, and a timely grasp of the situation can be realized. For automatic evaluation, based on the results of OK and NG inspections at the time of digestion of the inspection items, based on the internal evaluation points, the weighting point of the viewpoint, the importance of the program failure, and the function-specific points are taken into consideration. Ask. In the evaluation, which viewpoint and operation are weak are derived from the opinion table based on the evaluation points, and automatically summarized in a report as a summary. The opinion is also displayed in a chart or graph so that the strong and weak points can be seen at a glance.

  Thereby, in the program test system of this example, the evaluation viewpoint can be limited in advance, and inspection items can be automatically created and automatically evaluated using the limited items. Therefore, it is not necessary to narrow down to the minimum necessary, and it is not necessary to create inspection items, and the evaluation is automatically created, so the time required for preparation for program inspection can be reduced, and test inspection items can be determined and tested. The evaluation can be carried out in a unified manner, and it is possible to carry out an efficient and accurate program inspection, and the efficiency can be increased accordingly. Furthermore, since the evaluation contents can be visually seen and the remaining defects can be predicted, it can greatly contribute to future quality improvement.

  In addition, this invention is not limited to the example demonstrated using FIGS. 1-13, In the range which does not deviate from the summary, various changes are possible. For example, in this example, the processing program according to the present invention is placed in the client computer 100 and the data is placed in the server computer 105 and processed 1: 1 via the network. It is also possible to share information and process data and processing programs on one client computer 100.

  The computer configuration of this example may be a computer configuration without a keyboard or optical disk drive. In this example, an optical disk is used as a recording medium. However, an FD (Flexible Disk) or the like may be used as a recording medium. As for the program installation, the program may be downloaded and installed via a network via a communication device.

It is a block diagram which shows the structural example of the program test system which concerns on this invention. It is explanatory drawing which shows the structural example of viewpoint and item DB which the data storage part of the server computer in FIG. 1 manages. It is explanatory drawing which shows the structural example of selection information DB which the data storage part of the server computer in FIG. 1 manages. It is explanatory drawing which shows the structural example of object information DB which the data storage part of the server computer in FIG. 1 manages. It is explanatory drawing which shows the structural example of evaluation point DB which the data storage part of the server computer in FIG. 1 manages. It is explanatory drawing which shows the structural example of opinion information DB which the data storage part of the server computer in FIG. 1 manages. It is a flowchart which shows the process operation example which concerns on this invention of the program test system in FIG. It is explanatory drawing which shows the example of a form output screen in selection operation | movement of the evaluation viewpoint by the program test system in FIG. It is explanatory drawing which shows the key acquisition operation example in selection operation | movement of the evaluation viewpoint by the program test system in FIG. It is explanatory drawing which shows the viewpoint and item database search operation example in selection operation | movement of the evaluation viewpoint by the program test system in FIG. It is explanatory drawing which shows the selection information database registration operation example in the selection operation of the evaluation viewpoint by the program test system in FIG. It is explanatory drawing which shows the search operation example of the selection information database in the selection operation | movement of an evaluation viewpoint by the program test system in FIG. 1, and a viewpoint and item database. It is explanatory drawing which shows the example of a form output screen displayed and output by the selection operation | movement of the evaluation viewpoint by the program test system in FIG. It is explanatory drawing which shows the example of evaluation processing operation | movement by the program test system in FIG. It is explanatory drawing which shows the importance rank setting operation example before the test | inspection start in the evaluation process operation | movement of FIG. FIG. 10 is an explanatory diagram showing an example of setting an evaluation point in the evaluation processing operation of FIG. 9. FIG. 10 is an explanatory diagram illustrating an example of an evaluation determination operation after completion of an inspection in the evaluation processing operation of FIG. 9. FIG. 10 is an explanatory diagram illustrating an example of an overall evaluation determination operation after completion of an inspection in the evaluation processing operation of FIG. 9. It is explanatory drawing which shows the specific operation example of evaluation according to function in the whole evaluation determination operation | movement of FIG. 10D, evaluation according to viewpoint, and comprehensive evaluation. It is explanatory drawing which shows the operation example of evaluation according to viewpoint and function according to FIG. 11A, and evaluation according to viewpoint. It is explanatory drawing which shows the operation example of evaluation according to function in FIG. 11A, and comprehensive evaluation. It is explanatory drawing which shows the example of individual view process operation based on the operation result of evaluation according to function in FIG. 11A, evaluation according to viewpoint, and comprehensive evaluation. It is explanatory drawing which shows the specific example of the individual opinion process operation | movement in FIG. 12A. It is explanatory drawing which shows the output processing operation example of the evaluation result by the program test system in FIG.

Explanation of symbols

  100: client computer 101: input / output unit 102: form output unit (“form output”) 103: data control unit 104: processing unit 104a: data selection distribution processing unit (“data selection distribution processing”) ), 104b: Evaluation analysis 1 processing (remaining estimation) section (“Evaluation analysis 1 processing (remaining estimation)”), 104c: Report editing processing section (“report editing processing”), 104d: Information storage processing section (“ Information storage process "), 104e: evaluation analysis 2 process (result evaluation) section (" evaluation analysis 2 process (result evaluation) "), 104f: viewpoint item edit processing section (" viewpoint item editing process "), 104g: result registration Processing unit ("result registration process"), 104h: screen form display processing unit ("screen form display process"), 105: server computer, 106: data control unit, 107: data storage unit ("database") 107a: viewpoint / item DB, 107b: selection information DB, 107c: target information DB, 107d: evaluation point DB, 107e: opinion information DB, 300: evaluation viewpoint selection screen, 301 to 304: processing procedure, 305: inspection item 400: Information (information for predicting remaining defects) 401: Calculation procedure (binary probability calculation) 402: Calculation procedure (implementation area calculation) 403: Binary probability paper 404: Implementation area diagram 500 Importance rank information, 501: Evaluation point information, 502: Defect ratio point information, 503: Evaluation process procedure, 601: Evaluation table, 602: Inspection item, 603: Inspection item total table, 604: Number of defect total table, 605: Evaluation summary table, 606: Evaluation summary table by viewpoint, 607: Summary table by viewpoint, 608: Parameter point calculation table, 609: Evaluation summary table, 610: Rank weighting Case table, 611: Rank weighting table, 612: Evaluation tabulation table, 613: Comprehensive evaluation tabulation table, 620: Evaluation table, 621: Viewpoint calculation processing procedure by viewpoint, 622: View information DB, 623: View calculation processing procedure by function 624: Comprehensive evaluation opinion calculation processing procedure, 625: opinion information DB, 700: report format.

Claims (9)

A system that generates, by computer processing, list information (inspection item list information) of inspection items representing the contents of the inspection to be tested in the black box test of the program,
Each inspection item is divided into a plurality of groups according to inspection purposes (viewpoints), and the viewpoint / item information in which items representing the contents of each group's purpose (viewpoint) are hierarchized as a higher layer is stored in the storage device. First processing means;
A second processing means for generating and outputting a GUI screen that displays the purpose (viewpoint) of each inspection in a hierarchical manner;
The inspection item corresponding to the purpose (viewpoint) of each inspection selected on the GUI screen is read with reference to the viewpoint / item information, and the inspection item list information including the read inspection items is generated. 3. A program inspection item generation system characterized by comprising: A system for performing a black box test of a program by computer processing,
As means for executing the computer processing,
Fourth processing means for inputting inspection item list information generated by the program inspection item generation system according to claim 1;
Fifth processing means for inspecting the program based on the inspection items included in the inspection item list information;
A program test system comprising: sixth processing means for analyzing a test result of the fifth processing means and performing quality evaluation of the program. The program test system according to claim 2,
The sixth processing means includes
It is characterized in that weighting information for each inspection purpose and each inspection item stored in advance in a storage device is read out and referred to, and the weight is added to the inspection result of each inspection item to evaluate the quality of the program. Program test system. A program test system according to claim 2 or claim 3, wherein
As means for executing the computer processing,
A program test system comprising: seventh processing means for calculating a predicted value of the number of defects remaining in the program by a statistical method using inspection result information during the program inspection by the fifth processing means. . The program test system according to claim 4,
The intermediate test result information of the fifth processing means includes the test digestion number, the number of defective extractions, and the risk rate,
The seventh processing means calculates a predicted value of the number of defects remaining in the program based on a binomial probability using the number of inspection digests, the number of defective extractions, and the risk rate.   A program for causing a computer to function as each processing means in the program inspection item generation system according to claim 1.   The program for functioning a computer as each process means in the program test system in any one of Claims 2-5. A method for generating inspection item list information (inspection item list information) representing the content of an inspection to be tested in a black box test of a program by computer processing,
As a processing procedure of the above computer,
Each inspection item is divided into a plurality of groups according to the inspection purpose (viewpoint), and the viewpoint / item information is stored in the storage device in which the items representing the purpose (viewpoint) contents of each group are hierarchized as upper layers. The first procedure;
A second procedure for generating and outputting a GUI screen that displays the purpose (viewpoint) of each of the above inspections in a hierarchical manner;
Each inspection item corresponding to the purpose (viewpoint) of each inspection selected on the GUI screen is read with reference to the viewpoint / item information, and the inspection item list information including the read inspection items is generated. 3. A program inspection item generation method comprising the steps of: A method of performing a black box test of a program by computer processing,
As a processing procedure of the above computer,
A processing procedure by each processing means in the program inspection item generation system according to claim 1,
6. A program test method comprising: a processing procedure by each processing means in the program test system according to claim 2.

Images