JP2020115248A - Test priority determination apparatus - Google Patents

Abstract

To provide a test priority determination apparatus that determines priorities of test performed for programs.SOLUTION: A test priority determination apparatus 100 includes an analysis unit 31 that reads a program 8 and statically analyzes features of the program 8, a data extraction unit 32 that retrieves a quality DB 41 according to an analysis result of the analysis unit and extracts and acquires information of a defect corresponding to a feature of a part, a detection ratio of the defect, a test case for detecting the defect, and man-hour (man day) required for performing the test case, and a priority determination unit 33 that determines priority for performing the test target part and the test case based on the information extracted by the data extraction unit and important information 9 input via an information input unit 2.SELECTED DRAWING: Figure 1

Description

The present invention relates to a test priority order determination device that determines the priority order of tests performed on a program.

Since the scale of software tends to increase year by year, it is difficult to comprehensively execute a test for detecting all possible defects on a software program with a limited number of man-hours. Therefore, in order to prioritize important tests, a technique for prioritizing tests is known. For example, Patent Document 1 discloses a system that determines the priority of each test item related to a test for a modified version of arbitrary software.

JP, 2016-212633, A

However, the technique of Patent Document 1 can be applied only to a modified version of software. The present invention provides a test priority order determination device that determines the priority order of tests to be performed on a new program that is not a modified version.

One aspect of the present invention provides a test priority order determination device that determines the priority order of tests to be performed on a program. The test priority order determination device includes an analysis unit that reads a program and analyzes the characteristics of the program, and a priority order determination unit that determines the priority order of the test to be performed on the program according to the analysis result of the analysis unit.

According to the present invention, it is possible to obtain a test priority order determination device that determines the priority order of tests performed on a new program. As a result, it becomes possible to efficiently carry out the test with a limited man-hour.

It is a figure which shows the structure of the test priority determination apparatus which concerns on Embodiment 1 of this invention. 5 is a flowchart showing a flow of operations of the test priority order determination apparatus according to the first embodiment. FIG. 4 is a diagram showing an example of importance information according to the first embodiment. 5 is a diagram showing an example of a static analysis result by the control unit according to the first embodiment. FIG. It is a figure which shows an example of the quality DB of Embodiment 1. FIG. 5 is a diagram showing an example of information input from the data extraction unit of the first embodiment to a priority order determination unit. FIG. 6 is a diagram showing a priority order when the product of the importance and the detection rate is set as the priority in the first embodiment. FIG. 9 is a diagram showing an example of a dynamic analysis result by the control unit according to the first modification of the first embodiment. FIG. 8 is a diagram showing an example of a quality DB according to the first modification of the first embodiment. FIG. 7 is a diagram showing a configuration of a test priority order determination device according to a second modification of the first embodiment. FIG. 9 is a diagram showing a first quality DB according to the second modification of the first embodiment. FIG. 9 is a diagram showing a second quality DB according to the second modification of the first embodiment. It is a figure which shows the structure of the test priority determination apparatus which concerns on Embodiment 2 of this invention. It is a figure which shows the structure of the test priority determination apparatus which concerns on Embodiment 3 of this invention. It is a figure which shows the learning device of Embodiment 3. It is a figure which shows an example of the test result information of Embodiment 3. FIG. 16 is a diagram showing an example of priority information according to the third embodiment.

The test priority order determination device according to the embodiment of the present invention analyzes a program and determines the priority order of tests to be performed on the program. Hereinafter, a test priority order determination device according to an embodiment of the present invention will be described with reference to the drawings. In each embodiment, the same components are designated by the same reference numerals and the description thereof will be omitted.

(Embodiment 1)
FIG. 1 is a diagram showing the configuration of a test priority order determination device 100 according to the first embodiment of the present invention. The test priority order determination device 100 includes a program input unit 1, an information input unit 2, a control unit 3, a storage unit 4, and an output unit 5.

The program input unit 1 is an input interface circuit that reads the program 8 and inputs it to the control unit 3. The program 8 is a text file such as source code that can be executed as a program. The information input unit 2 is an input interface circuit that reads information or data such as importance information 9 described later and inputs the information or data to the control unit 3. The program input unit 1 and the information input unit 2 are configured by separate input interface circuits as shown in FIG. 1, for example, but not limited to this, and may be configured by the same input interface circuit.

The control unit 3 is a processing device that performs various calculations and controls the overall operation of the test priority order determination device 100. The control unit 3 includes a general-purpose processor such as a CPU or MPU that realizes a predetermined function by executing a program. The control unit 3 is not limited to one that realizes a predetermined function by cooperation of hardware and software, and is a hardware circuit such as FPGA, ASIC, DSP, etc. designed as a dedicated circuit for realizing the predetermined function. It may be configured.

The storage unit 4 is a recording medium that records various kinds of information including programs and data necessary to realize the functions of the test priority order determination device 100. A quality database (DB) 41 described later is stored in the storage unit 4, for example. The storage unit 4 is realized by, for example, a semiconductor memory device such as a flash memory or an SSD, a magnetic storage device such as a hard disk, or another storage device alone or in combination thereof. The storage unit 4 may include a volatile memory such as SRAM or DRAM capable of high-speed operation that temporarily stores various information.

The output unit 5 is a device or an interface circuit that outputs various information. The output unit 5 includes a display device such as a liquid crystal display, an organic EL display, and a projector. The output unit 5 also includes a printer and a speaker that outputs sound. The output unit 5 is an interface circuit for enabling communication connection to an external device via a network in accordance with standards such as IEEE802.3, IEEE802.11 or Wi-Fi, LTE, 3G, 4G and 5G. Good.

FIG. 2 is a flowchart showing the flow of the operation of the test priority order determination device 100. First, the control unit 3 inputs the program 8 via the program input unit 1 and the importance degree information 9 via the information input unit 2 (S1).

FIG. 3 is a diagram showing an example of the importance level information 9. The degree-of-importance information 9 is information in which the degree-of-importance M of the portion is associated with the portion such as the first portion, the second portion, and the third portion of the program 8. The degree-of-importance information 9 is formed, for example, by a program developer, a test executor, or a project manager (hereinafter referred to as “program developer or the like”) assigning the degree of importance M to each part of the program 8. .. The importance M is, for example, a value of 0 or more and 1 or less. For example, the higher the importance of a part is, the greater the importance M is given.

Returning to FIG. 2, after step S1, the control unit 3 functioning as the analysis unit 31 executes static analysis on the program 8 (S2). Static analysis is a technique for analyzing a program without executing the program. Static analysis can detect defects such as typographical errors in source code and syntax errors. In addition, the static analysis can detect the number of lines of the source code and the functions included in the source code, the number of conditional branches, the number of functions to be called, and the like.

FIG. 4 is a diagram showing an example of a static analysis result by the control unit 3. According to the static analysis result of FIG. 4, the first part of the program 8 has a large function scale and a large complexity, and the second part has a large number of judgment condition steps.

For example, when the number of lines in a certain part of the source code is equal to or larger than the threshold value, the control unit 3 determines that the function scale of the part is large, and otherwise determines that the function scale is small. Further, for example, when the number of functions and modules called by a certain part of the program is equal to or larger than the threshold value, the control unit 3 determines that the complexity of the part is large, and otherwise the complexity is small. to decide. Further, if the number of judgment condition steps by the if statement or the like in a specific part of the source code is equal to or more than the threshold value, the control unit 3 determines that the number of judgment condition steps of the part is large. Judge that there are few. The static analysis result is output from the analysis unit 31 and input to the data extraction unit 32 (see FIG. 1).

Returning to FIG. 2, after step S2, the control unit 3 functioning as the data extraction unit 32 searches the quality DB 41 using the feature of each part as the static analysis result as shown in FIG. Information corresponding to the defect, the detection rate of the defect, the test case for detecting the defect, and the number of man-hours (man-day) required to perform the test are extracted and acquired from the quality DB 41 (S3). ..

FIG. 5 is a diagram showing an example of the quality DB 41. In the quality DB 41, information on the features of the program, the defects corresponding to the features, the detection rate of the defects, the test case for detecting the defects, and the number of man-hours (man-day) required to perform the test, etc. Corresponding information and are stored.

The quality DB 41 is a compilation of results of tests executed on programs created in past projects, for example. For example, the first line of the table in FIG. 5 shows that a program with a large function scale created in a past project is apt to cause defects in code clones, and the detection rate is 0.3 cases/kL. It indicates that there was. The first row of the table in FIG. 5 also shows that the test case used to detect the defect was T1, and the man-hour required for the test was 3 man-days. The detection rate and the required man-hours stored in the quality DB 41 are, for example, the maximum value or the average value of the past test results.

The information stored in the quality DB 41 is not limited to the information acquired in past development. For example, the correspondence information stored in the quality DB 41 may include information detected by performing a test on the program 8. As a result, it is possible to know what kind of defect the program 8 is likely to include, and thus it is possible to more efficiently determine the priority order. For example, by regularly (for example, every morning) reflecting the test results in the quality DB 41, it is possible to periodically change the priority order of the tests and perform the tests in accordance with the time.

Further, for example, the correspondence information stored in the quality DB 41 may be created by a program developer or the like based on theory, empirical rules, or the like.

Referring to FIG. 1, the data extraction unit 32 outputs the information extracted in step S3 to the priority order determination unit 33. The information input from the data extraction unit 32 to the priority order determination unit 33 is a combination of the static analysis result shown in FIG. 4 and the corresponding information in the quality DB 41 of FIG. 5 as shown in FIG. Information.

In addition, importance level information 9 (see FIGS. 1 and 3) is also input to the priority order determination unit 33 via the information input unit 2.

Returning to FIG. 2, the priority order determination unit 33 determines the priority order of the test based on the input information (S4). Specifically, the priority order determination unit 33 determines the priority order to be executed for the test target part and the test case.

For example, the priority order determination unit 33 gives a priority to a test target part and a test case, and gives a higher priority order to a higher priority item. The priority is, for example, the importance M (see FIG. 3), the detection rate (see FIG. 6), the product of the importance M and the detection rate, or a combination thereof. Alternatively, the priority order determination unit 33 may increase the priority of a test that requires less man-hours.

FIG. 7 is a diagram showing priorities when the product of the importance M and the detection rate is the priority. FIG. 7 shows that the priority of the test (for example, the test by the test case T4) for the judgment combination error of the second part having the highest priority is first.

The control unit 3 may output the determined priority order via the output unit 5. For example, the control unit 3 displays the determined priority order on the liquid crystal display and informs the user of the priority order.

As described above, the test priority order determination apparatus 100 according to the present embodiment reads the program 8, analyzes the characteristics of the program 8, and analyzes the characteristics of the program 8. A priority order determination unit 33 that determines the priority order of the test to be performed.

As a result, the priority order of the tests can be determined according to the characteristics of the program 8, and the tests can be efficiently performed.

The test priority order determination device 100 may further include a quality database 41 in which the characteristics of the program and the quality information regarding the quality of the program corresponding thereto are associated with each other. In this case, the priority order determination unit 33 extracts the quality information of the program corresponding to the analysis result from the quality database 41. In this case, the process in which the priority order determination unit 33 determines the priority order of the test performed on the program according to the analysis result is the process of determining the test priority order based on the quality information of the program corresponding to the analysis result. It may be.

As a result, corresponding quality information can be extracted from the quality database 41 according to the characteristics of the program 8, and the priority order of the test can be determined based on the quality information, and the testing according to the characteristics of the program 8 can be performed. Will be possible. Therefore, the test can be performed more efficiently.

The quality information may include a defect detection rate for a program having a corresponding characteristic. The quality information may also include the results of tests performed on programs developed in the past.

As a result, it is possible to reflect the detection rate and the result of the test executed for the program developed in the past, and determine the priority order of the test according to the characteristics of the program 8. Therefore, the test can be performed more efficiently.

(Modification 1)
Hereinafter, the first modification of the first embodiment will be described. In the above-described first embodiment, the control unit 3 (see FIG. 1) functioning as the analysis unit 31 executes static analysis on the program 8. However, analysis of the program 8 is not limited to static analysis. For example, the control unit 3 functioning as the analysis unit 31 may perform dynamic analysis on the program 8.

Dynamic analysis is a technique of executing a program and analyzing the program. In dynamic analysis, in addition to the structure obtained by static analysis such as complexity and scale per program unit, it is also necessary to analyze the relationship between program parts and the behavior of data used in the program. You can With dynamic analysis, it is possible to detect the risk of defects that cannot be recognized until the program is run.

FIG. 8 is a diagram showing an example of a dynamic analysis result by the control unit 3 according to the first modification of the first embodiment. According to the dynamic analysis result of FIG. 8, the first part of the program 8 has a feature that a memory leak occurs and deviates from the parameter range, and the second part has a feature that a source that does not reach is included. ..

In the first modification of the first embodiment, the correspondence information in the quality DB 41 also corresponds to the characteristics of the program that can be analyzed by the dynamic analysis. FIG. 9 is a diagram showing an example of the quality DB 41 according to the first modification of the first embodiment. The first line of the table in FIG. 9 indicates that a program that causes a memory leak created in a past project is apt to have a defect that the control structure is erroneously, and the detection rate is 0.3/ It has shown that it was kL. The first row of the table in FIG. 9 also shows that the test case used to detect the defect was T7, and the man-hour required for the test was 3 man-days.

The data extraction unit 32 outputs the information extracted from the quality DB 41 to the priority order determination unit 33. The information input from the data extraction unit 32 to the priority order determination unit 33 is information that combines the dynamic analysis result and the corresponding information in the quality DB 41.

In addition, importance level information 9 (see FIGS. 1 and 3) is also input to the priority order determination unit 33 via the information input unit 2.

The priority order determination unit 33 determines the priority order of the test based on the input information. Specifically, the priority order determination unit 33 determines the priority order to be executed for the test target part and the test case.

As described above, in the first modification of the first embodiment, the dynamic analysis is executed, and the priority order of the tests is determined according to the characteristics obtained by the dynamic analysis. As a result, it is possible to determine the priority order of the tests according to the characteristics of the program 8. Therefore, the test can be performed more efficiently.

The first embodiment and the first modification of the first embodiment may be combined. That is, the analysis unit 31 can execute at least one of static analysis and dynamic analysis, and may execute both, for example.

(Modification 2)
FIG. 10 is a diagram showing the configuration of the test priority order determination device 120 according to the second modification of the first embodiment. A first quality DB 41a and a second quality DB 41b are stored in the storage unit 4 of the test priority order determination device 120. Unlike the first embodiment, the first quality DB 41a and the second quality DB 41b may be switched and used according to the field of the project, the degree of difficulty, the resource, and the like.

FIG. 11 is a diagram showing the first quality DB 41a. FIG. 12 is a diagram showing the second quality DB 41b. The first quality DB 41a is a compilation of the results of tests executed on programs created in projects in one field in the past. The second quality DB 41b is a compilation of the results of tests executed on programs created in projects in other fields in the past.

For example, in the test priority order determination device 120, when the current project is similar to the project of the one field in the field, the first quality DB 41a is used. On the other hand, when the current project is similar to the project of the other field in the field, the second quality DB 41b is used.

As described above, in the second modification of the first embodiment, by using the data of the project similar to the current project, the judgment material suitable for the project is adopted, and the priority order can be more efficiently determined.

(Embodiment 2)
In a project involving program development, not all program tests are feasible. For example, in the execution of a test, resources are not unlimited but there are certain restrictions. The resource constraint means that, for example, there is an upper limit on the number of workers, the area of the work place, the time when the work place can be used, the cost, the resource, and the like. In the second embodiment of the present invention, there is provided a test priority order determination device which determines a test priority order in consideration of these constraints.

FIG. 13 is a diagram showing the configuration of the test priority order determination device 200 according to the second embodiment. Compared to the first embodiment, the constraint information 210 is input to the information input unit 202 of the test priority order determination device 200 in addition to the importance information 9.

The constraint information 210 is condition information indicating conditions for program development, and includes, for example, the number of workers or man-hours available, the number of test devices, the area of the work place, and the time when the work place can be used. The constraint information 210 may include information on a site to be tested with priority.

The constraint information 210 is input to the priority order determination unit 233 of the control unit 3 via the information input unit 202 together with the importance degree information 9. The priority order determination unit 233 determines the priority order of the test based on the input information.

For example, there are two tests A and B having the same importance M and the detection rate, the number of workers required to perform one test A is N1, and the number of workers required to perform the other test B is It is assumed that the number is N2 (>N1). The number of workers who can work in the project is N3, and the relationship of N1<N3<N2 is established. In this case, test B cannot be performed in the project because the number of workers is insufficient. In such a case, the priority order determination unit 233 lowers the priority order of the test B. Alternatively, the priority of test A is increased.

As described above, the test priority order determination apparatus 200 according to the present embodiment determines the test priority order based on the constraint information 210. As a result, it is possible to effectively determine the priority order according to the resources, and it is possible to efficiently determine the priority order.

(Embodiment 3)
FIG. 14 is a diagram showing the configuration of the test priority order determination device 300 according to the third embodiment of the present invention. In the third embodiment, the storage unit 4 stores a learning model 342 constructed by learning past test result data. In the third embodiment, the priority order determination unit 333 applies the input information to the learning model 342 and determines the priority order based on the result obtained thereby.

The learning model 342 is generated by the learning device 350. FIG. 15 is a diagram showing a learning device 350 according to the present embodiment. The learning device 350 performs machine learning based on the test result information 360 that stores the results of the tests executed on the programs created in the past projects.

The learning device 350 includes a learning unit 351 and a storage unit 352. The learning unit 351 is a processing device similar to the control unit 3, and includes a CPU that realizes a learning function by executing a program, and a general-purpose processor such as an MPU. The storage unit 352 is a recording medium and stores, for example, the learning model 342 generated by the learning unit 351. The storage unit 352 is realized by, for example, a flash memory, a semiconductor memory device such as an SSD, a magnetic storage device such as a hard disk, or another storage device alone or in combination thereof. The storage unit 352 may include a volatile memory such as SRAM or DRAM capable of operating at high speed for temporarily storing various information.

The learning unit 351 performs machine learning based on the test result information 360 and the priority information 370 given to each test result by the program developer or the like. FIG. 16 is a diagram showing an example of the test result information 360.

As shown in FIG. 16, the test result information 360 includes information such as P1, P2, and P3 for identifying programs created in past projects. The test result information 360 includes information of “importance”, “feature”, “test case”, “defect”, “detection rate”, and “required man-hour” for each part of each program.

The “importance” column of the test result information 360 shows the importance of the relevant part given by the program developer or the like in the past project. The higher the importance of the part, the higher the value of importance.

The “feature” column of the test result information 360 shows the result of executing a program analysis such as static analysis or dynamic analysis in a past project.

The “test case” column of the test result information 360 shows the test case used for the test of the relevant part in the past project. The "defect" column of the test result information 360 shows the defect detected by the test of the relevant part in the past project, and the "detection rate" column shows the detection rate of the defect.

The “man-hours” column of the test result information 360 shows the man-hours required for testing the relevant part in the past project.

FIG. 17 is a diagram showing an example of the priority information 370. Each part of each program is given a priority. The priority is, for example, a value of 0 or more and 1 or less. The priority is input by the program developer or the like of the current project in consideration of the test result information 360, for example. Alternatively, the priority may be given by a program developer of each project in the past. Further, the priority may be included in the test result information 360.

The learning unit 351 constructs a learning model 342 by performing supervised learning using the priority information 370 as teacher data based on each information in the test result information 360. The learning model 342 may be any learning model as long as it can predict the priority from the characteristics and importance of the program. For example, the learning model 342 is a learning model generated by performing learning using a neural network, decision tree learning, support vector machine, or the like.

Returning to FIG. 14, the learning model 342 generated by the learning device 350 as described above is stored in the storage unit 4 of the test priority order determination device 300. The control unit 3 functioning as the priority order determination unit 333 sets the learning model 342 based on the importance information 9 input via the information input unit 2 and the analysis result of the program 8 input from the analysis unit 31. Determine the priority used. For example, when a neural network is used as the learning model 342, the control unit 3 inputs the importance degree information 9 and the analysis result of the program 8 into the neural network, and thereby acquires the priority output from the neural network.

When the control unit 3 functioning as the priority order determination unit 333 acquires the test priorities for the respective parts from the learning model 342, the control unit 3 gives higher priority orders in descending order of priority. The priority order thus determined is transmitted to the output unit 5.

As described above, the test priority order determination apparatus 300 according to the present embodiment associates the characteristics of the programs developed in the past with the results of the tests executed on the programs developed in the past. The storage unit 4 further stores a learning model 342 constructed by supervised learning in which the priority information 370 attached to a program developed in the past is used as teacher data based on the test result information 360. In this case, the process of determining the priority order of the test performed on the program 8 by the priority order determining unit 333 according to the analysis result is a process of using the learning model 342 to determine the priority order from the analysis result.

With this configuration, the priority of the test can be determined according to the characteristics of the program 8, and the test can be efficiently performed.

(Modification)
In the third embodiment, the example in which the learning model 342 generated by the learning device 350 is used by the test priority order determination device 300 which is a device different from the learning device 350 has been described. However, the present invention is not limited to this, and for example, the control unit 3 of the test priority order determination apparatus 300 has a learning function, and based on each information in the test result information 360 input to the test priority order determination apparatus 300. The learning model 342 may be constructed by performing supervised learning using the priority information 370 as teacher data.

That is, the test priority order determination apparatus according to the modification of the present embodiment learns by supervised learning using the priority information 370 attached to the program developed in the past as teacher data based on the test result information 360. The learning unit 351 that constructs the model 342 is further included. In this example, the test priority order determination device 300 includes a function of performing a machine learning to construct a learning model and a function of using the constructed learning model.

1 program input unit, 2 information input unit, 3 control unit, 4 storage unit, 5 output unit, 8 program, 9 importance information, 31 analysis unit, 32 data extraction unit, 33 priority order determination unit, 41 quality database, 100 Test priority determination device, 342 learning model, 350 learning device, 351 learning unit, 360 test result information, 370 priority information.

Claims (10)

A test priority determination device for determining the priority of tests to be performed on a program,
An analysis unit that reads the program and analyzes the characteristics of the program,
A priority order determination unit that determines the priority order of the tests to be performed on the program according to the analysis result of the analysis unit,
Test priority determination device. Further comprising a quality database in which the characteristics of the program and the quality information regarding the quality of the program corresponding thereto are associated with each other,
The priority order determination unit extracts quality information of the program corresponding to the analysis result from the quality database,
The process of determining the priority of the test performed on the program by the priority determining unit according to the analysis result is a process of determining the test priority based on the quality information of the program corresponding to the analysis result. Is
The test priority order determination device according to claim 1. The test priority order determination device according to claim 2, wherein the quality information includes a defect detection rate of a program having the corresponding characteristic. The test priority order determination device according to claim 2, wherein the quality information includes a result of a test executed for a program developed in the past. The priority order determination unit, when performing the process of determining the priority order of the test, further determines the priority order of the test based on the constraint information indicating the development condition of the program,
The test priority order determination device according to claim 1. The constraint information includes at least one information of the number of workers, the number of man-hours, the number of test devices, the area of a work place, and the time when the work place can be used in the development of the program. 5. The test priority order determination device according to item 5. Based on the test result information in which the characteristics of the program developed in the past and the result of the test executed on the program developed in the past are associated with each other, the program is attached to the program developed in the past. Further comprising a storage unit that stores a learning model constructed by supervised learning using priority information as teacher data,
The process of determining the priority of the test performed on the program by the priority determining unit according to the analysis result is a process of determining the priority from the analysis result using the learning model.
The test priority order determination device according to claim 1. The test according to claim 7, further comprising: a learning unit that builds the learning model by supervised learning using priority information attached to the program developed in the past as teacher data based on the test result information. Priority determination device. Further comprising an input unit for inputting importance information indicating the importance attached to the test,
The priority order determination unit, when performing the process of determining the priority order of the test, further determines the priority order of the test, based on the importance degree information indicating the importance degree given to the test,
The test priority order determination device according to claim 1. The test priority order determination apparatus according to claim 1, wherein the analysis executed by the analysis unit includes at least one of static analysis and dynamic analysis.

Images