JP7296501B1 - Bug Detection Rate Threshold Updating System, Bug Detection Rate Threshold Updating Method, and Bug Detection Rate Threshold Updating Program - Google Patents

Abstract

A bug detection rate threshold updating system, method, and program are provided for appropriately updating the threshold of the detection rate of bugs in software testing.
In a bug detection rate threshold update system, a center server acquires a first detection rate of a bug of a first test object for which a test result was obtained during a first period in the past, A first candidate for the bug detection rate threshold is determined based on the rate, and if the difference between the determined first candidate and the threshold is less than a predetermined value, the threshold is updated with the first candidate. Also, if the threshold is not updated with the first candidate, the second detection rate of bugs for the second test subject for which test results were obtained during a past second period including days after the first period is A second candidate for the threshold is determined based on the obtained second detection rate. Furthermore, if the difference between the second candidate and the threshold is greater than or equal to the predetermined value, the threshold is updated with the second candidate.
[Selection drawing] Fig. 1

Description

The present invention relates to a method for making testing-related decisions using the detection rate of bugs detected in software testing.

Conventionally, it is known to test software in order to ensure the quality of developed software. For example, a test engineer or the like prepares in advance test items or test cases indicating test contents, procedures, and the like. A tester executes tests according to the created test items or test cases. If testing reveals a bug in the software, the developer modifies the software to remove the bug. Also, a so-called additional test may be performed as a software test. Additional tests are additional tests that are performed when deemed necessary, for example, based on the results of scheduled tests. Additional testing may be performed, for example, if the number of bugs detected in testing is relatively high and it is determined that the quality of the software needs to be boosted. As information indicating the number of bugs, there is a bug detection rate. A bug detection rate may be calculated, for example, by dividing the number of bugs detected in a test by the number of tests performed. It is also known to compare this bug detection rate to a preset threshold to determine the need for any action in relation to the tested software.

For example, Patent Literature 1 discloses a software quality determination device for determining convergence of bugs occurring in a system. In the database of this determination device, for each test viewpoint, a specific value set by the user is held as a specific value that serves as a reference for the detection rate of bugs occurring in the test from that test viewpoint. After the amount of testing from a certain testing viewpoint reaches the standard amount of testing, additional tests are repeatedly performed for the standard amount of additional testing until the bug detection rate becomes equal to or less than a specific value. Each time the test is completed, the determination device determines whether or not the bugs in the test from that viewpoint have converged according to whether or not the bug detection rate at that time is equal to or less than a specific value.

JP 2019-101581 A

Just because the bug detection rate exceeds the threshold does not necessarily mean that additional testing should be performed. However, the fact that the bug detection rate is equal to or higher than the threshold value may serve as one of the criteria for conducting additional tests, or may trigger a review of the software development system, development environment, or the like. In order to make such judgments accurately, it is considered desirable to set the threshold appropriately.

The present invention has been made in view of the above points, and an example of the problem thereof is a bug detection rate threshold updating system, a bug detection rate threshold updating method, and a bug detection rate threshold updating system capable of appropriately updating the threshold of the bug detection rate in software testing. and to provide a bug detection rate threshold update program.

In order to solve the above-mentioned problems, the invention according to claim 1 provides acquisition means for acquiring a first detection rate of bugs in a first test target for which test results were obtained during a past first period; determining means for determining a first candidate for a bug detection rate threshold based on the acquired first detection rate; updating means for updating the threshold value with the first candidate, wherein the obtaining means includes days later than the first time period after the threshold value has not been updated with the first candidate by the updating means. Obtaining a second detection rate of bugs in a second test target for which test results were obtained during a past second period of time, the determining means, based on the obtained second detection rate, Two candidates are determined, and if the difference between the second candidate and the threshold is equal to or greater than the predetermined value, the updating means updates the threshold with the second candidate.

According to this invention, after the threshold is not updated with the first candidate because the difference between the first candidate for the threshold and the threshold is equal to or greater than the predetermined value, the second candidate for the threshold is similarly applied at the next timing. and the threshold is greater than or equal to a predetermined value, the threshold is updated with the second candidate. If there is a difference between the threshold and the first candidate for the threshold determined based on the first detection rate of the bug of the first test target for which the test result was obtained during the first period, for that period: There may have been some changes in the software development system or development environment. The threshold is updated to reflect this change. On the other hand, the altered state is temporary and may soon return to its original state. Alternatively, there may have been a difference between the first candidate for the threshold and the threshold by chance, rather than a change in the development system or development environment. Therefore, considering the influence of a large change in the threshold, the threshold is not updated immediately when the difference between the first candidate threshold and the threshold is equal to or greater than a predetermined value. After that, as a result of comparison between the threshold and the second candidate for the threshold determined based on the second detection rate of bugs of the second test target for which the test result was obtained during the new second period, the difference is also large. , the changed state may continue to some extent. The threshold is then updated with the second candidate. Therefore, it is possible to appropriately update the threshold for the detection rate of bugs in software testing.

In the invention according to claim 2, each of the first test object and the second test object is a test object for which it is determined whether or not to perform an additional test based on the results of a scheduled test, It is characterized by being a test subject for which it has been decided to perform the additional test.

According to the present invention, threshold candidates are determined using the bug detection rate for a test target on which an additional test has been performed. Accordingly, the updated thresholds can be used to better determine the likelihood of additional testing for a given test object.

The invention according to claim 3 is characterized in that the acquisition means acquires detection rates of bugs detected in the scheduled test as the first detection rate and the second detection rate, respectively. .

According to the present invention, threshold candidates are determined using the detection rate of bugs detected in scheduled tests for test targets on which additional tests have been performed. By using the bug detection rate, which may affect the decision of whether or not to perform additional testing, it is possible to appropriately determine threshold candidates.

According to a fourth aspect of the present invention, the update means determines that the difference between the second candidate and the threshold is equal to or greater than the predetermined value and the difference between the first candidate and the threshold is equal to or greater than the predetermined value. In some cases, when the magnitude relationship between the second candidate and the threshold matches the magnitude relationship between the first candidate and the threshold, the threshold is updated with the second candidate. It is characterized by

According to this invention, when the magnitude relationship between the second threshold candidate and the threshold matches the magnitude relationship between the first threshold candidate and the threshold, the threshold is updated with the second candidate. . Therefore, the possibility that the changed state continues can be appropriately reflected in the threshold value.

The invention according to claim 5 is characterized in that the determining means calculates a representative value of the bug detection rate for each of the first candidate and the second candidate.

The invention according to claim 6 is characterized in that the representative value is either one of an average value and a median value.

In the invention according to claim 7, when the detection rate of bugs detected in the test of a given test target under test is equal to or higher than the threshold, an additional test can be performed on the given test target. outputting means for outputting possibility information indicating that the likelihood of the error is higher than the case where the detection rate of the bug is less than the threshold.

According to this invention, by outputting the possibility information, it is possible to give the developer an opportunity to review the development system and development environment in order to avoid the execution of additional tests.

According to an eighth aspect of the present invention, there is provided a computer-implemented bug detection rate threshold updating method, in which a first detection rate of a bug of a first test object for which a test result was obtained during a first period in the past is obtained. a first acquisition step; a first determination step of determining a first candidate for a bug detection rate threshold based on the acquired first detection rate; and a difference between the determined first candidate and the threshold. is less than a predetermined value, a first updating step of updating the threshold with the first candidate, and after the threshold is not updated with the first candidate by the updating step, after the first period of time a second acquiring step of acquiring a second detection rate of bugs of a second test target for which test results were obtained during a past second period including days; and based on the acquired second detection rate, the a second determining step of determining a second candidate for a threshold; and a second updating step of updating the threshold with the second candidate if a difference between the second candidate and the threshold is equal to or greater than the predetermined value. characterized by comprising

According to a ninth aspect of the present invention, a computer comprises: acquisition means for acquiring a first bug detection rate of a first test target for which a test result was obtained during a past first period; determining means for determining a first candidate for a bug detection rate threshold based on the detection rate; and determining the threshold if the difference between the determined first candidate and the threshold is less than a predetermined value. updating means for updating with a candidate, and the acquiring means retrieves a past second period including days later than the first period after the threshold is not updated with the first candidate by the updating means. Obtaining a second detection rate of the bug of the second test target for which the test result was obtained during the test, the determining means determines a second candidate for the threshold based on the obtained second detection rate. The updating means updates the threshold with the second candidate when the difference between the second candidate and the threshold is equal to or greater than the predetermined value.

According to the present invention, it is possible to appropriately update the threshold of the detection rate of bugs in software testing.

1 is a diagram showing an example of a schematic configuration of a test management system S according to an embodiment; FIG. It is a block diagram showing an example of a schematic structure of center server 1 concerning one embodiment. FIG. 4 is a diagram showing an example of information stored in a database; FIG. 3 is a diagram showing an example of functional blocks of a system control unit 11 in the center server 1 according to one embodiment; FIG. It is a figure which shows the update example of a threshold value. It is a figure which shows the update example of a threshold value. 4 is a graph showing an example of bug detection rate distribution; 4 is a flowchart showing an example of threshold update control processing by the system control unit 11 of the center server 1 according to one embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In an embodiment, testing of developed software is performed on a test target basis. A test target may be one or more specific pieces of software. For example, a test object may be software developed in a particular software development project. Hereinafter, software development projects are simply referred to as "projects". Alternatively, the test target may be a function, service, or the like implemented by the software being developed. Alternatively, the test target may be a module, function, object, routine, script, library, or the like. Testing on a test target basis may mean that a plurality of tests are performed on target software and the results of those tests are aggregated.

Software testing may include scheduled testing and additional testing. A scheduled test may be a test that was originally built into the software's development schedule. Hereinafter, the scheduled test will be referred to as the "main test". Additional tests may be tests that were not originally included in the development schedule. Further, the additional test may be a test for which a participant in software development (for example, a person in charge, etc.) decides whether or not to implement the test based on the results of the main test. Completion of the software may be delayed beyond schedule if additional testing is performed. The additional test may be a re-execution of the test in which the bug was detected among the tests executed in the main test. That is, the additional test may be a retest. Alternatively, the additional test may be a test performed on a different point of view, item or case from the point of view, item or case of the main test. Further additional tests may be performed as a result of performing the additional tests.

One factor that can affect the likelihood or need for additional testing is the bug detection rate. A bug detection rate may be calculated by dividing the number of bugs detected in a test by the number of tests performed. The number of tests may be, for example, the number of test items or the number of test cases. One test case may check one or more test items. A bug detection rate may be calculated for each test object. For example, a bug detection rate may be calculated for each project. In general, the higher the bug detection rate, the more likely additional testing may be performed.

In the following, a case will be described where the present invention is applied to a system for managing testing of software developed in each of a plurality of projects as test objects.

[1. Configuration of test management system]
First, the configuration and functional overview of the test management system S according to this embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an example of a schematic configuration of a test management system S according to this embodiment.

As shown in FIG. 1, the test management system S includes a center server 1 and multiple development terminals 2 . The center server 1 and each development terminal 2 are connected to each other via a network NW. Examples of the network NW include the Internet, LAN (Local Area Network), VPN (Virtual Private Network), and the like.

The center server 1 is a server device for managing software testing. The center server 1 may actually be a server system composed of a plurality of server devices. The center server 1 may store, for example, information indicating the test implementation status and test results for each project. The result of the test may be the final result after all the tests in the main test or the additional test are completed. The execution status of the test may include an interim result of the test. The center server 1 may also manage the progress of each project. For example, the center server 1 may manage which stage each project is currently in. The center server 1 may update the test implementation status, test results, and project progress status based on information input to the development terminal 2 by project participants. The center server 1 transmits information related to tests stored in the center server 1 in response to a request from the development terminal 2 .

The center server 1 calculates the bug detection rate as one of the interim results or final results of the test. Then, based on the bug detection rate in the main test, the center server 1 may provide the development terminal 2 with risk information indicating the possibility of conducting an additional test for the project associated with the bug detection rate. . Risk information can be said to be information indicating the risk of the project. Specifically, when the bug detection rate is less than a preset threshold, the center server 1 may provide risk information indicating that the possibility of additional testing is relatively low. On the other hand, when the bug detection rate is equal to or higher than the threshold, the center server 1 may provide risk information indicating that the possibility of additional testing is relatively high. If risk information is provided as an interim result of this test, the person in charge of the project may refer to the risk information and review the software development system and development environment. Examples of reviewing the development system may include reviewing members, groups, or companies participating in the project, guidance or education for project participants, review of rules applied to software development, and the like. A review of the development environment may include a review of the devices and systems used for software development, a review of the place where the development takes place, and the like. Through these revisions, the project aims to maintain the quality of the software being developed above a certain level without additional testing. In general, when the bug detection rate is equal to or higher than the threshold, the need for the aforementioned review is clearly higher than when the bug detection rate is below the threshold.

When the main test is completed or is nearing completion, the project manager will decide whether or not to conduct additional tests. For example, the person in charge, etc. should consider the bug detection rate, the content of the bug, the possibility that undetected bugs similar to the detected bugs remain, and the undetected bugs remaining in the It may be determined whether or not to conduct additional tests based on the possibility of Just because the bug detection rate is above the threshold does not necessarily mean that additional testing will be performed, and just because the bug detection rate is below the threshold does not mean that additional testing will not be performed. However, when the bug detection rate is above the threshold, additional testing is more likely than when the bug detection rate is below the threshold.

The center server 1 may automatically set and update the above-described bug detection rate threshold. For example, the center server 1 may update the threshold for each category into which the project is classified or each category into which the software developed in the project is classified. Categories may be defined, for example, such that projects or software that are estimated to have common or similar bug tendencies are classified into the same category. Examples of bug trends include trends in the number of bugs (or bug detection rates) and trends in bug content. For example, if the software developed in each project is software related to the same service, those projects may be classified into the same category. For example, assume that a software development company using the test management system S develops software for various services provided through the Internet. Examples of services provided through the Internet include online shopping, flea markets, reservations for lodging facilities, reservations for beauty facilities, reservations for sports facilities, acceptance of home delivery orders, ticket sales, sales of voting tickets for public competitions, online Examples include search, video distribution, and music distribution. These services may be provided by website server software, or may be provided by application software that operates on terminal devices used by service users. For each service, multiple projects are in progress. In such cases, the threshold may be set for each service. Alternatively, projects may be categorized based on the identity of the members, groups or companies participating in the project.

Each development terminal 2 is a terminal device used by participants in software development. Each development terminal 2, for example, creates specifications, creates design documents, creates programs, creates and views test items or test cases, executes tests, reports and views results of test execution, and views status of test execution. etc. may be used. An application for using the test management system S may be installed in each development terminal 2 . Alternatively, it may be possible to use the test management system S via a web browser installed in each development terminal 2 . Examples of the development terminal 2 include a personal computer, a tablet computer, and the like.

[2. Configuration of the center server]
Next, the configuration of the center server 1 will be described with reference to FIGS. 2 and 3. FIG. FIG. 2 is a block diagram showing an example of the schematic configuration of the center server 1 according to this embodiment. As shown in FIG. 2 , the center server 1 includes a system control section 11 , a system bus 12 , an input/output interface 13 , a storage section 14 and a communication section 15 . The system control unit 11 and the input/output interface 13 are connected via a system bus 12 .

The system control unit 11 includes a CPU (Central Processing Unit) 11a, a ROM (Read Only Memory) 11b, a RAM (Random Access Memory) 11c, and the like.

The input/output interface 13 performs interface processing between the storage unit 14 and the communication unit 15 and the system control unit 11 .

The storage unit 14 is configured by, for example, a hard disk drive or the like. Databases such as a project DB 14a, a test report DB 14b, a service threshold DB 14c, a threshold candidate DB 14d, and a project threshold DB 14e may be stored in the storage unit 14, for example. "DB" is an abbreviation for database. FIG. 3 is a diagram showing an example of information stored in a database.

As shown in FIG. 3, the project DB 14a stores project information on each project. The project information may be set by, for example, the person in charge of the project. For example, the project DB 14a may store, as project information, a project ID, a service ID, the number of main tests, the completion date of the main tests, an additional test flag, the number of additional tests, etc., in association with each other. A project ID is identification information for identifying a project. The service ID is identification information that identifies the service to which the project belongs. The number of main tests indicates the number of tests to be performed in the main test of the project. The main test completion flag indicates whether or not the main test of the project has been completed. The main test completion date indicates the date when the main test of the project was completed. The main test completion flag and the main test completion date may be set by the person in charge of the project or the like, or may be automatically set by the center server 1 . For example, the center server 1 may calculate the number of actual tests conducted in the project based on the test report DB 14b, which will be described later. The center server 1 may set the main test completion flag to the completion of the main test when the number of tests performed matches the number of main tests stored in the project DB 14a. At this time, based on the test report DB 14b, the center server 1 may determine the execution date of the final test among the main tests as the completion date of the main test. The additional test flag indicates the decision status of the additional test for the project. For example, the additional test flag may be set to one of "unconfirmed", "not performed", and "must be performed". "Undecided" indicates a state in which it has not yet been decided whether or not to carry out additional tests. "Not Performed" indicates that it was decided not to perform additional tests. "Required" indicates that it has been decided to perform additional tests. The number of additional tests indicates the number of tests that should be performed in the additional tests of the project.

The test report DB 14b stores report information indicating the results of the test each time the test is performed. The report information may be registered, for example, by the person conducting the test. For example, the test report DB 14b may store, as report information, test numbers, project IDs, test types, implementation dates, number of bugs, etc., in association with each other. A test number is a number for identifying a test item or a test case of an executed test. Project ID indicates for which project the test was performed. The test type indicates whether the executed test is a main test or an additional test. The implementation date indicates the date on which the test was implemented. The number of bugs indicates the number of bugs detected in the test.

The threshold value of the bug detection rate is stored for each service in the service threshold DB 14c. For example, the service threshold DB 14c may store the service ID and the current threshold in association with each other. A service ID indicates a service for which the threshold is set. Current Threshold indicates the current threshold. The current threshold may be updated periodically by the center server 1 .

The threshold candidate DB 14d stores candidate information regarding values that have been candidates for the threshold of the bug detection rate so far for each service. For example, at least a service ID may be stored in the threshold candidate DB 14d. The service ID indicates a service for which threshold candidates have been determined. The threshold candidate DB 14d may further store one or more pieces of candidate information in association with the service ID. For example, each time a new threshold candidate is determined, candidate information may be added to the threshold candidate DB 14d. Each candidate information may include an index value, a current threshold, and a threshold candidate. The index value indicates the order in which candidate information is stored among one or a plurality of pieces of candidate information stored in association with the service ID. The current threshold is the current threshold at the time the threshold candidate associated with the current threshold was determined. Note that when the current threshold stored in the service threshold DB 14c is updated, all candidate information of the same service stored in the threshold candidate DB 14d is deleted. Therefore, the current threshold stored in the threshold candidate DB 14d matches the current threshold stored in the service threshold DB 14c.

The project threshold DB 14e stores the threshold of the bug detection rate for each project. For example, the project threshold DB 14e may store the project ID and the threshold in association with each other. Project ID indicates the project for which the threshold is set. For example, the storage unit 14 may store a database (not shown) indicating the progress of each project. In this database, when a certain project progresses from, for example, implementation or smoke test to main test, the center server 1 changes the current threshold stored in the service threshold DB 14c at that time for the service to which the project belongs to the project. may be stored in the project threshold DB 14e as the threshold of .

The storage unit 14 further stores various programs such as an operating system, a DBMS (Database Management System), and a server program. The server program is a program that causes the system control unit 11 to execute a process for managing tests, a process for managing the progress of a project, a process for setting a threshold for a bug detection rate, and the like. For example, the server program may be obtained from another device via the network NW, or may be recorded on a recording medium such as a magnetic tape, optical disk, or memory card and read via a drive device. good too.

The communication unit 15 is configured by, for example, a network interface controller or the like. The communication unit 15 connects to a device such as the development terminal 2 via the network NW, and controls the state of communication with the connected device.

[3. Function overview of the system control unit]
Next, an overview of the functions of the system control unit 11 in the center server 1 will be described with reference to FIGS. 4 to 7. FIG. FIG. 4 is a diagram showing an example of functional blocks of the system control unit 11 in the center server 1 according to this embodiment. The CPU 11a reads and executes various program codes included in the server program, so that the system control unit 11, as shown in FIG. It functions as an update unit 114, a reference difference determination unit 115, an information output unit 116, and the like.

[3-1. Threshold update]
The system control unit 11 performs update control of the threshold value of the bug detection rate. Update control includes determination of whether to update the threshold or not, and actual update of the threshold when it is determined to update the threshold. For example, the system control unit 11 may periodically (for example, every month, every three months, every six months, every year, etc.) execute update control for each of all services. Alternatively, the system control unit 11 may execute update control for a specific service or all services at the timing of receiving an instruction from the administrator of the center server 1 or the like.

At the timing of performing update control, the bug detection rate acquiring unit 111 acquires the bug detection rate of the test target for which the test results were obtained during the past period. In this embodiment, the test target is software developed in a particular project. The past period for which the bug detection rate is obtained is called the "reference period". The reference period may be, for example, a period of a predetermined length specified from the execution date of update control (that is, today). For example, the reference period is the date a predetermined number of months before today (e.g., 1 month, 3 months, 6 months, etc.) or a predetermined number of years (e.g., 1 year, 2 years ago, 3 years ago, etc.). to the current time. Alternatively, the reference period may be a period from the day before the first day of the last month to the last day of the last month. The length of the reference period and the execution interval of update control may or may not match. The reference period may include a date after the last execution date of update control. A test target for which a test result was obtained during the reference period may be, for example, a project for which the main test was completed during the reference period. The date on which the test started need not be included in the reference period. The bug detection rate acquisition unit 111 may determine whether or not the main test was completed during the reference period based on the main test completion date stored in the project DB 14a. Alternatively, by referring to the test report DB 14b, the bug detection rate acquisition unit 111 calculates the number of completed tests for the main test of the project, and combines that number with the number of main tests stored in the project DB 14a. By comparing, it may be determined whether or not the main test is completed. Then, the bug detection rate acquisition unit 111 may determine whether or not the main test was completed during the reference period based on the execution date of the last test in the main test. The results obtained from testing may include, for example, the final bug detection rate in this test.

The bug detection rate acquisition unit 111 may acquire the bug detection rate for each identified project. For example, the bug detection rate acquisition unit 111 may refer to the test report DB 14b to calculate the total number of bugs detected in the main test. Then, the bug detection rate acquisition unit 111 may calculate the bug detection rate by dividing the total number of bugs by the number of actual tests. The test targets for which the bug detection rate is to be acquired may be limited to projects belonging to services subject to threshold update control, among projects that have completed the main test during the reference period. Furthermore, the test targets for which bug detection rates are obtained may be limited to projects for which additional testing has been determined.

The threshold candidate determination unit 112 determines threshold candidates based on the bug detection rate acquired by the bug detection rate acquisition unit 111 . For example, the threshold candidate may be the obtained representative value of the bug detection rate. A representative value of the bug detection rate may be, for example, an average value or a median value. Projects that undergo additional testing will have a relatively high bug detection rate in the main test. On the other hand, even in projects where additional testing is performed, the bug detection rate of the main testing is relatively rare. Also, as long as additional tests are performed, it is relatively rare for the bug detection rate to be extremely low. In addition, projects with the same or very similar bug detection rates may be divided into projects that undergo additional testing and projects that do not. On the other hand, the higher the bug detection rate, the more likely additional testing will be performed. Considering these circumstances, it is considered appropriate to set the threshold to a representative value such as the average value or the median value of the bug detection rates.

For the sake of convenience, the reference period specified at this timing will be referred to as a "first reference period". Also, the bug detection rate obtained at this timing is referred to as a "first bug detection rate." Also, the threshold candidate determined at this timing is referred to as a “first threshold candidate”.

The threshold acquisition unit 113 acquires the current threshold of the bug detection rate. For example, the threshold acquisition unit 113 may acquire the current threshold stored in the service threshold DB 14c for the service subject to threshold update control.

If the difference between the first threshold candidate determined by the threshold candidate determination unit 112 and the threshold acquired by the threshold acquisition unit 113 is less than a predetermined reference difference, the threshold updating unit 114 updates the threshold to the first threshold. Update with suggestions. The reference difference may be preset by, for example, an administrator of the center server 1, or may be determined by the center server 1 as described later. On the other hand, if the difference between the first threshold candidate and the threshold is greater than or equal to the reference difference, the threshold updating unit 114 does not update the threshold with the first threshold candidate. More generally speaking, when the difference between the threshold candidate and the threshold is greater than or equal to the reference difference, the threshold may or may not be updated. A case of updating the threshold will be described later.

FIG. 5 is a diagram illustrating an example of threshold update. As shown in Figure 5, the current threshold for one service is 3.0%. Also, the threshold candidate calculated based on the bug detection rate is 2.8%. Assume that the reference difference is 0.5%. Therefore, the current threshold is updated to 2.8% because the difference between the candidate threshold and the current threshold is less than the reference difference.

After the threshold is not updated with the first threshold candidate by the threshold acquisition unit 113 because the difference between the first threshold candidate and the threshold is equal to or greater than the reference difference, the bug detection rate acquisition unit 111 performs update control. At the timing of , acquire the bug detection rate of the test target for which the test result was obtained during the past period including the days after the first reference period. A period in the past at the time of the next update control is called a "second reference period". The first reference period and the second reference period may or may not overlap. The second reference period may not include days before the first reference period. The length of the first reference period and the length of the second reference period may be the same or different. For example, the bug detection rate acquiring unit 111 may set, as the second reference period, a period of a predetermined length specified from the execution date of the next update control, as in the case of specifying the first reference period. The method for obtaining the bug detection rate may be the same as the method for obtaining the first bug detection rate, except that the reference period is different. The bug detection rate during the current update control is referred to as the "second bug detection rate".

The threshold candidate determination unit 112 determines threshold candidates based on the second bug detection rate acquired by the bug detection rate acquisition unit 111 . The method for determining the threshold candidate may be the same as the method for determining the first threshold candidate. A threshold candidate for the current update control is referred to as a “second threshold candidate”.

If the difference between the second threshold candidate determined by the threshold candidate determination unit 112 and the threshold is less than the reference difference, the threshold updating unit 114 updates the threshold with the second threshold candidate. This point is the same as the update using the first threshold candidate. On the other hand, if the difference between the second threshold candidate and the threshold is equal to or greater than the reference difference, the difference between the first threshold candidate and the threshold was equal to or greater than the reference difference during the previous update control. The threshold may be updated with the second threshold candidate. In other words, since the threshold was not updated during the previous update control, the threshold updating unit 114 can be said to update the threshold.

The bug detection rate threshold and threshold candidate are representative values of the bug detection rates of the test objects for which results were obtained during the reference period. When the reference period changes from the first reference period to the second reference period and there is a change in the tendency of the bug detection rate indicated by the representative value, there is a possibility that the development system, development environment, etc. have changed. Alternatively, there may have been changes in the criteria for judging the necessity of conducting additional tests. It is considered that the threshold is appropriately updated by reflecting changes in the development system, development environment, criteria for conducting additional tests, etc. in the threshold. However, the changed situation is temporary, and if a few days pass, the development system, development environment, etc. may return to the original state. Alternatively, the development system, development environment, etc. may not have changed, or the tendency of the bug detection rate may have happened to change only at that time, regardless of the change. In such cases, it may not always be appropriate to update the threshold with the threshold candidate. Therefore, even if the difference between the threshold candidate indicating the tendency of the bug detection rate and the threshold exceeds the reference difference only once, the threshold updating unit 114 does not update the threshold at that time. If the difference between the threshold candidate and the threshold is continuously greater than or equal to the reference difference, the threshold update unit 114 determines that the change in the tendency of the bug detection rate is not accidental and that the state of change is continuous. is updated with the threshold candidate. On the other hand, if the difference between the threshold candidate and the threshold is less than the standard difference, even if the change in the tendency of the bug detection rate is accidental, or the changed development system or development environment was temporary. However, the effect of updating the threshold is relatively small. Therefore, if the difference between the threshold candidate and the threshold is less than the reference difference, the threshold update unit 114 may update the threshold without considering other conditions.

Threshold updating unit 114 does not update the threshold with the first threshold candidate because the difference between the first threshold candidate and the threshold is greater than or equal to the reference difference, and then the threshold updating unit 114 determines that the difference between the second threshold candidate and the threshold is greater than or equal to the reference difference. , it may be determined whether or not to update the threshold with the second threshold candidate based on a further condition. For example, the threshold updating unit 114 may update the threshold with the second threshold candidate only when the magnitude relationship between the first threshold candidate and the threshold matches the magnitude relationship between the second threshold candidate and the threshold. good. The magnitude relationship between a threshold candidate and a threshold is a relationship indicating whether or not the threshold candidate is higher than the threshold. Specifically, when the first threshold candidate is higher than the threshold and the second threshold candidate is also higher than the threshold, the magnitude relationships match. Also, when the first threshold candidate is lower than the threshold and the second threshold candidate is also lower than the threshold, the magnitude relationships match. Even if the difference between the threshold candidates and the thresholds is continuously equal to or greater than the reference difference, one of the first threshold candidate and the second threshold candidate is higher than the threshold, while the other threshold candidate is higher than the threshold. If it is lower than , it cannot be said that the changed development system, development environment, etc. are continuing.

A generalized example of the threshold updating method described so far will be described. If the difference between the current threshold candidate and the threshold is less than the reference difference, the threshold updating unit 114 may update the threshold with the threshold candidate. If the difference between the current threshold candidate and the threshold is greater than or equal to the reference difference, the threshold update unit 114 may determine whether or not the threshold was updated with the threshold candidate at that time during the previous update control. If the threshold was updated during the previous update control, the threshold updating unit 114 does not need to update the threshold with the current threshold candidate. If the threshold has not been updated during the previous update control, the threshold updating unit 114 determines whether the magnitude relationship between the previous threshold candidate and the threshold matches the magnitude relationship between the current threshold candidate and the threshold. may be determined. If the magnitude relationships match, the threshold updating unit 114 may update the threshold with the current threshold candidate. If the magnitude relationship does not match, the threshold updating unit 114 does not have to update the threshold with the current threshold candidate.

The initial value of the threshold may be set in advance by, for example, the administrator of the center server 1, or may be set by the center server 1 based on the bug detection rate as described above.

FIG. 6 is a diagram illustrating an example of threshold update. For example, assume that update control is performed every month. Also assume that the reference difference is 0.5%. As shown in FIG. 6, the current threshold is 3.0% in April. The threshold candidate calculated at this time is 3.8%. Therefore, the difference between the threshold candidate and the current threshold at this time is greater than or equal to the reference difference. If the current threshold was updated in March, the current threshold is not updated. Entering May, the calculated threshold candidate is 3.7%. The difference between this threshold candidate and the current threshold is greater than or equal to the reference difference. Current thresholds were not updated in April. The April threshold candidate is higher than the current threshold, and the May threshold candidate is also higher than the current threshold. Therefore, the current threshold is updated to 3.7.

[3-2. Reference difference setting]
Next, a method for automatically setting the reference difference will be described. Based on the bug detection rate acquired by the bug detection rate acquisition unit 111, the reference difference determination unit 115 determines a specific value that serves as a boundary between an outlier and a value other than the outlier of the bug detection rate, and a value of the bug detection rate. The difference between the representative value and , may be determined as the reference difference. A range of bug detection rates that does not become an outlier is called a “normal interval”. For example, the reference difference determination unit 115 may determine the reference difference based on how far the lower limit value or upper limit value of the normal interval is from the representative value. If the difference between the candidate threshold and the threshold is greater than or equal to the reference difference thus determined, the candidate threshold may be an outlier. Therefore, if the threshold candidate may be an outlier, it may not be appropriate to update the threshold with the threshold candidate. As described above, when the difference between the threshold candidate and the threshold is continuously greater than or equal to the reference difference, the threshold may be updated.

The specific value may be at least one of the lower limit value and the upper limit value of the normal interval. A representative value of the bug detection rate may be an average value or a median value. This representative value may be the threshold candidate itself.

The standard difference determination unit 115 may determine the standard difference using the standard deviation of the bug detection rate. In this case, the reference difference determination unit 115 does not have to calculate the specific value. When the bug detection rate is a random variable and the number of projects is a frequency, the distribution of the bug detection rate may be a normal distribution or a distribution that is relatively close to the normal distribution. The reference difference determination unit 115 may calculate the reference difference, for example, by multiplying the standard deviation by a predetermined coefficient that is a real number greater than zero. For example, assuming that normal intervals account for approximately 95% of the total bug detection rate, the coefficient may be 2.0. Note that even if the bug detection rate distribution is not actually a normal distribution, the standard difference determination unit 115 may determine the standard difference using the standard deviation.

FIG. 7 is a graph showing an example of bug detection rate distribution. In FIG. 7, the distribution of the bug detection rate is a normal distribution. Let μ be the average value of the bug detection rates, and let μ be the standard deviation multiplied by a predetermined coefficient as the standard difference x. The normal interval in this case ranges from μ−x to μ+x. A bug detection rate that is less than μ−x and a bug detection rate that is greater than μ+x are outliers.

The reference difference determination unit 115 may use the percentile value of the bug detection rate to determine the reference difference. This percentile value is a value different from the median value and the average value. For example, assume that normal intervals account for N% of the total bug detection rate. In this case, the reference difference determination unit 115 uses the ((100-N)/2) percentile value as the lower limit value of the normal interval, and the (100-(100-N)/2) percentile value as the upper limit value of the normal interval. At least one of the values may be calculated. Then, the reference difference determination unit 115 may determine the difference between the percentile value and the representative value as the reference difference. If the bug detection rate distribution is not a normal distribution, the difference between the lower limit value and the representative value of the normal interval may differ from the difference between the upper limit value and the representative value of the normal interval. At this time, for example, if the threshold candidate is higher than the threshold, the threshold update unit 114 uses the reference difference calculated from the lower limit of the normal interval, and if the threshold candidate is lower than the threshold, the threshold update unit 114 uses the upper limit of the normal interval. You may use the standard difference calculated by .

[3-3. Information provision based on bug detection rate and threshold]
The information output unit 116 outputs information based on the bug detection rate and the threshold. For example, suppose that the detection rate of bugs detected in a given test object under test is greater than or equal to a threshold. In that case, the information output unit 116 may output risk information indicating that the possibility of performing additional tests on the given test target is higher than when the bug detection rate is less than the threshold. The state of being tested may be, for example, a state in which, of all the tests included in the main test, some tests have been performed, but the rest of the tests have not yet been performed. A given test target may be (software developed in) a project whose information is displayed on the screen of the development terminal 2, for example. For example, when the progress status of one or more projects is displayed on the screen of the development terminal 2, the information output unit 116 develops the risk information so that the risk information is displayed for each project under test. You may transmit to the terminal 2. The information output unit 116 may calculate the bug detection rate based on the report information of the main test stored in the test report DB 14b in association with the project ID of the project. Further, the information output unit 116 may make a determination using a threshold stored in the project threshold DB 14e in association with the project ID of the project.

If the final bug detection rate at the stage where this test is completed exceeds the threshold, the possibility of conducting additional tests increases. Therefore, if the bug detection rate exceeds the threshold in the middle of the main test, by presenting this to the developer as risk information, It can provide the developer with an opportunity to consider reviewing the environment.

Various methods of displaying risk information are conceivable. For example, the information output unit 116 causes the development terminal 2 to display the progress of each project belonging to the service specified by the developer. For example, the information output unit 116 may display information indicating progress on a web page. Project progress information indicating each project is displayed on the screen of the development terminal 2 in association with the progress of the project. Project progress information may include at least the project name. Project progress information may also include bug detection rates if the project is in the testing phase. The information output unit 116 may change the display mode of the project progress information of the project in the testing stage based on whether the bug detection rate is equal to or higher than the threshold. Project progress information with different display modes is an example of risk information. Examples of display modes include the color, shape, and size of the project progress information, the color and thickness of the outline of the project progress information, and the color, style, size, and thickness of the characters in the project progress information. be done. Project progress information may be displayed in yet another manner for projects that have already undergone additional testing, or projects that are currently undergoing main testing but have already been determined to undergo additional testing. Alternatively, the information output unit 116 may include, as risk information, a message indicating that there is a high possibility of an additional test in the project progress information.

[4. Operation of the center server]
Next, operation of the center server 1 will be described with reference to FIG. FIG. 8 is a flowchart showing an example of threshold update control processing by the system control unit 11 of the center server 1 according to this embodiment. The system control unit 11 may periodically execute the threshold update control process for each service, or may execute the threshold update control process for each service in response to receiving a request from the development terminal 2 for implementation by the developer. may be executed.

As shown in FIG. 8, the bug detection rate acquisition unit 111 acquires the bug detection rate of a project whose main test completion date is within the reference period for the service to be processed (step S101). For example, the bug detection rate acquisition unit 111 may set a period from a predetermined number of months or a predetermined number of years before today to today as the reference period. From the project DB 14a, the bug detection rate acquisition unit 111 selects the project information including the service ID of the target service and the additional test flag set to "execution required", the project information including the main test completion date in the reference period. can be searched. The bug detection rate acquisition unit 111 may search for report information including a combination of a project ID included in the project information and a test type indicating the final test for each project information retrieved. The bug detection rate acquisition unit 111 may calculate the number of searched report information as the number of tests. The bug detection rate acquisition unit 111 may acquire the number of bugs from each report information and calculate the total number of bugs. The bug detection rate acquisition unit 111 may calculate the bug detection rate by dividing the total number of bugs by the number of tests.

Next, the reference difference determination unit 115 calculates the standard deviation of the bug detection rate acquired by the bug detection rate acquisition unit 111 (step S102). Next, the reference difference determining unit 115 calculates the reference difference by multiplying the standard deviation by a preset coefficient K (step S103). Next, the threshold acquisition unit 113 acquires the current threshold associated with the service ID of the target service from the service threshold DB 14c (step S104). Next, the threshold candidate determination unit 112 calculates a representative value of the bug detection rate acquired by the bug detection rate acquisition unit 111 as a threshold candidate (step S105).

Next, the threshold update unit 114 calculates the current deviation by subtracting the current threshold from the threshold candidate (step S106). Next, the threshold update unit 114 determines whether or not the absolute value of the current deviation is less than the reference difference (step S107).

If the absolute value of the current deviation is less than the reference difference (step S107: YES), the threshold updating unit 114 updates the current threshold with the threshold candidate (step S108). For example, the threshold updating unit 114 may rewrite the current threshold associated with the service ID of the target service and stored in the service threshold DB 14c with the threshold candidate (step S108). Next, if at least one item of candidate information is stored in the threshold candidate DB 14d in association with the service ID of the target service, the threshold update unit 114 deletes all of the candidate information from the threshold candidate DB 14d (step S109). , the threshold update control process ends.

On the other hand, if the absolute value of the current deviation is greater than or equal to the reference difference (step S107: NO), the threshold update unit 114 associates the service ID of the target service with the candidate information and stores it in addition to the service threshold DB 14c ( step S110). At this time, the threshold update unit 114 includes the current threshold and the threshold candidate as the current threshold in the candidate information. Further, if there is no candidate information already stored in association with the service ID of the target service, the threshold updating unit 114 sets the index value of the candidate information to be added to 1. On the other hand, if there is already stored candidate information, 1 is added to the maximum index value among the index values included in the candidate information to calculate the index value of the candidate information to be added. Next, the threshold updating unit 114 determines whether or not the previous candidate information is stored in the service threshold DB 14c (step S111). That is, the threshold updating unit 114 determines whether candidate information has already been stored in association with the service ID of the target service before step S110. The previous candidate information is candidate information having an index value that is one less than the index value of the candidate information added this time. If the previous candidate information is not stored (step S111: NO), the threshold update unit 114 determines not to update the current threshold (step S114), and ends the threshold update control process.

On the other hand, if the previous candidate information is stored (step S111: YES), the threshold update unit 114 subtracts the current threshold included in the previous candidate information from the previous threshold candidate included in the previous candidate information. Thus, the previous deviation is calculated (step S112). Next, the threshold update unit 114 determines whether or not the sign of the current deviation and the sign of the previous deviation match (step S113). If the current deviation is a positive value and the previous deviation is also a positive value, the signs match. Also, if the current deviation is a negative value and the previous deviation is also a negative value, the signs match. If the signs match (step S113: YES), the threshold updating unit 114 updates the current threshold with the threshold candidate (step S108). If the signs do not match (step S113: NO), the threshold updating unit 114 determines not to update the current threshold (step S114). If the absolute value of the current deviation is greater than or equal to the reference difference (step S107: YES), the threshold updating unit 114 further determines whether or not the absolute value of the previous deviation is greater than or equal to the currently calculated reference difference. may Then, when the absolute value of the previous deviation is less than the current reference difference, the threshold updating unit 114 does not have to update the current threshold.

As described above, in the present embodiment, the center server 1 may acquire the first bug detection rate of projects for which test results were obtained during the past first reference period. Alternatively, the center server 1 may determine the first threshold candidate based on the first bug detection rate. Further, when the difference between the first threshold candidate and the current threshold is less than the reference difference, the center server 1 may update the current threshold with the first threshold candidate. Also, after the current threshold is not updated with the first threshold candidate, the center server 1 determines the first 2 bug detection rate may be obtained. Alternatively, the center server 1 may determine the second threshold candidate based on the second bug detection rate. Further, when the difference between the second threshold candidate and the current threshold is greater than or equal to the reference difference, the center server 1 may update the current threshold with the second threshold candidate. In this case, after the current threshold is not updated with the first threshold candidate because the difference between the first threshold candidate and the current threshold is equal to or greater than the reference difference, the second threshold of the current threshold is similarly applied at the next timing. If the difference between the candidate and the current threshold is greater than or equal to the reference difference, the current threshold is updated with the second candidate threshold. If there is a difference between the current threshold and the first threshold candidate for the current threshold determined based on the first bug detection rate of the bugs in the project for which the test results were obtained during the first reference period, the period There may have been some changes in the software development system and development environment. The current threshold is updated to reflect this change. On the other hand, the altered state is temporary and may soon return to its original state. Alternatively, there may have been a difference between the first threshold candidate of the current threshold and the current threshold, rather than there being a change in the development system or development environment. Therefore, the current threshold is not updated immediately in view of the impact of a large change in the current threshold. After that, as a result of comparing the current threshold with the second candidate threshold determined based on the second bug detection rate of the bugs of the project for which the test results were obtained during the new second reference period, the difference If is large, there is a possibility that the changed state continues to some extent. The current threshold is then updated with the second candidate threshold. Therefore, it is possible to appropriately update the threshold for the detection rate of bugs in software testing.

Here, the project for which the bug detection rate is obtained may be the project for which the implementation of the additional test has been decided among the projects for which the necessity of the implementation of the additional test has been decided based on the results of the main test. . In this case, the bug detection rates for the projects that have undergone additional testing are used to determine threshold candidates. Therefore, the updated current thresholds can be used to better determine the likelihood of additional testing for a given project.

At this time, the center server 1 may acquire the detection rate of bugs detected in the main test as each of the bug detection rates. In this case, the threshold candidate is determined using the detection rate of bugs detected in the main test for the project for which the additional test was performed. By using the detection rate of bugs, which may affect the decision of whether or not to perform additional tests, it is possible to appropriately determine the threshold candidates.

Further, when the difference between the second threshold candidate and the current threshold is equal to or greater than the reference difference, and the difference between the first threshold candidate and the current threshold is equal to or greater than the reference difference, the center server 1 selects the second threshold candidate and the current threshold match the magnitude relationship between the first threshold candidate and the current threshold, the current threshold may be updated with the second threshold candidate. In this case, the possibility that the changed state continues can be appropriately reflected in the current threshold.

Alternatively, the center server 1 may calculate a representative value of the bug detection rate as a threshold candidate. Also, the representative value of the bug detection rate may be either the average value or the median value.

Further, when the bug detection rate detected in the tests of a given project under test is equal to or higher than the current threshold, the center server 1 determines the possibility of conducting additional tests for the project. Risk information may be output indicating higher than if less than. In this case, by outputting the risk information, it is possible to give the developer an opportunity to review the development system and development environment in order to avoid the execution of additional tests.

Also, the center server 1 may acquire the current threshold of the bug detection rate in software testing. Also, the center server 1 may acquire the bug detection rate of each of a plurality of projects tested in the past. Alternatively, the center server 1 may determine threshold candidates based on the bug detection rate. Further, the center server 1 determines a reference difference between a specific value that is a boundary between an outlier of the bug detection rate and a value other than the outlier, and a representative value of the bug detection rate, based on the bug detection rate. may Further, when the difference between the current threshold and the threshold candidate is less than the reference difference, the center server 1 may update the current threshold with the threshold candidate. In this case, the reference difference corresponds to the range of detection rates of bugs that are not outliers. For reasons similar to those described above, the current threshold is updated when the difference between the threshold candidate and the current threshold is less than the reference difference, considering the impact of a large change in the current threshold. On the other hand, if the difference between the threshold candidate and the current threshold is greater than or equal to the reference difference, the current threshold may not be updated immediately, or whether to update the current threshold may be determined based on other conditions. Therefore, it is possible to appropriately update the threshold for the detection rate of bugs in software testing.

Here, the center server 1 may determine the reference difference based on the standard deviation of the bug detection rate. In this case, the standard deviation of the bug detection rate can be used to appropriately determine the reference difference.

1 center server 2 development terminal 11 system control unit 12 system bus 13 input/output interface 14 storage unit 14a project DB
14b Test report DB
14c Service threshold DB
14d threshold candidate DB
14e Project threshold DB
NW network 111 bug detection rate acquisition unit 112 threshold candidate determination unit 113 threshold acquisition unit 114 threshold update unit 115 reference difference determination unit 116 information output unit

Claims (9)

an acquisition means for acquiring a first detection rate of bugs of a first test target for which test results were obtained during a past first period;
determining means for determining a first candidate for a bug detection rate threshold based on the acquired first detection rate;
updating means for updating the threshold with the first candidate when the difference between the determined first candidate and the threshold is less than a predetermined value;
with
The obtaining means obtains a second test result during a past second period including days later than the first period after the threshold was not updated with the first candidate by the updating means. obtain a second detection rate of the bug under test;
The determining means determines a second candidate for the threshold based on the obtained second detection rate,
The bug detection rate threshold updating system, wherein the updating means updates the threshold with the second candidate when a difference between the second candidate and the threshold is equal to or greater than the predetermined value. Each of the first test subject and the second test subject is determined to perform the additional test among the test subjects for which it is determined whether or not to perform the additional test based on the results of the scheduled tests. 2. The bug detection rate threshold update system according to claim 1, wherein the bug detection rate threshold update system is a test object that has been tested. 3. The bug detection rate according to claim 2, wherein said acquisition means acquires detection rates of bugs detected in said scheduled test as said first detection rate and said second detection rate, respectively. Threshold update system. When the difference between the second candidate and the threshold value is greater than or equal to the predetermined value and the difference between the first candidate and the threshold value is greater than or equal to the predetermined value, the updating means updates the second candidate and the threshold, and the magnitude relationship between the first candidate and the threshold match, the threshold is updated with the second candidate. 4. The bug detection rate threshold updating system according to any one of 3. 5. The bug detection rate threshold according to any one of claims 1 to 4, wherein the determining means calculates a representative value of the bug detection rate for each of the first candidate and the second candidate. update system. 6. The bug detection rate threshold updating system according to claim 5, wherein the representative value is one of an average value and a median value. If the detection rate of bugs detected in the test for a given test target that is being tested is equal to or greater than the threshold, the possibility of performing an additional test for the given test target is determined if the detection rate of the bug is 7. The bug detection rate threshold update system according to claim 1, further comprising output means for outputting possibility information indicating that the probability is higher than the case of being less than the threshold. A computer-implemented bug detection rate threshold updating method comprising:
a first acquisition step of acquiring a first detection rate of bugs of a first test target for which test results were obtained during a past first period;
a first determination step of determining a first candidate for a bug detection rate threshold based on the obtained first detection rate;
a first updating step of updating the threshold with the first candidate if the difference between the determined first candidate and the threshold is less than a predetermined value;
of a second test subject for whom a test result was obtained during a past second time period including days later than the first time period after the first updating step did not update the threshold with the first candidate; a second obtaining step of obtaining a second detection rate of bugs;
a second determination step of determining a second candidate for the threshold based on the obtained second detection rate;
a second updating step of updating the threshold with the second candidate when the difference between the second candidate and the threshold is equal to or greater than the predetermined value;
A method for updating a bug detection rate threshold, comprising: the computer,
an acquisition means for acquiring a first detection rate of bugs of a first test target for which test results were obtained during a past first period;
determining means for determining a first candidate for a bug detection rate threshold based on the acquired first detection rate;
updating means for updating the threshold with the first candidate when the difference between the determined first candidate and the threshold is less than a predetermined value;
function as
The obtaining means obtains a second test result during a past second period including days later than the first period after the threshold was not updated with the first candidate by the updating means. obtain a second detection rate of the bug under test;
The determining means determines a second candidate for the threshold based on the obtained second detection rate,
The bug detection rate threshold updating program, wherein the updating means updates the threshold with the second candidate when a difference between the second candidate and the threshold is equal to or greater than the predetermined value.

Images