JP2020123287A - Information processing apparatus, test method, and test program - Google Patents

Abstract

To provide an information processing apparatus automatically testing software that, when the software is rejected in a plurality of tests due to the same failure, can efficiently analyze the failure that causes the rejection.SOLUTION: In an information processing system 100, an information processing apparatus has a test control unit, a first determination unit, a second determination unit, an output unit, a setting receiving unit, and a storage unit. The test control unit controls execution of a plurality of tests on software to be tested. When the software is rejected in the tests, the first determination unit determines whether the rejected software has functional characteristics. When the software is rejected in the tests, the second determination unit determines whether the rejected software has phenomenal characteristics. The output unit outputs a test result related to a failure that causes the rejection, based on the phenomenal characteristics and the functional characteristics of a test, of the plurality of tests, in which the software is rejected.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing device, a test method, and a test program.

2. Description of the Related Art There is known a test system that automatically tests software to be tested and outputs a test result.

In a software development project, there is known a bug analysis method and apparatus for analyzing a bug and selecting a task to be prioritized based on the man-hours required for rework due to the bug (for example, refer to Patent Document 1).

However, in a test system that automatically tests the software to be tested, even if multiple tests fail due to the same failure (bug), it is possible to determine whether the cause is the same failure or not. It is difficult to determine automatically. Therefore, even if multiple tests fail due to the same failure, the test system issues a failure report for all failed tests, and the designer has multiple failure reports. There is the problem of having to analyze the votes to determine the failure that caused the failure.

One embodiment of the present invention is made in view of the above problems, and in an information processing system that automatically tests software, when multiple tests fail due to the same failure. , To efficiently analyze the failure that caused the failure.

In order to achieve the above object, an information processing apparatus according to an embodiment of the present invention is an information processing apparatus that tests software to be tested, and includes a test control unit that controls execution of a plurality of tests related to the software. An output unit that outputs a test result regarding the failure that caused the failure, based on the characteristics of the phenomenon surface and the characteristics of the functional surface in the failure test of the plurality of tests. ..

According to an embodiment of the present invention, in an information processing system that automatically tests software, when a plurality of tests fail due to the same failure, analysis of the failure causing the failure is performed. You can do it efficiently.

It is a figure showing an example of a system configuration of an information processing system concerning one embodiment. It is a figure which shows the structural example of the test tool which concerns on one Embodiment. It is a figure which shows the example of the hardware constitutions of the information processing apparatus which concerns on one Embodiment. It is a figure showing an example of functional composition of an information processor concerning one embodiment. It is a figure for demonstrating the test specification which concerns on one Embodiment. It is a figure for explaining an example of a test case concerning one embodiment. 6 is a flowchart showing an example of processing of the information processing apparatus according to the first embodiment. 9 is a flowchart showing an example of processing of the information processing apparatus according to the second embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<System configuration>
(System configuration of information processing system)
FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system according to an embodiment. The information processing system 100 is a system in which a designer 101 who designs software tests the software to be tested.

In recent software development, problems such as continuous integration (CI: Continuous Integration) are frequently repeated by repeatedly performing builds and tests to detect problems early, improving development efficiency, saving power, and shortening delivery times. Etc. are used. In the CI system, the procedure is automated, and the efficiency of the entire workflow is improved by cooperating with the change management tool 102, the test management tool 106, the failure management tool 109, and the like. The CI system is an example of the information processing system 100.

In the information processing system 100 shown in FIG. 1, a designer 101 creates a source code 103 of software to be tested and a test code 107 for testing the software and registers it in the information processing system 100.

The build tool 104 and the test tool 108 automatically execute a build and a test according to a predetermined schedule, and if the test result is a failure, the failure management tool 109 gives a test result that has failed. Etc. to the designer 101. At this time, the fault management tool 109 can track the countermeasure status of the fault by cooperating with the change management tool 102, the test management tool 106, and the like.

If the test result is unacceptable, the designer 101 analyzes the failure (bug) and corrects the source code 103 or the test code 107 as necessary. In analyzing the failure, the designer 101, for example, specifies the failure from what kind of error occurred in which test case, and log information when the error occurred, and takes countermeasures.

By repeatedly executing such a flow, the designer 101 always maintains a state in which the test result is acceptable. As a result, the designer 101 can keep the software (artifact) to be tested ready to be released, so that the software can be released to the market in a timely manner.

(Example of test tool configuration)
FIG. 2 is a diagram showing a configuration example of a test tool according to an embodiment. Here, a test tool in embedded software development will be described as an example. The automatic test in the embedded software development is basically performed by executing the test target object 105 on the target device 202 as the target. Here, the “target device” includes not only an electronic device in which software is actually installed (hereinafter, referred to as “actual device”) but also an emulator that emulates the operation of the actual device.

As an example of the method of executing the test target object 105 on the target device 202, as shown in FIG. 2A, the test tool 108 uses the test runner 203 on the target device 202 from the test tool 108. There is a method of executing 105 and the test code 107. As another example, as shown in FIG. 2B, the test tool 108 executes the test code 107, and the target device 202 uses an interface (I/F: Interface) open to the test target. There is also a method of operating the object 105. The method shown in FIG. 2A is mainly used for a white box test, and the method shown in FIG. 3B is mainly used for a black box test in many cases.

In the above configuration, the information processing system 100 automatically executes a test, for example, and when the test fails, automatically issues a failure report and notifies the designer 101. In such an information processing system 100, for example, when a plurality of tests fail due to the same failure (bug), the information processing system 100 automatically determines that the cause is the same failure. Things are difficult. Therefore, the information processing system 100 issues a failure report for all the failed tests, and the designer 101 analyzes a plurality of failure reports to analyze the plurality of failed tests. There is the problem of having to determine that the cause is due to the same failure.

At this time, for example, a method in which the information processing system 100 groups the failures of the same failure as the “same failure” (grouping) and notifies the designer 101 is possible. However, with this method alone, the failure of the same phenomenon is not always due to the same cause, and thus a potential failure (bug) may be overlooked.

Further, as another method, the information processing system 100 divides the basic function test and the branch and leaf function test, and performs the basic function test first, so that a large number of failure votes are generated. A method of suppressing votes is also possible. However, in this method, it is necessary to classify which functions are basic functions and which are branch and leaf functions. In addition, this method has a problem that if a basic function test fails, another function cannot be tested.

Therefore, in the information processing system 100 according to the present embodiment, when a test is automatically executed and the test fails, based on the phenomenological aspect and the functional aspect of the failed test. , Outputs test results for the failure that caused the failure.

Preferably, the information processing system 100 divides the failed tests (for example, failure report) into groups based on the characteristic of the phenomenon side and the characteristic of the function in the failed test, and outputs the test result. Is output.

Here, the functional characteristics include, for example, characteristics regarding “function” described in the functional specification of the software to be tested. For example, the information processing system 100 functions as a failed test based on the number of failed test items among one or more test items related to a predetermined function described in the functional specification. Whether there is a characteristic characteristic (for example, whether there is a problem in a predetermined function) or not.

Further, the characteristics of the phenomenon surface include, for example, characteristics relating to the “phenomenon” generated by a predetermined command used for the test. For example, the information processing system 100 may determine whether the failed test has a phenomenological characteristic based on the number of times an error has occurred in a predetermined command used in a software test (an error of the same phenomenon occurs. Yes) or not. It should be noted that the command may include various instructions such as an instruction to execute a predetermined procedure and an instruction to access a predetermined hardware.

As a specific example, when there is a phenomenon-side feature such as an error occurring multiple times with a predetermined command, the information processing system 100 gives priority to the function-side feature and the test result regarding the phenomenon-side feature. Is notified to the designer 101. This is because if many errors occur in a specific command, it is considered that there is a high possibility that an appropriate test cannot be performed on the function to be tested.

For example, it is assumed that the device to be tested is an MFP (Multifunction Peripheral) having a scan function, a copy function, a printer function, a facsimile function, and the like mounted in one housing. Further, in the software test of the MFP, a failure occurs in the scan function test, but it is actually assumed that the error occurred due to the address book registration command included in the preparatory work. In such a case, according to the information processing system 100 according to the present embodiment, it becomes easy to request the person in charge of the address book function, not the person in charge of the scan function, for failure analysis. Note that the specific processing contents will be described later.

<Hardware configuration>
FIG. 2 is a diagram illustrating an example of the hardware configuration of the information processing device according to the embodiment. The change management tool 102, the build tool 104, the test management tool 106, the test tool 108, the failure management tool 109, and the like shown in FIG. 1 are, for example, an information processing apparatus 300 having a computer configuration as shown in FIG. It is executed by the information processing device 300.

The information processing apparatus 300 includes, for example, a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, an HD (Hard Disk) 304, an HDD (Hard Disk Drive) controller 305, and a display 306. An external device connection I/F 307, a network I/F 308, a keyboard 309, a pointing device 310, a DVD-RW (Digital Versatile Disk Rewritable) drive 312, a media I/F 314, a bus line 315, and the like.

Of these, the CPU 301 controls the overall operation of the information processing device 300. The ROM 302 stores, for example, a program used for starting the CPU 301 such as an IPL (Initial Program Loader). The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data such as programs. The HDD controller 305 controls reading or writing of various data with respect to the HD 304 under the control of the CPU 301.

The display 306 displays various information such as a cursor, a menu, a window, characters, or an image. The external device connection I/F 307 is an interface for connecting various external devices. The external devices include, for example, a USB (Universal Serial Bus) memory, a printer, and the like, as well as a target device 202 to be tested. The network I/F 308 is an interface for performing data communication using a communication network. The information processing apparatus 300 may control the target device 202 via the network I/F 308.

The keyboard 309 is a type of input means having a plurality of keys for inputting characters, numerical values, various instructions, and the like. The pointing device 310 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The DVD-RW drive 312 controls reading or writing of various data with respect to a DVD-RW 311 as an example of a removable recording medium. The DVD-RW 311 is not limited to the DVD-RW and may be a DVD-R or the like. The media I/F 314 controls reading or writing (storage) of data with respect to the medium 313 such as a flash memory. The bus line 315 includes an address bus, a data bus, various control signals, etc. for electrically connecting the above components.

<Functional configuration>
Next, the functional configuration of the information processing device 300 will be described.

FIG. 4 is a diagram illustrating an example of a functional configuration of the information processing device according to the embodiment. The information processing apparatus 300, for example, executes a predetermined test program in the CPU 301 of FIG. 3 to cause the test control unit 401, the first determination unit 402, the second determination unit 403, the output unit 404, and the setting reception unit 405. , And the storage unit 406 and the like. Note that at least a part of the above functional configurations may be realized by hardware.

The test control unit 401 is realized by, for example, the test tool 108 or the like executed by the CPU 301 of FIG. 3, and controls execution of a plurality of tests relating to software to be tested.

The first determination unit 402 is realized by, for example, a test program such as the failure management tool 109 executed by the CPU in FIG. 3, and when the software test fails, the failing software has a functional aspect. It is determined whether there is a feature of. For example, based on the number of test items that have failed among the one or more test items related to a predetermined function, the first determination unit 402 has a functional feature in the test that has failed. It is determined whether (for example, there is a problem with a predetermined function).

FIG. 5 is a diagram for explaining the test specifications according to the embodiment. For example, when creating a test case for verifying functional conformity, the designer 101 will design a test case for the functional specification 501 of the software to be tested. For example, when testing the sub-function A-1 of the function A in the functional specification 501 of FIG. 5, one or more test items related to the sub-function A-1 of the function A are tested as shown in the test specification 411 of FIG. To do. Similarly, when testing the sub-function A-2 of the function A in the functional specification 501, as shown in the test specification 411, one or more test items related to the sub-function A-2 of the function A are tested.

Preferably, for example, when a plurality of test items related to a predetermined function are defined in the test specification 411, the first determination unit 402 sets the number of rejected test items to a preset first value. When it is equal to or more than the threshold of, it is determined that the predetermined function has a functional feature. When there is one test item related to a predetermined function, it may be determined that the predetermined function has a functional characteristic when the test item related to the predetermined function fails.

The second determination unit 403 is realized by, for example, a test program such as the failure management tool 109 executed by the CPU in FIG. 3, and when the software test fails, the failure software is displayed as a phenomenon surface. It is determined whether there is a feature of. For example, the second determination unit 403 determines whether the failed test has a phenomenological feature based on the number of times an error has occurred in a predetermined command (for example, whether the predetermined command has a problem). Determine whether or not.

FIG. 6 is a diagram for explaining an example of a test case according to an embodiment. For example, when performing a test on a predetermined function, the precondition and the postcondition are clarified before the test scenario is executed, and the test is executed. For example, as shown in a test case 601 of FIG. 6, a “pre-condition”, a “post-condition”, etc. are automatically executed before and after a “test scenario” in which a test regarding a predetermined function is performed.

The automatic execution of each procedure included in the test case 601 can be executed by various methods. For example, in many cases, the manufacturer that provides the target device 202 provides various tools for testing similarly to the build tool 104. Therefore, using these tools, the test target object 105 can be obtained. It can be operated. Further, for example, the test target object 105 can use an I/F provided to the outside for automatic execution of each procedure included in the test case 601. The I/F may be a test target in another test case.

Preferably, in the software test, the second determination unit 403 fails when the number of errors that has occurred in a predetermined command (for example, the execution instruction of the procedure B1) is the second threshold value or more. Judge that the software has phenomenological features.

The output unit 404 is realized by, for example, a test program such as the failure management tool 109 executed by the CPU in FIG. The output unit 404 outputs a test result relating to the failure that caused the failure, based on the characteristics of the phenomenon surface and the characteristics of the functional surface in the failed test among the plurality of tests.

For example, the output unit 404 gives priority to the characteristic of the phenomenon surface over the characteristic of the function surface and gives a predetermined command (procedure) when the failed test has the characteristic of the function surface and the characteristic of the phenomenon surface. The test result indicating that there is a fault is output.

Further, the output unit 404 outputs a test result regarding a predetermined function when the rejected test has a functional feature and no phenomenon feature. Furthermore, the output unit 404 outputs a test result related to a predetermined command when the failed test has a phenomenon-side characteristic and no functional-side characteristic.

Preferably, the output unit 404 divides the failed tests (or failure forms, etc.) into groups based on the characteristics of the phenomenon side and the characteristics of the functional side in the failed tests, and outputs them as test results. Output. For example, the output unit 404 groups and outputs a plurality of failure votes that are considered to be caused by the same failure, based on the characteristics of the phenomenon surface and the characteristics of the functional surface in the failed test.

The setting reception unit 405 is realized by, for example, a test program such as the failure management tool 109 executed by the CPU in FIG. 3, and receives the above-described setting of the first threshold value or the second threshold value. For example, the setting reception unit 405 displays a setting screen for setting the first threshold value or the second threshold value on the display unit such as the display 306 in FIG. 3, and the first threshold value set by the designer 101 or the like. Alternatively, the second threshold setting operation is accepted. Further, the setting reception unit 405 stores the received first threshold value or second threshold value as the set value 413 in the storage unit 406. The setting reception unit 405 may receive a setting request for the first threshold value or the second threshold value from another information processing device or the like.

The storage unit 406 is realized by, for example, the program executed by the CPU of FIG. 3, the HDD controller 305, the HD 304, and the like, and stores the test code 107, the test specification 411, the test result 412, the set value 413, and the like.

In the example of FIG. 4, the functional configurations of the test tool 108 and the failure management tool 109 of FIG. 1 have been described, but the information processing apparatus 300 includes the change management tool 102, the build tool 104, and the test management of FIG. The tool 106 and the like may be further executed. The functional configurations of the change management tool 102, the build tool 104, the test management tool 106, and the like may be the same as those of general tools, and thus the description thereof is omitted here.

<Process flow>
Next, the flow of processing of the test method according to each embodiment will be described.

[First Embodiment]
FIG. 7 is a flowchart showing an example of processing of the information processing apparatus according to the first embodiment. This process shows an example of the process performed by the information processing apparatus 300 when the test control unit 401 executes a plurality of tests related to the software to be tested and there is a failed test.

In step S701, the first determination unit 402 of the information processing device 300 extracts the feature of the functional surface in the failed test. For example, the first determination unit 402 refers to the test specification 411 as shown in FIG. 5 and extracts the function or the like corresponding to the failed test. As an example, in the test specification 411 illustrated in FIG. 5, when the test item “boundary value test normal system” fails, the first determination unit 402 determines that the sub-function “corresponding to the failed test item” "A-1" and the like are extracted.

In step S702, the second determination unit 403 of the information processing device 300 extracts the characteristics of the phenomenon surface in the failed test. For example, the second determination unit 403 extracts a command in which an error has occurred in the failed test, the number of errors in the command, and the like.

In step S703, the first determination unit 402 determines whether or not the failed test has a functional feature. For example, when the number of failed test items among the one or more test items related to the function extracted in step S701 is equal to or larger than the preset first threshold value, the first determination unit 402 fails It is judged that the passed tests have functional characteristics.

If the rejected test has a functional feature, the information processing apparatus 300 moves the process to step S704. On the other hand, if the rejected test has no functional characteristics, the information processing apparatus 300 moves the process to step S705.

Upon proceeding to step S704, the second determination unit 403 determines whether or not the failed test has a phenomenological aspect. For example, when the number of errors generated by the command extracted in step S702 is equal to or greater than the preset second threshold, the second determination unit 403 determines that the failed test has a phenomenological aspect. to decide.

If the failed test does not have a phenomenon feature, the information processing apparatus 300 moves the process to step S705. On the other hand, if the failed test has a phenomenological feature, the information processing apparatus 300 moves the process to step S707.

When the process proceeds to step S705, the output unit 404 outputs a test result indicating the degradation of the specific function. For example, the output unit 404 outputs a failure vote or the like pointing out the degradation of the function determined to have a functional feature in step S703.

When the process proceeds from step S703 to step S706, the second determination unit 403 determines whether or not the failed test has a phenomenological aspect, for example, as in step S704.

If the failed test has a phenomenological feature, the information processing apparatus 300 moves the process to step S707. On the other hand, if the failed test has no phenomenological feature, the information processing apparatus 300 ends the process.

When the process proceeds to step S707, the output unit 404 of the information processing device 300 outputs the test result indicating the degradation of the specific procedure. For example, the output unit 404 outputs a trouble slip or the like pointing out the degradation of the command (an example of the specific procedure) extracted in step S702.

In this way, the output unit 404 of the information processing apparatus 300 outputs the test result regarding the failure causing the failure based on the characteristics of the phenomenon surface and the characteristics of the functional surface in the failed test.

As a result, according to the information processing system 100 according to the present embodiment, the priority order to be analyzed is presented to the designer 101 based on the functional characteristics and the phenomenon characteristics of the failed test. Therefore, the failure analysis by the designer 101 can be made more efficient.

[Second Embodiment]
FIG. 8 is a flowchart showing an example of processing of the information processing apparatus according to the second embodiment. This process shows another example of the process performed by the information processing apparatus 300 when the test control unit 401 executes a plurality of tests related to the software to be tested and there is a failed test. There is. Since the basic processing contents are the same as the processing of the information processing apparatus according to the first embodiment shown in FIG. 7, a detailed description of the same processing as the first embodiment will be omitted here.

In step S801, the first determination unit 402 of the information processing device 300 extracts the feature of the functional surface in the failed test. In addition, when a plurality of tests have failed, the first determination unit 402 extracts functional characteristics of each of the plurality of tests that have failed.

In step S802, the second determination unit 403 of the information processing device 300 extracts the characteristics of the phenomenon surface in the failed test. When a plurality of tests have failed, the second determination unit 403 extracts the phenomenological feature of each of the plurality of failed tests.

In step S803, the first determination unit 402 determines whether or not each of the failed tests has a functional feature.

Here, the information processing apparatus 300 executes the process of step S804 for the test determined to have the functional feature. Further, the information processing apparatus 300 executes the process of step S806 for the test determined to have no functional feature.

In step S804, the second determination unit 403 determines whether or not each of the tests that have been judged to have a functional feature out of the rejected tests has a phenomenological feature.

Here, the information processing apparatus 300 executes the process of step S805 for the test determined to have no phenomenon-side characteristic. Further, the information processing apparatus 300 executes the process of step S807 for the test determined to have the phenomenon-side characteristic.

In step S805, the output unit 404 of the information processing device 300 creates a failure vote (an example of a test result) that points out the degradation of the specific function. For example, the output unit 404 determines that the test item having the first threshold value or more fails for each of the tests determined in step S803 that there is a feature in the functional aspect and in step S804 that there is no feature in the phenomenon aspect. Create a trouble ticket indicating that there is a problem with the function that has become invalid.

On the other hand, in step S806, the second determination unit 403 determines whether or not each of the tests determined to have no functional feature among the rejected tests has a phenomenological feature. ..

Here, the information processing apparatus 300 executes the process of step S807 for the test determined to have the characteristic of the phenomenon side. Note that the information processing apparatus 300 moves the process to step S808 without performing any particular process for the test determined to have no phenomenon-side feature.

In step S807, the output unit 404 creates a trouble slip pointing out the degradation of the specific procedure.

For example, the output unit 404 determines that a command in which an error equal to or more than the second threshold value has occurred for each of the tests that is determined to have a functional feature in step S803 and is determined to have a phenomenon feature in step S804 (specific Create an obstacle slip that points out the degradation of (example of procedure).

In addition, the output unit 404 determines in step S<b>803 that there is no functional feature and in step S<b>804 that each of the tests is determined to have a phenomenon feature, a command in which an error equal to or greater than the second threshold value occurs (specific Create an obstacle slip that points out the degradation of (example of procedure).

In step S808, the output unit 404 outputs the failure forms created in steps S805 and S807 by grouping them according to the content of the failure. For example, the output unit 404 collectively outputs a plurality of failure votes determined to have a failure in a predetermined function into one group. Further, the output unit 404 collectively outputs a plurality of failure forms determined to have a failure in a predetermined phenomenon into one group.

In this way, the output unit 404 of the information processing apparatus 300 groups the test results (failure slips) of the failed tests according to the failure content based on the characteristics of the phenomenon side and the characteristics of the functional side. Output.

As a result, according to the information processing system 100 according to the present embodiment, the priority order to be analyzed is presented to the designer 101 based on the functional characteristics and the phenomenon characteristics of the failed test. Therefore, the failure analysis by the designer 101 can be made more efficient.

<Application example>
Subsequently, a specific application example of the software testing method according to the second embodiment will be described.

Here, as an example, the following description will be made on the assumption that the target device 202 is an image forming apparatus such as an MFP (Multifunction Peripheral) in which a scan function, a copy function, a printer function, a facsimile function and the like are mounted in one housing. To do.

However, the target device 202 is not limited to the image forming apparatus, and may be various devices that execute software to be tested by the information processing system 100. For example, the target device 202 is an output device such as a PJ (Projector: projector), an IWB (Interactive White Board: a white board having an electronic blackboard function capable of mutual communication), a digital signage, and a HUD (Head Up Display) device. You can have it. The target device 202 may be a device such as an industrial machine, an imaging device, a sound collector, a medical device, a network home appliance, a car (Connected Car), a digital camera, a wearable terminal, or the like. Further, the target device 202 may be a general-purpose information processing device such as a notebook PC, a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a digital camera, a wearable PC desktop PC, or the like.

For example, when considering a software test that realizes a scan function in an MFP, various scan functions include, for example, “scan to mail function”, “scan to folder function”, and “OCR (Optical Character Recognition) function”. There are various functions.

The “scan to mail function” also includes a plurality of sub-functions such as “scan condition setting” and “mail destination setting”. For example, when testing functional conformity, it is common to create test cases corresponding to these functional specifications.

As an example, when creating a test case of “scan condition setting” of “scan to mail function”, a method of testing whether the scan result is actually sent by mail under various scan conditions can be considered. At this time, what we want to test is whether or not the settings such as "double-sided designation", "color designation", "resolution designation" etc. as "scan conditions" work as set. In order to do so, there is a procedure to be executed in advance as a precondition. For example, the preconditions include "the mail transmission server is set", "the mail address of the transmission destination is registered", and the like.

In an automatic test environment, it is also necessary to automatically perform the process of adjusting these preconditions. Therefore, before executing the "scan to mail function" that is the original test target, for example, "Setting the mail sending server" The procedure may fail the test. In this example, the causal relationship between the precondition and the test target is relatively clear, but when designing and executing a large number of test cases, the test is executed in various steps, so the causal relationship is It is difficult to keep clear.

In such a case, each embodiment of the present invention is applied to design the test result indicating the priority order to be analyzed, considering both the functional feature and the phenomenon feature in the failed test. By presenting it to the person 101, the failure analysis can be made efficient.

For example, it is assumed that there are 100 test cases of "scan condition setting" of "scan to mail function", and all the tests passed in the previous build. In this build, it is assumed that 30 or more of the test cases have failed. In such a case, the information processing apparatus 300 may extract a possibility that the "scan condition setting" function has somehow degraded as a functional feature. Note that the above “30 cases” is an example, and the first threshold value for determining that there is a feature in terms of function can be set to any value.

On the other hand, it is assumed that the failures in these 30 test cases are tabulated and, for example, all 30 cases have an error in the "register address book" command. In such a case, it is conceivable that the information processing apparatus 300 may extract that the “address book registration” command may have caused some degradation as a phenomenological feature.

In addition, when the information processing device 300 can extract such functional features and phenomenon features, these information processing devices 300 group these 30 faults, and, for example, “suspicion of degradation of address book registration command”. To the designer. As a result, a more appropriate designer 101 (for example, the designer of the address book registration command) can be prompted to perform analysis, so that failure analysis can be made more efficient.

In the above example, the test is described as the function conformance test, but the test is not limited to this. For example, if the internal structure of the software is a test target, it is sufficient to extract the features of the target structural surface. For example, when a failure is detected in the state transition comprehensive test, it is conceivable to extract the failure rate shown in the comprehensive test as a feature.

As described above, according to the embodiments of the present invention, in the information processing system 100 that automatically tests software, when a plurality of tests fail due to the same failure, the failure that causes the failure Can be efficiently analyzed.

<Supplement>
Each function of each embodiment described above can be realized by one or a plurality of processing circuits. Here, the “processing circuit” in the present specification is a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, or designed to execute each function described above. In addition, devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), and a conventional circuit module are included.

100 Information processing system 300 Information processing device 401 Test control unit 402 First determination unit 403 Second determination unit 404 Output unit 405 Setting reception unit

JP, 2005-222108, A

Claims (10)

An information processing device for testing software to be tested,
A test controller that controls the execution of multiple tests for the software;
Among the plurality of tests, based on the characteristics of the phenomenon surface and the characteristics of the function in the failed test, an output unit that outputs a test result related to the failure that caused the failure,
An information processing apparatus having: The information processing apparatus according to claim 1, wherein the output unit divides the failed tests into groups based on the characteristics of the phenomenon surface and the characteristics of the functional surface, and outputs the test results. The plurality of tests includes one or more test items related to a predetermined function,
A first determination is made based on the number of failed test items among one or more test items related to the predetermined function, whether or not the failed test has the feature of the functional aspect. The information processing apparatus according to claim 1 or 2, further comprising: A second judgment unit for judging whether or not the failed test has a characteristic of the phenomenon surface based on the number of times of errors generated by predetermined commands used in the plurality of tests, The information processing apparatus according to claim 3. The output unit prioritizes the characteristic of the phenomenon surface and outputs the test result when the failed test has the characteristic of the phenomenon surface and the characteristic of the functional surface. Information processing equipment. The information according to claim 4 or 5, wherein the output unit outputs the test result related to the predetermined command when the failed test has the characteristic of the phenomenon surface and the characteristic of the functional surface. Processing equipment. The output unit is
In the test that has failed, there is a feature of the functional aspect, and if there is no feature of the phenomenon aspect, output the test result for the predetermined function,
If the test that has failed has the characteristic of the phenomenon side and does not have the characteristic of the functional side, the test result regarding the predetermined command is output.
The information processing device according to any one of claims 4 to 6. 8. The setting accepting unit for accepting a setting of a first threshold value for judging that the function side is characteristic, or a second threshold value for judging that the phenomenon side is characteristic. The information processing device according to 1. A test method for testing the software to be tested,
Computer
A process for controlling the execution of multiple tests on the software;
Of the plurality of tests, based on the characteristics of the phenomenon side and the characteristics of the function in the failed test, a process of outputting a test result related to the failure that caused the failure,
A test method to perform. On the computer that tests the software to be tested,
A process for controlling the execution of multiple tests on the software;
Of the plurality of tests, based on the characteristics of the phenomenon side and the characteristics of the function in the failed test, a process of outputting a test result related to the failure that caused the failure,
A test program that runs.

Images