JP2023074798A - Method for determining error handling in application test, method for testing application, and system for determining error handling in application test - Google Patents

Abstract

To provide a method for determining error handling in an application test, a method for the test, and a system for determining the error handling, which can determine appropriate handling for an error.SOLUTION: An application test is performed by accessing an application server that provides an application from a test terminal having a test tool and a processor. A method for determining error handling in this test includes causing the test tool to acquire error information, identifying a countermeasure using a correspondence table containing a plurality of error patterns and only one or a plurality of countermeasures corresponding to the respective error patterns, and determining the error handling. When only one countermeasure is identified, this single countermeasure is determined. When the plurality of countermeasures are identified, one countermeasure selected from the plurality of countermeasures is determined.SELECTED DRAWING: Figure 6

Description

The present invention relates to a method for determining error handling in application testing, an application testing method, and a system for determining error handling in application testing.

When a new application is provided on the web, or when an application is modified, tests are conducted to confirm whether various functions of the application operate correctly. Specifically, the operator who conducts the test accesses the application server that provides the application from the terminal, tries various operations, and obtains error information when an error is found. Such tests may use test tools to automate the tests.

When a test tool is introduced, an error may occur due to an error in the automation method, for example, a test script. US Pat. No. 6,200,000 discloses an apparatus and method for automatically correcting such test script errors.

JP 2019-101537 A

Causes of errors in operation tests include application bugs, errors in test scripts, and incomplete test data. Furthermore, it is said that there are many temporary errors caused by communication conditions. For example, if an error caused by a script occurs, script correction is required, but if it is a temporary error, correction is not required. Once all these errors have been dealt with, the test ends. Therefore, in order to test efficiently, it is necessary to determine an appropriate response according to the error.

The present disclosure provides a method for determining error handling in application testing, an application testing method, and a system for determining error handling in application testing, capable of determining an appropriate handling in response to an error. With the goal.

In a method for determining error handling in application testing according to an aspect of the present disclosure, the test is performed by accessing an application server that provides the application from a test terminal having a test tool and a processor, and the method includes: the test tool executed by the processor acquires error information about an error that occurred in the test; and the processor acquires a plurality of error patterns and one or more countermeasures corresponding to each of the error patterns. identifying one or more countermeasures corresponding to the error based on the error information using a correspondence table including is identified, and if multiple countermeasures are identified, one countermeasure selected from among the plurality of countermeasures is determined.

In the application testing method according to one aspect of the present disclosure, the test is performed by accessing an application server that provides the application from a test terminal having a test tool and a processor, and the method is executed by the processor. The test tool executes a primary test including a plurality of test cases, the test tool acquires error information regarding an error that occurred in the primary test for each test case, and the processor, Using a correspondence table containing a plurality of error patterns and one or more countermeasures corresponding to each error pattern, one or more countermeasures corresponding to the error are specified based on the error information. determining the error handling for each test case by the processor; if only one measure is specified, the only measure is determined; if multiple measures are specified, the measures are determined; determining one countermeasure selected from among a plurality of countermeasures; implementing the determined error handling; and performing the error handling on the test case executed by the processor. and the testing tool performing a secondary test.

A system for determining error handling in a test of an application according to one aspect of the present disclosure is configured to access an application server that provides the application by executing a processor and the processor to perform the test. a test tool configured to obtain error information about an error that occurred in the test, wherein the processor includes a plurality of error patterns and a unique error pattern corresponding to each error pattern. or using a correspondence table containing a plurality of countermeasures, identifying a single or a plurality of countermeasures corresponding to the error based on the error information, and determining the error countermeasures. and determining the error handling includes determining the single handling measure when a single handling measure is identified, and selecting the multiple handling measures when a plurality of handling measures are specified. determining on a selected one of the countermeasures.

According to the present invention, it is possible to determine an appropriate response to an error in application testing.

FIG. 1 is a diagram showing a schematic configuration of the system of this embodiment. FIG. 2 is a diagram showing an example of an error screen of a server error. FIG. 3 is a diagram showing an example of a correspondence table. FIG. 4 is a diagram showing an example of an error screen for system errors. FIG. 5 is a flow chart showing an application test method. FIG. 6 is a flow chart showing an error coping method in the test of FIG.

First, the embodiments of the present disclosure are listed and described.
[1] A method for determining error handling in an application test, wherein the test is performed by accessing an application server that provides the application from a test terminal having a test tool and a processor, the method comprising:
obtaining error information about errors that the test tool executed by the processor has encountered in the test;
By the processor, using a correspondence table containing a plurality of error patterns and a single or a plurality of countermeasures corresponding to each of the error patterns, a single or a plurality of countermeasures corresponding to the error based on the error information identifying measures;
Determining the error handling by the processor,
If only one remedy is identified, decide on that sole remedy;
determining a selected one of the plurality of countermeasures if multiple countermeasures are identified.

According to this configuration, when an error occurs in the test, it is possible to efficiently identify one or more countermeasures using the correspondence table. Also, when multiple countermeasures are identified, by selecting and implementing one of the multiple countermeasures, it is possible to determine an appropriate countermeasure according to the error in the application test. . The correspondence table may include a single countermeasure corresponding to the first error pattern and multiple countermeasures corresponding to the second error pattern. That is, at least one error pattern may have a plurality of countermeasures set.

In the present disclosure, "error" refers to an event in which an expected result cannot be obtained as a function or service provided by an application. Errors include, for example, the inability to access the application, the inability to complete the application's processing, the interruption of processing, or the output of unexpected results for input data or operations. included. Also, "error pattern" refers to each group classified based on the source of the error, the situation of occurrence, or the cause of the error. Furthermore, a "tool" is a kind of software, and particularly includes one or more programs for efficiently testing applications. For example, a test tool is a tool for automating or labor-saving application testing.

[2] The above [1], including acquiring, by the processor, the one countermeasure selected from the plurality of countermeasures by an operator operating the test terminal when the plurality of countermeasures are specified. The method described in .

According to this configuration, by specifying a plurality of countermeasures for each error pattern, the operator can efficiently select an appropriate countermeasure. In addition, by allowing the operator to select countermeasures himself/herself, even in cases that are difficult to automate, such as cases that are not reflected in the machine learning model, or cases that are difficult to discriminate with the machine learning model, appropriate measures can be taken. be able to. As a result, repetitive work and work for which countermeasures have been determined can be reduced by using a test tool or the like, and the operator can select countermeasures based on his/her judgment, thereby improving test accuracy.

[3] The test terminal comprises a countermeasure identification tool executed by the processor;
The method of [2] above, wherein the countermeasure identification tool is configured to identify the one or more countermeasures by searching a database having the correspondence table.

According to this configuration, it is possible to efficiently identify countermeasures against a plurality of error patterns by searching the database.
[4] the correspondence table includes one or more keywords corresponding to each of the error patterns;
The test terminal comprises a keyword selection tool configured to select the one or more keywords from the error information;
the keyword picker tool executed by the processor picking the one or more keywords from the error information;
the countermeasure identification tool executed by the processor searching for the one or more countermeasures based on the one or more keywords;
The method according to [3] above, further comprising

According to this configuration, it is possible to efficiently identify countermeasures for a plurality of error patterns by searching for countermeasures based on keywords. In "keyword selection", the selection is not limited to a single word, but also includes connected words such as phrases, clauses, or short sentences. That is, a "keyword" may be a "key phrase". In the present disclosure, "identifying countermeasures" may be "acquiring search results" or "outputting search results."

[5] the error information includes an error image;
the test terminal comprises a textual information extraction tool configured to extract textual information from the error image;
the textual information extraction tool executed by the processor extracting the textual information from the error image;
selecting the one or more keywords from the textual information by the keyword selection tool executed by the processor;
The method according to [4] above, further comprising

According to this configuration, the keyword can be extracted from the error image.
[6] the test is a primary test;
the test tool executed by the processor performs one or more retests without changing the test methodology after the primary test;
Determining the error handling based on the fact that there are cases in which the error information is acquired and cases in which the error information is not acquired in the primary test and the retest;
The method according to any one of [1] to [5] above, further comprising

According to this configuration, if an error occurs in either the primary test or the retest, it can be determined that the error is a temporary error. Therefore, by retesting, transient errors can be classified only with the test tool.

[7] In the primary test and the one or more retests, if there are cases where the error information is acquired and cases where the error information is not acquired, the error is generated by the processor as the error response. The method according to [7] above, including presenting contact information related to the page.

According to this configuration, it is possible to decide to present the contact information related to the page in which the error occurred in response to the temporary error.
[8] the test includes a plurality of test cases;
the error information includes an error report for each test case in which the error occurred;
The method according to any one of [1] to [7] above.

According to this configuration, since the error information includes an error report for each test case, appropriate error handling can be determined for each test case.
[9] The error is
an application error caused by the application;
a false positive error that is not an application error;
The method according to any one of [1] to [8] above, wherein the error correspondence is determined for the false positive error.

With this configuration, it is possible to efficiently classify false-positive errors and determine an appropriate response to the false-positive errors.
[10] The false positive error includes a test error due to a test method for performing the test,
The method according to [9] above, wherein the plurality of countermeasures are set for the error pattern related to the test error.

According to this configuration, when there are a plurality of countermeasures against test errors, one countermeasure selected from the plurality of countermeasures can be determined.
[11] A method of testing an application, wherein the testing is performed by accessing an application server that provides the application from a test terminal having a test tool and a processor, the method comprising:
the test tool executed by the processor executing a primary test including a plurality of test cases;
the test tool acquiring error information regarding an error that occurred in the primary test for each test case;
By the processor, using a correspondence table containing a plurality of error patterns and a single or a plurality of countermeasures corresponding to each of the error patterns, a single or a plurality of countermeasures corresponding to the error based on the error information to identify measures;
Determining error handling for each test case by the processor,
If only one remedy is identified, decide on that sole remedy;
If a plurality of countermeasures are specified, one countermeasure selected from among the plurality of countermeasures is determined;
implementing the determined error handling;
executing a secondary test by the test tool executed by the processor for the test case for which the error handling has been performed;
According to this configuration, when an error occurs in the test, it is possible to efficiently identify one or more countermeasures using the correspondence table. Also, when multiple countermeasures are identified, by selecting and implementing one of the multiple countermeasures, it is possible to determine an appropriate countermeasure according to the error in the application test. . If the secondary test is executed after the error countermeasures determined in this way are executed and no error occurs, it can be confirmed that the countermeasures were correct. Therefore, a test including multiple test cases can be efficiently executed.

[12] A system for determining error handling in testing an application, comprising:
a processor;
a test tool configured to perform the test by accessing an application server that provides the application by execution of the processor, the test tool being configured to obtain error information about an error that occurred in the test; a test tool and
with
The processor
Using a correspondence table containing a plurality of error patterns and one or more countermeasures corresponding to each error pattern, one or more countermeasures corresponding to the error are specified based on the error information. and
determining the error response; and
Determining the error handling includes:
deciding on a single remedy, if one is identified;
determining a selected one of the plurality of countermeasures when a plurality of countermeasures are identified.

According to this configuration, the test can be automated by the test tool. In addition, when an error occurs in the test, the correspondence table can be used to efficiently identify one or more countermeasures. Then, when multiple countermeasures are specified, by selecting and implementing one countermeasure from the multiple countermeasures, it is possible to determine an appropriate countermeasure according to the error in the application test. .

[13] The system includes:
the application server that provides the application;
a test terminal operated by an operator;
The system according to [12] above, wherein the test terminal includes a storage in which the test tool is stored, and the processor.

According to this configuration, the test can be performed by accessing the application server from the test terminal.
[Details of the embodiment of the present disclosure]
An example of a method for determining error handling in the test of the application of the present disclosure and a system for determining the error handling will be described below with reference to the drawings. The present invention is not limited to these examples, but is indicated by the scope of the claims, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

As shown in FIG. 1, a communication system 11 is a system for determining error handling in testing applications (especially web applications). Communication system 11 includes, for example, one or more application servers 20 and one or more test terminals 30 . Communication system 11 may further include one or more operator terminals (not shown) operated by operator 13 . The operator terminal may be, for example, a PC, smart phone, or tablet.

The test terminal 30 may be directly operated by the operator 13 who tests the application, or may be indirectly operated by the operator 13 via an operator terminal directly operated by the operator 13 . The operator terminal may be connected to the test terminal 30 by a LAN (Local Area Network) or other wired or wireless communication method, or may be connected via the network 12 . In the following description, a case where the operator 13 directly operates the test terminal 30 will be exemplified.

Although one application server 20 and one test terminal 30 are shown in FIG. 1, the communication system 11 may include a plurality of application servers 20 to be tested, and may include a plurality of test terminals 30. . Also, the system for determining error handling in application testing may not include the application server 20 .

Communication system 11 may also include at least one of one or more database servers 14 and one or more machine learning devices 60 . The basic configuration of the database server 14 , operator terminal, and machine learning device 60 may be similar to that of the application server 20 or test terminal 30 . Database server 14 may store database 15 including correspondence table 16 in its storage. Instead of the database server 14, the test terminal 30 or the operator terminal may have the database 15, in which case the database server 14 can be omitted. Also, the test terminal 30 or the operator terminal may operate as the machine learning device 60, in which case the machine learning device 60 can be omitted.

The database 15 may be created using machine learning models. The machine learning model may be created by machine learning device 60 . The correspondence table 16 may include multiple error patterns and one or more countermeasures corresponding to each error pattern. The correspondence table 16 may contain one or more keywords corresponding to each error pattern (see FIG. 3).

The application server 20 provides a site (website or web page) on the network 12 that works with the application. Examples of websites include, but are not limited to, information providing sites, e-commerce sites, reservation sites, SNS (Social networking service) sites, and the like. The network 12 is, for example, the Internet, a mobile communication network and its radio base stations. Test terminal 30 is configured to access application server 20 , database server 14 and machine learning device 60 via network 12 .

[Application Server]
The application server 20 (hereinafter simply referred to as "server 20") may be a workstation or a general-purpose computer such as a PC (Personal Computer). The server 20 may have a processor 21 , a communication IF (interface) 22 , an input/output IF (interface) 23 , a memory 24 and a storage 25 . These multiple components of server 20 are connected to each other by communication bus 29 .

The processor 21 is, for example, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), MPU (Micro Processor Unit), FPGA (Field-Programmable Gate Array), or other arithmetic device. Processor 21 executes a series of instructions contained in a program stored in memory 24 or storage 25 in response to a signal given to server 20 or in response to establishment of a predetermined condition.

The processor 21 is, for example, a processing circuit configured to execute various software processes. The processing circuitry may comprise dedicated hardware circuitry (eg, an ASIC, etc.) that handles at least part of the software processing. That is, software processing may be performed by processing circuitry comprising one or more software processing circuits and/or one or more dedicated hardware circuits.

Communication IF 22 is configured to connect to network 12 . The communication IF 22 is configured to communicate with other devices connected to the network 12 , such as the test terminal 30 . The communication IF 22 is implemented as, for example, a LAN or other wired communication IF. The communication IF 22 can be implemented as, for example, Wi-Fi (registered trademark), Bluetooth (registered trademark), or other wireless communication IFs, but is not limited to these.

The input/output IF 23 is configured to communicate with an external input/output device. For example, the input/output IF 23 is implemented as a USB (Universal Serial Bus), DVI (Digital Visual Interface), HDMI (registered trademark) (High-Definition Multimedia Interface), or other wired communication IF. For example, the input/output IF 23 can be implemented as Bluetooth or other wireless communication IF, but is not limited to these.

The memory 24 is, for example, RAM (Random Access Memory) or other volatile memory. Memory 24 is configured to temporarily store programs and data. A program is read from the storage 25, for example. The data stored by memory 24 may include data received by server 20 and data generated by processor 21 .

The storage 25 may be, for example, a ROM (Read-Only Memory), hard disk drive, flash memory, or other non-volatile storage device. The storage 25 may be a removable storage device such as a memory card. The server 20 may execute programs stored in an external storage device instead of the storage 25 . Storage 25 is configured to permanently store programs and data.

The programs stored in storage 25 include web scripts 26 for displaying web pages. The web script 26 may be, for example, structured document data such as HTML (Hyper Text Markup Language) or XHTML (Extensible Hyper Text Markup Language). The storage 25 may store error display data 27 to be displayed when an error occurs. The error display data 27 may include at least one of character data and image data.

[Test terminal]
The test terminal 30 may be, for example, a PC, smart phone, or tablet, or may be a test server that accepts access from one or more operator terminals. The test terminal 30 may have a processor 31 , communication IF 32 , input/output IF 33 , memory 34 , storage 35 , input device 36 and display 37 . These multiple components of test terminal 30 are connected to each other by communication bus 39 . At least one of the storage 35 , the input device 36 and the display 37 may be an external device detachable from the test terminal 30 .

The input device 36 is, for example, a keyboard, a mouse, or the like, includes buttons, keys, switches, or a touch pad, and is configured to receive operations by the operator 13 . The operations of the operator 13 include, for example, designation of buttons displayed on the web page, input of characters in input fields, and the like. Examples of buttons displayed on web pages include buttons for viewing information (including product information), buttons for searching information, and buttons for purchasing products.

The display 37 may be, for example, a liquid crystal monitor or an organic EL (Electro Luminescence) monitor, or a touch screen having a touch panel that also serves as the input device 36 . The display 37 displays various screens. For example, a screen prepared in advance, a screen of a web page, or a test result is displayed on the display 37 .

Processor 31 is, for example, a CPU, GPU, MPU, FPGA, or other arithmetic device. Processor 31 executes a series of instructions contained in a program stored in memory 24 or storage 25 in response to a signal given to server 20 or in response to establishment of a predetermined condition.

The processor 31 is a processing circuit configured to execute various software processes. The processing circuitry may comprise dedicated hardware circuitry (eg, an ASIC, etc.) that handles at least part of the software processing. That is, software processing may be performed by processing circuitry comprising one or more software processing circuitry and/or one or more dedicated hardware circuitry.

Communication IF 32 is configured to connect to network 12 . Communication IF 32 is configured to communicate with other devices connected to network 12 , such as server 20 , database server 14 and machine learning device 60 . The communication IF 32 is implemented as, for example, a LAN or other wired communication IF. The communication IF 32 can be implemented as, for example, Wi-Fi, Bluetooth, or other wireless communication IFs, but is not limited to these.

The input/output IF 33 is configured to communicate with an external input/output device. For example, the input/output IF 33 is implemented as USB, DVI, HDMI, or other wired communication IF. For example, the input/output IF 33 may be implemented as Bluetooth or other wireless communication IF, but is not limited to these.

Memory 34 is, for example, RAM or other volatile memory. Memory 34 is configured to temporarily store programs and data. The program is read from the storage 35, for example. The data stored by memory 34 may include data received from server 20 and data generated by processor 31 .

Storage 35 may be, for example, a ROM, hard disk drive, flash memory, or other non-volatile storage device. The storage 35 may be a removable storage device such as a memory card. Instead of the storage 35, the test terminal 30 may execute a program stored in an external storage device. Storage 35 is configured to permanently store programs and data.

The programs stored in the storage 35 include, for example, an OS (operating system) 51, a web browser 52, a test tool 53, a countermeasure identification tool 54, a keyword selection tool 55, a character information extraction tool 56, and an image determination tool 57. It's okay. These tools 53 - 57 are executed by processor 31 . The storage 35 may store test data 58 to be tested and result data 59 that is the result of the test. The database 15 may be stored in the storage 35 .

The test tool 53 is configured to access and test the server 20 that provides the application. More specifically, the test tool 53 inputs prepared test data to the application, obtains results output by the application, and saves, records, or outputs the results. Moreover, the test tool 53 may be configured to acquire error information regarding an error that occurred in the test as the result data 59 as a result of the test. For example, the test tool 53 displays the final screen (error screen) information, such as a screenshot of the screen. The workaround identification tool 54 may be configured to identify one or more workarounds for the error based on the error information.

The keyword picker 55 may be configured to pick keywords for the error from the error information. Countermeasure identification tool 54 may be configured to search for countermeasures based on one or more keywords selected by keyword selection tool 55 . The textual information extraction tool 56 may be configured to extract textual information from the error image. The character information extraction tool 56 may be a so-called OCR (Optical Character Reader) tool that recognizes characters included in image data and converts the recognized characters into electronic text. The keywords selected by the keyword selection tool 55 and the textual information extracted by the textual information extraction tool 56 may be used for retrieval after being examined or corrected by the operator 13 .

After image processing or feature extraction of the error image as necessary, the image determination tool 57 identifies the error pattern based on pre-registered or learned information (for example, image data and a label indicating what it is). may be configured to determine. The image determination tool 57 may utilize machine learning models. The machine learning model may be created by machine learning device 60 . The machine learning model may use error images as input data and output error patterns or countermeasures for the error images. The image determination tool 57 can classify error screens that do not contain character information. The image determination tool 57 may be used to determine, for example, application errors, error images that do not contain text, or frequent transient errors.

[Machine learning device]
The machine learning device 60 may have a processor 61 , communication IF 62 , input/output IF 63 , memory 64 and storage 65 . These multiple components of the machine learning device 60 are connected to each other by a communication bus 69 . The basic configuration of these multiple components may be the same as that of the server 20 or test terminal 30 .

Processor 61 may comprise a variety of processors 61 designed for machine learning. The processor 61 is configured to perform functions related to machine learning, such as model generalization, by executing a machine learning program 66 recorded in the storage 65 . A deep learning algorithm, for example, can be used to generate the machine learning model. Machine learning device 60 may receive training data 67 from test terminal 30, for example. The machine learning device 60 may construct one or more learned models 68 by performing machine learning based on the teacher data 67 .

[About application testing]
Test tool 53 is an example of software for giving instructions to server 20 . The test tool 53 is, for example, a script that pre-records a scenario of operations such as designation of buttons displayed on a web page, that is, a series of procedures, and sends commands according to this scenario to the server 20 . The test tool 53 reproduces the scenario using the test data 58 to test a plurality of test cases continuously or in parallel. The test is, for example, an operation test. A test performed first by the test tool 53 is called a primary test.

When testing, the test terminal 30 first accesses the server 20 via the communication IF 32 and network 12 . In such access, a connection establishment process (for example, handshake, etc.) is performed to establish a connection between the test terminal 30 and the server 20, and after the connection is established, the test tool 53 performs a series of test cases, that is, a plurality of test cases. test is performed.

In this specification, it is assumed that "access" continues from the start of connection establishment processing until the end of test processing (or until progress becomes impossible due to an error in the middle of these processing). However, "access" refers to the period from the start to the end of the connection establishment process (or until progress becomes impossible in the middle of this process) and from the start to the end of the test process (or progress due to an error in the middle of this process). until it becomes impossible).

When some error occurs in each test case, the test tool 53 acquires error information as a test result. A primary test may collectively execute multiple test cases in a single access. A test result of a case in which an error occurs is called an error report. The error information may contain multiple error reports. The test results may be acquired only for cases in which errors occur, or may be acquired for cases in which no errors occur. If there were no errors in the test, the fact that there were no error reports may be the result of the test.

The error report may be, for example, an error message output from the console of the OS 51 or test tool 53, a screenshot of an error screen, or both. The error screen may be generated based on the error display data 27 that the server 20 has, or may be a screen on which nothing is displayed. A screenshot of an error screen is called an error image. The error image may include at least one of text and images. The test tool 53 stores the acquired error report in the storage 35 as result data 59 .

[About error causes and countermeasures]
Errors in testing can be caused by a variety of factors. Error factors include application script bugs and test method errors. Errors caused by the test method include, for example, script bugs in the test tool 53, flaws in the test data 58, and flaws in the scenario. Errors found in testing require different responses depending on their cause.

For example, in the case of a bug in an application script, the operator 13 reports the content of the error to the application provider as error handling. These errors that originate in the application itself are the true errors that the test should find. A true error caused by an application is called an "application error".

Errors may include not only true errors, but also pseudo-errors that are not caused by the application. Such pseudo errors are called "false positive errors". These false positive errors account for more than true errors in the error reports obtained in tests, sometimes as high as 70% or more. Therefore, in testing, while streamlining the work by introducing automation tools, it is a challenge to properly distinguish between such false-positive errors and true errors.

Among the false-positive errors, the most common are errors caused by temporary unstable communication conditions. Such errors are called "transient errors". Temporary errors cannot be resolved by either the application provider or the test operator 13 . Therefore, as a response to the temporary error, the operator 13 may be presented with the contact information for the page based on the URL of the page where the error occurred or the link information described on the page.

FIG. 2 shows an example of an error screen when an error indicating that some problem has occurred in the server 20 (for example, 500 error) occurs as an example of a temporary error. Other temporary errors include, for example, an error indicating that the network 12 cannot be connected, an error indicating that the number of simultaneous accesses to the server 20 has exceeded the limit (eg, 503 error), and an error indicating that the server 20 is under maintenance. contains an error stating that

Errors due to test methods are included in false positive errors because they are secondary errors caused by test automation. An error caused by such a test method is called a "test error". As a response to the test error, the operator 13 modifies the test tool 53 or test data 58, or changes the scenario to avoid the error.

[Regarding the identification of countermeasures using the database]
As shown in FIG. 3, the correspondence table 16 includes a correspondence table 16 of multiple error patterns and single or multiple countermeasures corresponding to each error pattern. The correspondence table 16 may include keywords for errors as error information. The correspondence table 16 may contain information about the cause of the error. The countermeasures contained in database 15 can be set based on past error cases. A plurality of countermeasures are set for one or more of the error patterns. Although the correspondence table 16 may include countermeasures for both true errors and false positive errors, FIG. 3 shows only false positive error information.

A workaround identification tool 54 is configured to search the database 15 based on the error report and identify a corresponding workaround. Further, the countermeasure identification tool 54 lists the identified countermeasures for each test case, stores the list in the storage 35, displays it on the display 37, sends it to another terminal, or prints it. You can also output to a printer as Informing the operator 13 by storing, displaying, transmitting, or outputting countermeasures in this manner is referred to as "presenting countermeasures." Such countermeasures can be automatically identified by a countermeasure identification tool 54 programmed as an application. The operator 13 can confirm the countermeasure specified in this way for each error.

In order to efficiently search for countermeasures, the database 15 may be configured to be searchable with keywords. In this case, the correspondence table 16 may be created so that one or more keywords correspond to one or more countermeasures. Keyword picker 55 may be configured to pick keywords from error reports. If the error report includes an error image, textual information extraction tool 56 may extract textual information from the error image. Error reports or keywords not registered in the database 15 may be registered in the database 15 .

As shown in FIG. 3, one or a plurality of countermeasures against false positive errors may be set for one error. Examples of errors for which there is only one workaround are, for example, correcting the mistake for a script mistake or displaying contact information for a temporary error.

For example, an error report on a script might capture an error image of the browser's "restore" popup overlaying the site's button. In this case, for the keyword "Restore", the only workaround to avoid the error may be to close the "Restore" popup.

For example, as shown in FIG. 4, a "system error" error message may be obtained as a system-related error report and the page may not be displayed. In this case, the keyword "system error" can be picked from the error message. As an error report related to another system, you may get a "login error" error message and the page after login is not displayed. In this case, the keyword "login error" can be selected from the error message. Errors related to the system are transient errors, so the only remedy may be to provide contact information for the error.

Next, an example of a case where there are multiple countermeasures is shown. For example, the error report for the script shows an error message "Action not supported" and the click button does not perform the click. In this case, for the keywords "button" and "action not supported", the first countermeasure "create a button" and the second countermeasure "execute the click" are available. may be specified. According to the first workaround, the created button can be used to proceed with the test interrupted by the error. Also, according to the second countermeasure, the test can be advanced similarly by executing a click.

Also, error reports related to data may show values that differ from the expected values. For example, after testing to make a reservation with a user's retention points, "1,800 points" may be displayed when "2,000 points" should be displayed as the user's retention points. Such errors are considered to be problems in the test data, so correcting this point or canceling the operation related to the point will avoid the error. Therefore, in this case, for the keyword "points", the first countermeasure "add points to the user" and the second countermeasure "cancel the reservation that was made with points" are specified. may

Similar error reports related to data can result in different expected and actual behavior, and different expected and actual results. Thus, errors in not getting expected values, behaviors, or results can be caused by a variety of factors, such as test tool 53 scripts, test data, or scenarios. Therefore, there are many countermeasures for avoiding errors, and it takes a lot of effort to set all the countermeasures for such combinations in advance.

Machine learning models may be used to efficiently set appropriate countermeasures for such errors. The machine learning model may be created by machine learning device 60 . The machine learning model may output error countermeasures using error reports, error images, or keywords related to errors as input data. Using such a learning model eliminates the need to individually set countermeasures for a wide variety of error patterns.

Application Test
An application test method will be described with reference to FIG. The test is performed by a test terminal 30 operated directly or indirectly by the operator 13 . More specifically, the test tool 53 executes the primary test (step S11), and the test tool 53 obtains the results of the primary test (step S12). If no error has occurred in the primary test (step S13: NO), the test ends. If an error has occurred in the primary test (step S13: YES), in step S12, the test tool 53 has acquired error information regarding the error that occurred in the test.

If an error occurs in the primary test, the operator 13 handles the error (step S14). After handling the error, the test tool 53 executes the secondary test for at least the test case in which an error occurred in the primary test (step S15). Then, the test tool 53 acquires the result of the secondary test (step S17). If no error has occurred in the secondary test (step S17: NO), the test ends.

If an error occurs in the secondary test (step S17: YES), the process returns to step S14 and the operator 13 handles the error again. After that, the test tool 53 executes step S15 as a tertiary test for at least the test cases in which an error occurred in the secondary test. Then, the test tool 53 acquires the result of the tertiary test as step S16. In this way, error handling and testing are repeated, and the test ends when there are no more errors or when there are no more unknown errors.

The operator 13 may classify the temporary error from the other errors among the primary errors acquired in step S12 before taking error handling in step S14. For example, image judgment tool 57 may classify transient errors. Alternatively, machine learning models may be utilized to classify transient errors from other errors.

The machine learning model may be created by machine learning device 60 . A machine learning model may take an error image as input data and output whether or not the error is classified as a transient error. In this case, a trained model is created using an error report that is known to be a temporary error as training data. Since the image determination tool 57 and the learning model can recognize error images as they are and use them for classification, there is no need to extract character information from images or refine the extracted character information.

Alternatively, after the primary test, the test tool 53 may perform one or more retests without changing the testing methodology to classify transient errors as other errors. In this case, error handling can be determined based on the fact that there are cases where error information is acquired and cases where error information is not acquired between the primary test and the retest. That is, even if an error report is obtained in the primary test, if the error does not occur after retesting after some time, it can be determined that the error that occurred in the primary test was a temporary error. For example, as an error response to a temporary error, it may be decided to display the contact information associated with the page on which the error occurred. Retesting has the advantage that the test tool 53 alone can classify transient errors.

Temporary errors account for a large proportion of all errors, so classifying them prior to error handling can greatly reduce the amount of subsequent work. For example, labor for extracting character information from error information, selecting keywords, and performing image processing can be saved. A test case in which an error occurs even after one retest may be repeatedly retested. Then, if there is a case in which an error does not occur even once among multiple test cases, it may be determined that the error is a temporary error even if an error occurs in other cases.

Similarly, the operator 13 may classify the application error from the other errors based on the primary error acquired in step S12 before taking error handling in step S14. For example, operator 13 may utilize image judgment tools 57 to classify application errors. Then, as an error response to the application error, it may be decided to report the content of the error to the application provider. Thus, by first classifying errors other than test errors, such as temporary errors and application errors, it is possible to concentrate on handling test errors in the subsequent steps.

[Error Handling]
Error handling in step S14 will be described with reference to FIG. When the error information is obtained in step S12, the error information is processed (step S21). The processing of error information includes, for example, a process in which the text information extraction tool 56 extracts text information from an error image (error information), and a keyword selection tool 55 selects a keyword from text information (error information) including an error message. processing.

Subsequently, the countermeasure identification tool 54 searches the database 15 using the keyword (step S22) and obtains search results. If no countermeasure is found as a result of the search (step S23: NO), the operator 13 individually decides how to deal with the error (step S24). If there is a countermeasure as a result of the search (step S23: YES), the countermeasure identification tool 54 obtains the search result and identifies one or more countermeasures for each error based on the error information. (step S25).

If the correspondence table 16 does not include countermeasures for application errors, the operator 13 may determine the countermeasures for all application errors in step S24. That is, as a result of classifying many pseudo errors using the correspondence table 16, application errors are included in the errors for which there was no countermeasure in step 23. FIG.

If only one countermeasure is identified in step S25 (step S26: NO), it is determined that the only countermeasure will be used. In this case, processor 31 will obtain only one countermeasure in step S25. For example, if it is an application error, the only countermeasure is to report the error content to the application provider, and if it is a temporary error such as a system error, the only countermeasure is to display the contact information. is. In addition, correction of the test script or test data 58 may be the only countermeasure for test errors.

If a plurality of countermeasures are specified in step S25 (step S26: YES), one countermeasure is selected from the plurality of countermeasures (step S27). For example, the operator 13 may select one countermeasure from a plurality of countermeasures. In this case, the test itself is automated by the test tool, but the task of selecting one of the multiple countermeasures is manually performed by the operator 13 after the primary test while making individual judgments. By selecting countermeasures in this way, error countermeasures are determined. In this case, the processor 31 acquires one countermeasure selected from among the plurality of countermeasures in step S27.

Instead of the operator 13 selecting one countermeasure, the countermeasure may be automatically selected by a tool or the like of the test terminal 30, for example. For example, machine learning models may be used to select appropriate countermeasures depending on the situation. The machine learning model may be created by machine learning device 60 . The machine learning model may take error patterns or keywords as input data and output a unique countermeasure.

When the error handling is determined (obtained) for some or all of the errors through steps S24, S25, and S27, the error handling is presented (step S28). More specifically, processor 31 displays a list of countermeasures on display 37, for example. After that, the operator 13 takes the determined action (S28). For example, the operator 13 modifies the script of the test tool 53, the test data 58, or the scenario. Then, when the countermeasure is implemented, the secondary test of step S15 is executed.

Even if a retest is performed to classify the temporary error from other errors, the communication state may not be improved during that time, and the error may continue to occur. Even in such a case, if the temporary error is added to the correspondence table 16, the temporary error can be determined based on the error information. Similarly, even if an application error is pre-classified from other errors after the primary test, adding the application error to the correspondence table 16 can reduce the oversight of the application error.

[Action of the present disclosure]
By introducing the test tool 53, multiple test cases can be efficiently tested. For example, the test tool 53 executes a primary test at night and collectively acquires the results of a plurality of test cases. Then, on the next day, the operator 13 can check the test results and take error measures as necessary. In this way, in the present disclosure, by combining a process that can be automated and work that requires judgment by the operator 13, the efficiency of the test is improved.

However, the introduction of the test tool 53 causes secondary test errors. In addition, such test errors diversify error patterns. According to the method or system of the present disclosure, by using the correspondence table 16, it is possible to identify appropriate countermeasures for various error patterns.

Especially since a test script can take multiple approaches to achieve the same result, there can be multiple ways of avoiding or correcting test errors. Therefore, even if countermeasures against test errors can be patterned to some extent and determined, the work is time-consuming. In addition, even if a machine learning model is used for such patterning, it is necessary to continuously improve the model in response to new cases, and some cases are difficult to classify depending on the model.

Therefore, by setting one countermeasure for an error that can easily be patterned, the countermeasure can be easily determined by referring to the correspondence table 16 . On the other hand, for an error for which multiple countermeasures are assumed, the operator 13 can select an appropriate countermeasure individually. As a result, it is possible to save labor in testing by using various tools for repetitive work and work for which correspondence has been determined. The operator 13 makes judgments that are difficult to automate, so that errors can be handled with high accuracy.

Since handling of application errors is determined by reporting to the application provider, error handling is mainly determined for false-positive errors. Furthermore, since there is no substantial response to temporary errors among false positive errors, it can be said that error response is determined with respect to test errors. In other words, according to the present disclosure, it is possible to more efficiently decide how to deal with test errors that occur particularly with test automation.

Thus, the correspondence table 16 may include a single countermeasure corresponding to transient errors and application errors (first error pattern) and multiple countermeasures corresponding to test errors (second error pattern). Furthermore, only one countermeasure may be set for some error patterns of test errors, and a plurality of countermeasures may be set for error patterns of other test errors.

[Effect of the present disclosure]
According to the present disclosure, the following effects can be obtained.
(1) When an error occurs in a test, the correspondence table 16 can be used to efficiently identify one or more countermeasures. Also, when multiple countermeasures are identified, by selecting and implementing one of the multiple countermeasures, it is possible to determine an appropriate countermeasure according to the error in the application test. .

(2) By specifying a plurality of countermeasures for each error pattern, the operator 13 can efficiently select an appropriate countermeasure.
(3) By searching the database 15 with the countermeasure identification tool 54, it is possible to efficiently identify countermeasures for a plurality of error patterns.

(4) By searching for countermeasures based on keywords, it is possible to efficiently identify countermeasures for multiple error patterns.
(5) Keywords can be extracted from the error image by the text information extraction tool 56 and the keyword selection tool 55 .

(6) If an error occurs in either the primary test or the retest, it can be determined that the error is a temporary error. Therefore, by re-testing, the temporary error can be classified by the test tool 53 alone.

(7) If a temporary error can be classified from other errors, it is possible to determine the error response of presenting the contact information for the page on which the error occurred.
(8) Since error information includes an error report for each test case, appropriate error handling can be determined for each test case.

(9) False positive errors can be efficiently classified to determine appropriate responses to false positive errors.
(10) When there are a plurality of countermeasures against test errors, one countermeasure selected from the plurality of countermeasures can be determined.

(11) If the secondary test is executed after the determined error countermeasure is implemented and no error occurs, it can be confirmed that the countermeasure was correct. Therefore, a test including multiple test cases can be efficiently executed.

(12) By combining tools with different functions such as the test tool 53, the countermeasure identification tool 54, the keyword selection tool 55, and the character information extraction tool 56, multiple processes related to error handling are efficiently automated. can be used, and some steps can be performed manually if desired. As a result, it is possible to flexibly change the error handling method even when the test content or test method is changed.

This embodiment can be implemented with the following modifications. This embodiment and the following modified examples can be implemented in combination with each other within a technically consistent range.
For some or all of the errors, instead of using the countermeasure identification tool 54, the operator 13 may manually perform a keyword search using the database 15 to acquire countermeasures. In this case, the processor 21 operates as an identification unit that identifies countermeasures. Further, such manual search and automatic search by the countermeasure identification tool 54 may be used together.

- Instead of using the keyword selection tool 55, the operator 13 may manually select keywords from the error information. In this case, the processor 21 operates as a selection unit that selects keywords. Further, such manual selection and automatic selection by the keyword selection tool 55 may be used together. For example, the operator may select only specific cases, or the operator 13 may examine the keywords automatically selected by the keyword selection tool 55 .

- Instead of the text information extraction tool 56, the operator 13 may manually extract text information or keywords from the error image using known OCR software. In this case, known OCR software operates as a tool for selecting character information. Moreover, such manual extraction and automatic extraction by the character information extraction tool 56 may be used together. For example, the operator may extract only specific cases, or the operator 13 may examine the character information automatically extracted by the character information extraction tool 56 .

- Application testing can also be performed by accessing the server 20 within the LAN, for example, by the test terminal 30 within the same LAN, instead of via the web network.
・The primary test execution, primary test result (error information) acquisition, character extraction (OCR), keyword selection, database 15 search, and countermeasure identification may be performed continuously, that is, automatically, by linking various tools. . In other words, the test terminal 30 may include application software that links the test tool 53 , countermeasure identification tool 54 , keyword selection tool 55 and text information extraction tool 56 . Alternatively, after each step, the operator 13 may give instructions to proceed to the next step, or only some steps may be performed consecutively.

・Acquisition of primary test results (error information), character extraction (OCR), keyword selection, database 15 search, identification of countermeasures, implementation of countermeasures, execution of secondary tests, acquisition of secondary test results (error information), etc. A series of such steps may be performed continuously, that is, automatically, by cooperation of various tools. In other words, the test terminal 30 may include application software that links various tools such as the test tool 53, the countermeasure identification tool 54, the keyword selection tool 55, and the character information extraction tool 56. In this case, if there are multiple countermeasures, a machine learning model may be used to select the countermeasure. Alternatively, only when there is only one countermeasure, a series of steps related to the test are performed sequentially (automatically), and in cases where there are multiple countermeasures and there are no search results, a countermeasure is selected. may be performed by the operator 13.

- The operator terminal has at least one of various tools such as the test tool 53, the countermeasure identification tool 54, the keyword selection tool 55, and the character information extraction tool 56, and the test terminal 30 has the remaining tools. good too. Alternatively, various tools may be distributed and stored in three or more terminals including the test terminal 30 .

- When the test tool 53 inputs the test data of the first test case to the application and acquires the error information, the countermeasure identification tool 54 may identify countermeasures for the first error. The test tool 53 inputs the test data of the second test case to the application while or after the workaround identification tool 54 identifies the workaround for the first test case (first error). good too. Thus, in each test case, when identifying a workaround following the test, the test tool 53 may, as a result of the primary test, identify the error information and the single or multiple workarounds identified based on the error information. may be output.

This disclosure includes the following example implementations.
[Example 1]
A method for handling errors in testing an application, comprising:
The errors include application errors caused by the application, transient errors, and test errors caused by the testing method, the method comprising:
running a primary test that includes multiple test cases;
obtaining an error report for each of the test cases regarding an error that occurred in the primary test;
classifying the transient error from other errors, e.g. by retesting;
selecting one or more keywords from each of the error reports;
Using a correspondence table containing a plurality of error patterns and one or more countermeasures corresponding to each error pattern, identifying one or more countermeasures corresponding to each test error based on the keyword and
Selecting one countermeasure from the plurality of countermeasures when a plurality of countermeasures are identified;
How to handle errors in testing your application, including

[Example 2]
A method for determining error handling in testing an application, wherein the testing is performed by accessing an application server that provides the application from a test terminal having a test tool and a processor, the method comprising:
obtaining error information about errors that the test tool executed by the processor has encountered in the test;
By the processor, using a correspondence table containing a plurality of error patterns and a single or a plurality of countermeasures corresponding to each of the error patterns, a single or a plurality of countermeasures corresponding to the error based on the error information identifying measures;
A method, including

REFERENCE SIGNS LIST 11 communication system 12 network 13 operator 14 database server 15 database 16 correspondence table 20 application server 21 processor 22 communication IF 23 input/output IF 24 memory , 25... Storage, 26... Web script, 27... Error display data, 29... Communication bus, 30... Test terminal, 31... Processor, 32... Communication IF, 33... Input/output IF, 34... Memory, 35... Storage, 36... Input device, 37... Display, 39... Communication bus, 52... Web browser, 53... Test tool, 54... Countermeasure identification tool, 55... Keyword selection tool, 56... Character information extraction tool, 57... Image determination tool, 58... Test data, 59... Result data, 60... Machine learning device, 61... Processor, 62... Communication IF, 63... Input/output IF, 64... Memory, 65... Storage, 66... Machine learning program, 67... Teacher data, 68... Learned model, 69... Communication bus.

Claims (13)

A method for determining error handling in testing an application, wherein the testing is performed by accessing an application server that provides the application from a test terminal having a test tool and a processor, the method comprising:
obtaining error information about errors that the test tool executed by the processor has encountered in the test;
By the processor, using a correspondence table containing a plurality of error patterns and a single or a plurality of countermeasures corresponding to each of the error patterns, a single or a plurality of countermeasures corresponding to the error based on the error information identifying measures;
Determining the error handling by the processor,
If only one remedy is identified, decide on that sole remedy;
determining a selected one of the plurality of countermeasures if multiple countermeasures are identified. When the plurality of countermeasures are identified, an operator who operates the test terminal acquires the one countermeasure selected from the plurality of countermeasures by the processor,
The method of claim 1. the test terminal comprises a countermeasure identification tool executed by the processor;
the countermeasure identification tool is configured to identify the one or more countermeasures by searching a database having the correspondence table;
3. The method of claim 2. the correspondence table includes one or more keywords corresponding to each of the error patterns;
The test terminal comprises a keyword selection tool configured to select the one or more keywords from the error information;
the keyword picker tool executed by the processor picking the one or more keywords from the error information;
the countermeasure identification tool executed by the processor searching for the one or more countermeasures based on the one or more keywords;
4. The method of claim 3, further comprising: the error information includes an error image;
the test terminal comprises a textual information extraction tool configured to extract textual information from the error image;
the textual information extraction tool executed by the processor extracting the textual information from the error image;
selecting the one or more keywords from the textual information by the keyword selection tool executed by the processor;
5. The method of claim 4, further comprising: the test is a primary test,
the test tool executed by the processor performs one or more retests without changing the test methodology after the primary test;
Determining the error handling based on the fact that there are cases in which the error information is acquired and cases in which the error information is not acquired in the primary test and the retest;
The method of any one of claims 1-5, further comprising: In the primary test and the one or more retests, if there are cases where the error information is acquired and cases where the error information is not acquired, as the error response, the page where the error occurred by the processor including providing such contact information;
7. The method of claim 6. the test includes a plurality of test cases;
the error information includes an error report for each test case in which the error occurred;
A method according to any one of claims 1-7. Said error is
an application error caused by the application;
a false positive error that is not an application error;
the error response is determined for the false positive error;
A method according to any one of claims 1-8. The false positive error includes a test error due to a test method for performing the test,
The plurality of countermeasures are set for the error pattern related to the test error,
10. The method of claim 9. A method of testing an application, wherein the test is performed by accessing an application server that provides the application from a test terminal having a test tool and a processor, the method comprising:
the test tool executed by the processor executing a primary test including a plurality of test cases;
the test tool acquiring error information regarding an error that occurred in the primary test for each test case;
By the processor, using a correspondence table containing a plurality of error patterns and a single or a plurality of countermeasures corresponding to each of the error patterns, a single or a plurality of countermeasures corresponding to the error based on the error information to identify measures;
Determining error handling for each test case by the processor,
If only one remedy is identified, decide on that sole remedy;
If a plurality of countermeasures are specified, one countermeasure selected from among the plurality of countermeasures is determined;
implementing the determined error handling;
executing a secondary test by the test tool executed by the processor for the test case for which the error handling has been performed;
How to test your application, including A system for determining error handling in testing an application, comprising:
a processor;
a test tool configured to perform the test by accessing an application server that provides the application by execution of the processor, the test tool being configured to obtain error information about an error that occurred in the test; a test tool and
with
The processor
Using a correspondence table containing a plurality of error patterns and one or more countermeasures corresponding to each error pattern, one or more countermeasures corresponding to the error are specified based on the error information. and
determining the error response; and
Determining the error handling includes:
deciding on a single remedy, if one is identified;
determining a selected one of the plurality of countermeasures when a plurality of countermeasures are identified. The system includes:
the application server that provides the application;
a test terminal operated by an operator;
The test terminal has a storage in which the test tool is stored and the processor,
13. The system of claim 12.

Images