TW202311947A - Device and method for re-executing of test cases in software application - Google Patents

Abstract

A method for re-executing of test cases including obtaining (i) a first status of an execution of a test case, (ii) re-executing the test case if the first status of the execution of the test case indicates a fail status, obtaining a second status of the re-execution of the test case, repeating steps (i) and (ii) for each test case in a plurality of test cases, obtaining a log of a plurality of first statuses of a plurality of test cases and at least one second status of at least one re-executed test case, searching for matches between test case names of the plurality of test cases and the at least one re-executed test case and if a match between the test case names has been found, changing the first status of the match to a skip status and using the second status as a final status of the match.

Description

Apparatus and method for re-executing test cases in software applications

Aspects of the present disclosure relate to apparatus and methods for re-executing test cases in software applications.

Validation and testing is an important part of developing software applications. A variety of tools currently exist to run test cases for applications under development. In a micro service environment, test cases are usually executed on different machines or in different networks. Test execution may fail due to issues such as microservice network delay, latency, virtual machine or Docker instance. Therefore, test cases should be repeatable and conclusive.

Usually, the Go language (Golang) testing framework does not support automated retesting of failed test cases. Since there is no automatic re-execution function, when fault classification is performed, each fault case needs to be manually re-executed, thus requiring a lot of effort to re-execute fault test cases. In this case, when running multiple Golang test cases, in order to achieve a good return on investment (ROI) and high test efficiency, it is crucial to support test case re-execution.

Therefore, it is desirable to have a way to make the identification, re-execution and management of test cases in software applications more efficient.

Various embodiments are directed to a method for re-executing a test case in a software application, comprising (i) obtaining a first state of test case execution; (ii) re-executing the test if the first state of test case execution indicates a failure state case, and obtain the second state of the test case re-execution; for each test case of the plurality of test cases, repeat steps (i) and (ii); obtain the plurality of first states of the plurality of test cases and at least one of the re-executed Execute the log of at least one second state of the test case; search for a match between the test case name of the plurality of test cases and the test case name of at least one re-executed test case, and if a match between the test case names is found, the The first state of the match is changed to the skipped state, and the second state is used as the final state of the match.

According to an embodiment, the method includes removing test cases with a status of skipped from the log.

According to an embodiment, the method includes checking a rerun flag status of a test case.

According to an embodiment, the method includes re-executing the test case if the first state of test case execution indicates a failure state and the rerun flag state indicates that the test case is in rerun mode.

According to one embodiment, the software application is written in Golang.

According to an embodiment, obtaining the log includes receiving test result data of a plurality of test cases, and parsing the log from the test result data.

According to an embodiment, the method includes outputting test-related information by writing the test-related information to a report log.

According to an embodiment, the method includes determining the source of the error if the first state of the test case execution indicates a failure state.

According to an embodiment, the method includes outputting information about the source of the error.

According to an embodiment, outputting test-related information and error sources includes uploading report logs to a server.

According to an embodiment, outputting test-related information and outputting error types includes displaying test-related information and error sources.

According to one embodiment, displaying test-related information includes displaying information associated with tests in the test list.

According to an embodiment, a server computer is provided, including a communication interface, a memory, and a processing unit, configured to execute the method according to one of the above embodiments.

According to one embodiment, there is provided a computer program element, including program instructions, which, when executed by one or more processors, cause one or more processors to execute the method according to one of the above-mentioned embodiments.

According to one embodiment, there is provided a computer-readable medium, including program instructions, which, when executed by one or more processors, cause the one or more processors to perform the method according to one of the above-mentioned embodiments.

The following detailed description refers to the accompanying drawings that diagrammatically show specific details and embodiments in which the present disclosure can be practiced. These examples are described in sufficient detail to enable those of ordinary skill in the art to practice the present disclosure. Other embodiments may be utilized and logical changes may be made without departing from the scope of the present disclosure. The various embodiments are not necessarily mutually exclusive, and some embodiments can be combined with one or more other embodiments to form new embodiments.

Embodiments described in the context of one device or method are similarly valid for other devices or methods. Likewise, embodiments described in the context of a device are similarly valid for a system or method, and vice versa.

Features described in the context of one embodiment can be correspondingly applied to the same or similar features in other embodiments. Features described in the context of one embodiment, even if not described in other embodiments, still apply to these embodiments correspondingly. In addition, additions and/or combinations and/or substitutions described for features in the context of one embodiment can be correspondingly applied to the same or similar features in other embodiments.

In the context of the various embodiments, the terms "a" and "the" when used with reference to a feature or element include references to one or more features or elements.

As used herein, "and/or" includes any and/or all combinations of one or more of the listed items.

Examples will be described in detail below.

FIG. 1 shows a computer 100 for developing software applications.

The computer 100 includes a central processing unit (CPU) 101 and a system memory (RAM) 102 . The system memory (for example, random access memory (RAM)) 102 is used to load program codes (for example, from a hard disk (ie, HDD) 103 ), and the CPU 101 executes the program codes.

In this example, it is assumed that the user intends to use the computer 100 to develop software applications. To this end, the user executes the software development environment 104 on the CPU 101 . Users can use computer languages, such as Golang, a statically typed, compiled programming language, to develop applications.

The software development environment 104 allows users to develop applications 105 for various devices 106 such as smart phones. To this end, the CPU 101, as part of the software development environment 104, runs a simulator (simulator) to simulate the device corresponding to the application program, such as a mobile device.

When the user has successfully developed the application program, the user can distribute the application program to the corresponding device 106 via the communication network 107, such as uploading the application program to an app store to distribute it to a smart phone.

Before doing so, however, the user should test the application 105 to avoid distributing an application that does not work correctly to the device 106 . To this end, the user further runs the testing tool 108 on the CPU 101, for example, using the Golang testing framework (Golang testing framework) in a heterogeneous environment to perform automatic testing of the presentation layer state transition on the Application Programming Interfaces (API) (Representational State Transfer; REST). A heterogeneous environment may mean a system with hardware and system software from different vendors.

The current Golang test framework only supports three test states of test cases: pass (PASS), failure (FAIL) and skip (SKIPPED). There is no way to represent "RERUN". This can lead to inefficiencies in testing as manual testing may be required to re-execute test cases.

In typical automated testing, users can simply deploy all test cases in the same environment or network, which is less complicated than deploying test cases in heterogeneous environments. This inevitably increases the complexity and the possibility of intermittent failures exponentially. In order to pursue higher automation ROI and test efficiency in heterogeneous environments, it is very desirable to have the feature of re-execution or re-run.

Thus, according to various embodiments, a system for re-executing test cases is presented. This makes testing more efficient.

In various embodiments, the testing tool 108 can be used to re-execute test cases in a software application. The testing tool 108 can obtain a first state of test case execution of the software application under test. The first status may eg be a pass status or a fault status.

If the first state of test case execution indicates a failure state, the test tool 108 may re-execute the test case.

The testing tool 108 can obtain the second status of the test case re-execution. The second status may eg be a pass status or a fault status.

If the initial execution of each of the plurality of test cases indicates a failure condition, the test tool 108 may repeat the execution steps of the test case and the re-execution of the test case.

The testing tool 108 can obtain logs (or reports) of a plurality of first states of a plurality of test cases and at least a second state of at least one re-executed test case. The testing tool 108 may search for a match between the test case names of the plurality of test cases and the test name of at least one re-executed test case. If a match of the test case name is found, the first state of the match can be changed to the skipped state, and the second state can be used as the final state of the match.

As used herein, the term "re-execution" can be used interchangeably with "re-run".

In various embodiments, the computer 100 may remove test cases with skipped status from the log. Since the test cases with skipped status in the log have been re-executed, the log can be simplified to make data management more efficient by removing the skipped test cases.

In various embodiments, computer 100 may check the status of the test case's rerun flag. The rerun flag may indicate whether the test case is in rerun mode, or not in rerun mode. For example, if the rerun flag is set to "1", the rerun mode may be triggered, and when the rerun flag is set to "0", the rerun mode may be deactivated.

In various embodiments, the computer 100 may re-execute the test case if the first state of the test case execution indicates a failure state, and the state of the re-run flag indicates that the test case is in re-run mode.

In various embodiments, obtaining the logs includes receiving test result data of a plurality of test cases, and parsing the logs from the test result data. In one embodiment, parsing the logs from the test result data may include error detection or error source analysis from the test result data.

In various embodiments, the computer 100 can output test-related information by writing the test-related information into a log. For example, test-related information can be written to logs in XML format. Logs can be displayed in table format.

In various embodiments, if the first state of the test case execution indicates a failure state, the computer 100 can determine the source of the error.

In various embodiments, the computer 100 can output relevant information about the source of the error. Sources of errors could be, for example, intermittent failures or product coding issues.

In various embodiments, outputting error-related information and error sources may include uploading a report log to a server. In various embodiments, outputting test-related information and error types includes displaying test-related information and error sources. In various embodiments, displaying test-related information includes displaying information associated with the test in a test list or table.

In various embodiments, a server computer is provided, which includes a communication interface, a memory, and a processing unit, configured to perform the method according to one of the above-mentioned embodiments.

In various embodiments, a server computer component is provided, which includes program instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to one of the above-mentioned embodiments.

In various embodiments, there is provided a computer-readable medium containing program instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to one of the above-described embodiments.

FIG. 2 shows a flowchart 200 illustrating re-execution of test cases, according to one embodiment.

In flowchart 200, at step 202, the execution of the test suite (ie, the verification suite) of the software application begins.

In step 204, each test case in the test suite will be run recursively starting from the first test case. At step 205, if the test case (eg, the first test case) passes, a rerun will not be triggered and will therefore (return to step 203) proceed to the next test case (if any). In step 206, it is checked whether the rerun flag is set to "rerun". If the rerun flag is set to "rerun", the same case (ie failure case) will be triggered again in step 204 . In step 206, if the re-run flag has not been set, the failure case will be recorded as a "failure" status, and the system will proceed to the next test case in step 203 . If the test case is passed in step 205, step 207 is triggered to check whether the test suite has ended. If the test suite is not finished, the system will proceed to the next test case in step 203 . If the test suite is finished, the process ends at step 208 .

In various embodiments, an example source code for re-execution of a test case may be as follows: func runTests(matchString func(pat, str string) (bool, error), tests []InternalTest) (ran, ok bool) { rerunFlag := false reran := false if strings.ToUpper(os.Getenv("RERUN")) == "TRUE" { rerunFlag = true } ok = true for _, procs := range cpuList { runtime.GOMAXPROCS(procs) for i := uint(0); i < *count; i++ { if shouldFailFast() { break } ctx := newTestContext(*parallel, newMatcher(matchString, *match, "-test.run")) t := &T{ common: common{ signal: make(chan bool), barrier: make(chan bool), w: os.Stdout, chatty: *chatty, }, context: ctx, } tRunner(t, func(t *T) { for _, test := range tests { r := t.Run(test.Name, test.F) if rerunFlag && !r { time. Sleep(5 * time. Second) reran = true t.reReunFailed = !t.Run(test.Name, test.F) } } // Run catching the signal rather than the tRunner as a separate // goroutine to avoid adding a goroutine during the sequential // phase as this pollutes the stacktrace output when aborting. go func() { <-t.signal }() }) if !rerunFlag { ok = ok && !t.Failed() } else { if !reran { ok = ok && !t.Failed() } else { ok = ok && !t.reReunFailed } } ran = ran || t.ran } } return ran, ok }.

In the sample source code above, the system checks to see if the rerun flag is set. If the rerun flag has not been set, the test case will not be rerun. For example, an example source code could be "rerun flag := false, rerun := false". If the rerun flag is set and the test case fails, the test case will be rerun. For example, the sample source code could be "!rerunFlag { ok = ok && !t.Failed()". The test program can be in the form of a loop (such as a For loop), and can be implemented for each test case of the test suite. Test procedures can also contain conditional statements (such as if else statements) to handle different scenarios with rerun flags set or not set, and with test case status passing or failing.

FIG. 3 shows a flowchart 300 illustrating report cleanup of a test case, according to one embodiment.

In step 301, the test suite 301 ends and a log (XML file) is generated. In step 302, log (report) clearing is triggered to clear the log. In step 303, each test case of the test suite in the log will be processed. In step 304, the test tool checks whether the test case name of each test case is the same as the next test case. If yes, mark the current test case as "skip" in step 305 . Otherwise, the status of the test case (eg pass/fail) remains the same, and the system proceeds to step 306 to check whether the current test case is the last test case. In other words, the system checks if there are any pending test cases. If the current test case is not the last case, the system proceeds to step 303 to process a new test case. If the current test case is the last case, the procedure ends at step 307 .

In various embodiments, sample source code for reporting cleanup may be as follows: for i := 0; i < len(suites.Suites); i++ { for j := 0; j < len(suites.Suites[i].TestCases); j++ { if suites.Suites[i].TestCases[j].Failure != nil { //fmt.Println(k.Name) if j+1 < len(suites.Suites[i].TestCases) { if suites.Suites[i].TestCases[j].Name == suites.Suites[i].TestCases[j+1].Name { suites.Suites[i].TestCases[j].SkipMessage = &JUnitSkipMessage{Message: "RERUN: " + suites.Suites[i].TestCases[j].Failure.Contents} suites.Suites[i].TestCases[j].Failure = nil } } } } }.

In the above sample source code, the system checks whether the current test case has the same test case name as the next test case. If so, the current test case will be marked as "Skipped". For example, the sample source code to mark the current test case as "skip" could be "if suites.Suites[i].TestCase[j].Name == suites.Suites[i].TestCase[j+1 ].Name { suites.Suites[i].TestCase[j].SkipMessage". The cleanup procedure can be in the form of a loop (eg For loop) and can be implemented for each test case item in the log. Cleanup procedures can also contain conditional statements (such as if else statements) to handle different scenarios with the same or different test case names.

FIG. 4A shows a table 400 in a report showing an example status of a test case, according to one embodiment.

In the example table 400, for each test case (e.g., test case A 406, test case B 408, and test case C 410), there may be a first state 402 (i.e., an initial run state) and/or a second state 404 (that is, the status of re-running).

In an example, when the first status 402 of the test case (eg, test case A 406 ) is a pass status, rerun may not be required, and the second status 404 may be marked as a nil status. As another example, when the second state 402 of the test case (eg, test case B 408 or test case C 410 ) is a failure state, the test case (eg, test case B 408 or test case C 410 ) may be rerun. As shown by example, the second state 404 may be a pass state or a fail state. In one embodiment, if the second state 404 is a failure state, the system may allow the test case to be further rerun.

FIG. 4B shows a table 420 in a report showing an example status of test cases before and after cleaning according to one embodiment.

In the example table 420, for each test case, there may be a pre-clear state 422 and/or a post-clear state 424.

For example, for the test case A 426 , since the pre-clearing state 422 is a pass state, there is no need to re-run, and the post-clearing state 424 also reflects the pass state. As another example, for an initial run of test case B 428A, the initial run may show a fail status before clearing, while re-running test case B 428B may show a pass status in a rerun before clearing. Since the initial run test case B 428A and the rerun test case B 428B have the same name (ie, test case B), the initial run test case B 428A may have a post-clear status 424 of "skipped". As another example, for an initial run of test case C 430A, the initial run may demonstrate a pre-clear fault condition, while re-running test case C 430B may demonstrate a pre-clear fault condition in re-run. Since the initial run of test case C 430A and the re-run of test case C 430B have the same name (ie, test case C), the initial run of test case C 430A may have a post-clear status 424 of "skipped". In other words, if the initial test case and the rerun test case have the same test case name, the initial test case will have a status of "Skip", regardless of whether the rerun test case exhibits a pass or fail status.

FIG. 5A shows a schematic diagram 500 illustrating an example failure percentage without re-execution of the function, according to one embodiment.

In the example diagram 500, there are 9900 pass cases 504 (or 99% pass cases) and 100 fail cases 506 (or 1% fail cases) out of 10,000 test cases 502. Among the 100 failures 506, there are 80 intermittent failures 508 (or 0.8% intermittent failures) and 20 product coding issues 510 (or 0.2% product coding issues).

FIG. 5B shows a schematic diagram 550 illustrating example failure percentages with rerun functionality, according to one embodiment.

In the example schematic 550 , there are 9970 pass cases 554 (or 99.7% pass cases) and 30 fail cases 556 (or 0.3% fail cases) out of 10,000 test cases 552 . Of the 30 failures 556, there were 10 intermittent failures 558 (or 0.1% intermittent failures) and 20 product coding issues 560 (or 0.2% product coding issues).

In various embodiments, if each case takes 5 minutes to sort, the system without reruns will take 500 minutes (8 hours) to resolve the failures therein. With the re-run function, the intermittent fault 558 can be reduced to 0.1%, effectively saving about 6 hours of troubleshooting time per day. Therefore, this can result in better engineering productivity and achieve better engineering ROI, since triaging test failures can be time consuming. In a heterogeneous environment, test cases may fail due to factors such as network instability, operating system (OS) instability, and/or third-party indirect outages. Having tools that support test case reruns when a test case failed in a previous run improves test stability and improves engineering productivity.

In general, according to various embodiments, as depicted in FIG. 6 , a method is provided.

FIG. 6 shows a flowchart 600 illustrating a method of re-executing test cases in a software application.

At 602, a first status of a test case execution of a software application under test can be obtained.

At 604, the test case may be re-executed if the first status of the test case execution indicates a failure status, and the second status of the test case re-execution may be obtained.

At 606, steps 602 and 604 may be repeated for each of the plurality of test cases.

At 608, logs of a plurality of first states of a plurality of test cases and at least a second state of at least one re-executed test case may be obtained.

At 610, a match may be searched for the test case name of the plurality of test cases and the test case name of at least one re-executed test case.

At 612, if a match is found for the test case name, the first state of the match may be changed to a skipped state.

Steps 602 to 612 are shown in a particular order, however other arrangements are possible. Steps can also be combined into certain cases. Steps 602 through 612 may be in any suitable order.

The approach in Figure 6 can be used for different types of software automation testing, such as API automation for different platforms. According to various embodiments, a tool is provided for re-running of test cases with the capability of serving as a unified test automation reporting platform. The tool can be used by any software company that uses automated testing across multiple platforms (web, mobile, API), especially for software applications in Golang format.

The method in FIG. 6 is implemented by the server computer shown in FIG. 7, for example.

FIG. 7 shows a server computer system 700 according to one embodiment.

The server computer system 700 includes a communication interface 701 (eg, configured to receive test input data or logs, and configured to output test-related information, such as status or error sources). The server computer system 700 further includes a processing unit 702 and a memory 703 . The memory 703 can be used by the processing unit 402 to store, for example, data to be processed, such as logs. The server computer is set up to perform the method in Figure 6. It should be noted that the server computer system 700 may be a distributed system including a plurality of computers.

The methods described herein may be performed by one or more circuits, and various processing or computing units, devices and computing entities described herein (such as the processing unit 602 ) may be implemented by one or more circuits. In one embodiment, a "circuit" can be understood as any kind of logical implementation entity, which can be hardware, software, firmware or any combination thereof. Thus, in one embodiment, a "circuit" may be a hard-wired logic circuit or a programmable logic circuit, such as a programmable processor, such as a microcontroller. A "circuit" may also be software executed or implemented by a processor, such as any computer program, such as a computer program coded using a virtual machine. According to alternative embodiments, any other implementation of the respective functions described herein may also be understood as a "circuit".

Although the present disclosure has been shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various embodiments may be made therein without departing from the spirit and scope of the invention as defined in the appended claims. changes and details. Accordingly, the scope of the present invention is indicated by the appended claims and is intended to embrace all changes that come within the meaning and range of equivalents of the claims.

100: computer 101:Central Processing Unit (CPU) 102: System memory (RAM) 103: Hard disk (HDD) 104:Software development environment 105: Apps 106: Device 107: Network 108:Test tools 200: Flowchart 202-208: Steps 300: Flowchart 301-307: Steps 400: form 402: First state 404: second state 406: Test case A 408:Test case B 410: Test Case C 420: Form 422: Clear the previous state 424: Status after clearing 426:Test case A 428A: Initial run test case B 428B: Rerun test case B 430A: Initial Operation Test Case C 430B: Rerun test case C 500: Schematic diagram 502: Case in test 504: Pass the case 506: failure 508: Intermittent failure 510: Product coding problem 550: Schematic diagram 552: Case in test 554:Pass the case 556: failure 558: Intermittent failure 560: Product coding problem 600: Flowchart 602-612: Steps 700: Server computer system 701: communication interface 702: processing unit 703: Memory

The present invention may be better understood by reference to the following detailed description of the embodiments, and consideration of the non-limiting examples and accompanying drawings, in which: Figure 1 shows a computer used to develop software applications. FIG. 2 shows a flowchart illustrating re-execution of test cases according to one embodiment. FIG. 3 shows a flowchart illustrating report cleanup of a test case, according to one embodiment. Figure 4A shows a table in a report showing sample status of test cases according to one embodiment. FIG. 4B shows a table in a report showing an example status of test cases before and after cleanup, according to one embodiment. FIG. 5A shows a schematic diagram illustrating example failure percentages without re-execution of functions, according to one embodiment. FIG. 5B shows a schematic diagram illustrating example failure percentages with rerun functionality, according to one embodiment. FIG. 6 shows a flowchart illustrating a method of re-executing test cases in a software application. FIG. 7 shows a server computer system according to an embodiment.

200: Flowchart

202-208: Steps

Claims (15)

A method of re-executing test cases in a software application, comprising: (i) obtaining a first state of execution of a test case of the software application under test; (ii) if the first state of execution of the test case indicates a failure state, re-execute the test case, and obtain a second state of the re-execution of the test case; Repeat steps (i) and (ii) for each test case of the plurality of test cases; obtaining a log of a plurality of first states of the plurality of test cases and at least a second state of at least one re-executed test case; searching for a match between the test case name of the plurality of test cases and the test case name of the at least one re-executed test case; Wherein if a match between test case names is found, change the first state of the match to a skip state, use the second state as a final state of the match, and log in Test cases with this skipped status are removed. The method of claim 1 further includes: Check the rerun flag status for one of the test cases. The method of claim 1 or 2 further includes: If the first state of execution of the test case indicates a failure state and the rerun flag state indicates that the test case is in a rerun mode, re-execute the test case. The method as claimed in any one of items 1 to 3, wherein the software application program is written in Golang. The method according to any one of claims 1 to 4, wherein obtaining the log includes receiving test result data of the plurality of test cases, and parsing the log from the test result data. The method according to any one of claims 1 to 5, further comprising: Outputs the test-related information by writing the test-related information to the report log. The method according to any one of claims 1 to 6, further comprising: If the first state of the execution of the test case indicates a failure state, determine an error source. Such as the method of claim item 7, further comprising: Output information about the source of the error. The method according to any one of claims 1 to 8, wherein outputting the test-related information and the error source includes uploading the report log to a server. The method according to any one of claims 1 to 9, wherein outputting the test-related information and outputting the error type includes displaying the test-related information and the source of the error. The method of any one of claims 1 to 10, wherein displaying the test-related information includes displaying information associated with tests in a test list. A server computer, including a communication interface, a memory, and a processing unit, configured to execute the method of any one of claim items 1 to 11. A computer program element, including program instructions, which, when executed by one or more processors, causes the one or more processors to perform the method of any one of claim items 1 to 11. A computer-readable medium, comprising program instructions, which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 11. An apparatus for re-executing test cases in a software application, comprising a processor configured to: obtaining a first state of execution of one of the plurality of test cases of the software application under test; if the first state of execution of the test case indicates a failure state, re-execute the test case and obtain a second state of the re-execution of the test case; For each of the plurality of test cases, repeatedly obtaining the first state and at least one second state; obtaining a log of the plurality of first states of the plurality of test cases and the at least one second state of at least one re-executed test case; Searching for a match of the test case name of the plurality of test cases with the test case name of the at least one re-executed test case: Wherein if a match between test case names is found, change the first state of the match to a skip state, use the second state as a final state of the match, and log in Test cases with this skipped status are removed.

Images