EP4716890A1

EP4716890A1 - Test result analysis for automated testing

Info

Publication number: EP4716890A1
Application number: EP24728444.1A
Authority: EP
Inventors: Shaopeng BU; Yingjie Li; Hong Ding
Original assignee: Microsoft Technology Licensing LLC
Current assignee: Microsoft Technology Licensing LLC
Priority date: 2023-05-23
Filing date: 2024-05-08
Publication date: 2026-04-01
Also published as: CN119025398A; WO2024242875A1

Abstract

The present disclosure proposes a method, apparatus, and computer-readable medium for test result analysis for automated testing. A raw test result generated through performing automated testing on a target application may be obtained, the target application including a plurality of user interfaces, the raw test result including a user interface navigation map of the target application, the user interface navigation map including a plurality of interaction paths, and each interaction path consisting of a portion of the plurality of user interfaces. A set of application analysis rules may be obtained, each application analysis rule targeting a type of user interface issue. A user interface with a user interface issue in the target application may be identified through pattern matching the raw test result with each application analysis rule in the set of application analysis rules, respectively. A user interface improvement suggestion for the user interface may be generated.

Description

TEST RESULT ANALYSIS FOR AUTOMATED TESTING

BACKGROUND

[0001] In a development process of a software application, testing has a very critical effect as an extremely important part of ensuring quality of the application. Herein, a software application on which testing is performed is referred to as a target application. Usually, when performing a test on a target application, after determining a test case, a tester can execute the test step by step according to steps described in the test case and compare an actual test result with a desired test result to verify whether various functions of the target application are correctly. In this process, automated testing is introduced to save human power, time or hardware resources and to improve testing efficiency. Automated testing may be a process of converting human-driven tests into machine-executed tests. In the automated testing, a specific software or program may be utilized to control the execution of testing and the comparison between the actual test result and the desired test result. Through the automated testing, some repetitive but necessary testing tasks that exist in the testing process can be automated, or some testing tasks that would otherwise be difficult to perform in a manual manner can be performed.

SUMMARY

[0002] This Summary is provided to introduce a selection of concepts that are further described below in the Detailed Description. It is not intended to identify key features or essential features of the claimed subj ect matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Embodiments of the present disclosure propose a method, apparatus and computer- readable media for test result analysis for automated testing. A raw test result generated through performing automated testing on a target application may be obtained, the target application including a plurality of user interfaces, the raw test result including a user interface navigation map of the target application, the user interface navigation map including a plurality of interaction paths, and each interaction path consisting of a portion of the plurality of user interfaces. A set of application analysis rules may be obtained, each application analysis rule targeting a type of user interface issue. A user interface with a user interface issue in the target application may be identified through pattern matching the raw test result with each application analysis rule in the set of application analysis rules, respectively. A user interface improvement suggestion for the user interface may be generated. A selection of one of the plurality of interaction paths may be received. A series of actions corresponding to the selected interaction path may be obtained. A test case for the target application may be generated based on the series of actions.

[0004] It should be noted that the above one or more aspects compnse the features hereinafter fully described and particularly pointed out in the claims. The following description and the drawings set forth in detail certain illustrative features of the one or more aspects. These features are only indicative of the various ways in which the principles of various aspects may be employed, and this disclosure is intended to include all such aspects and their equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosed aspects will hereinafter be descnbed in connection with the appended drawings that are provided to illustrate and not to limit the disclosed aspects.

[0006] FIG. 1 illustrates an exemplary' architecture of a system for test result analysis for automated testing according to an embodiment of the present disclosure

[0007] FIG. 2 illustrates an exemplary user interface navigation map according to an embodiment of the present disclosure.

[0008] FIG. 3 illustrates an exemplary process for test result analysis for automated testing according to an embodiment of the present disclosure.

[0009] FIG. 4 illustrates an exemplary' process for identifying a user interface with a user interface global issue in a target application and generating a user interface global improvement suggestion according to an embodiment of the present disclosure.

[0010] FIG. 5 illustrates an exemplary process for identifying a user interface component with a user interface component issue in a target application and generating a user interface component improvement suggestion according to an embodiment of the present disclosure.

[0011] FIG. 6 illustrates an exemplary process for creating a prompt for a user interface component improvement suggestion according to an embodiment of the present disclosure.

[0012] FIG. 7 is an exemplary chat user interface illustrating a prompt and a user interface component improvement suggestion according to an embodiment of the present disclosure.

[0013] FIG. 8 illustrates an exemplary process for identifying an error message in a test log and generating an error reason and a fix suggestion for the error message according to an embodiment of the present disclosure.

[0014] FIG. 9 illustrates an exemplary process for creating a prompt for an error reason and a fix suggestion according to an embodiment of the present disclosure.

[0015] FIG. 10 is an exemplary chat user interface illustrating a prompt and an error reason and a fix suggestion according to an embodiment of the present disclosure.

[0016] FIG. 11 illustrates an exemplary process for generating a test case and test data according to an embodiment of the present disclosure.

[0017] FIG. 12 illustrates an example of generating a test case according to an embodiment of the present disclosure.

[0018] FIG. 13 illustrates an exemplary process for creating a prompt for test data according to an embodiment of the present disclosure.

[0019] FIG. 14 is an exemplary chat user interface illustrating a prompt and test data according to an embodiment of the present disclosure.

[0020] FIG. 15 is a flowchart of an exemplary method for test result analysis for automated testing according to an embodiment of the present disclosure.

[0021] FIG. 16 illustrates an exemplary apparatus for test result analysis for automated testing according to an embodiment of the present disclosure.

[0022] FIG. 17 illustrates another exemplary apparatus for test result analysis for automated testing according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

[0023] The present disclosure will now be discussed with reference to several example implementations. It is to be understood that these implementations are discussed only for enabling those skilled in the art to better understand and thus implement the embodiments of the present disclosure, rather than suggesting any limitations on the scope of the present disclosure.

[0024] Automated testing based on intelligent exploration has been proposed. The automated testing based on intelligent exploration may be abbreviated as intelligent exploration testing. The intelligent exploration testing is intended to automatically determine an action for a user interface of a target application based on an understanding of the user interface of the target application. The understanding of the user interface of the target application may be obtained through a machine learning technique. The action for the user interface may be determined through a rulebased approach or a model-based approach The rule-based approach is intended to identify a scenario category of a user interface, obtain an action corresponding to a rule for the scenario category, and determine the obtained action as an action for the user interface. The model-based approach is intended to predict an action for a user interface through an action decision model which is based on Reinforcement Learning. When the determined action is applied to the user interface, a next interface of the target application may be triggered. Through continuously determining an action and applying the action, user interfaces of the target application can be explored to obtain a plurality of user interfaces of the target application, and during this process, vulnerabilities or faults existed in the target application can be identified Through performing the intelligent exploration testing on the target application, a raw test result may be generated. The raw test result may include a user interface navigation map, a page source, a test log, etc. for the target application.

[0025] Embodiments of the present disclosure propose test result analysis for automated testing. The automated testing is, e.g. , an intelligent exploration testing. A raw test result generated through performing automated testing on a target application may be obtained, such as a user interface navigation map, a page source, a test log, etc., for the target application. The target application may include a plurality of user interfaces. Each user interface may include at least one user interface component. The user interface navigation map may include a plurality of interaction paths. Each interaction path may consist of a portion of the plurality of user interfaces of the target application. The page source may include a layout of each user interface in the target application, attributes of each user interface component, etc. The test log may include a test log produced by the target application, a test log produced by an automated testing system, etc., during the execution of automated testing on the target application through the automated testing system. The raw test result may be analyzed to generate a test result analysis. The test result analysis may include a user interface improvement suggestion for a user interface with a user interface issue in the target application. Herein, a user interface issue refers to an issue related to a design of a user interface. The user interface issue may include a user interface global issue at a user interface level, a user interface component issue at a user interface component level, etc. Accordingly, the user interface improvement suggestion may include a user interface global improvement suggestion, a user interface component improvement suggestion, etc. The test result analysis may also include an error reason and a fix suggestion for an error message in the test log. These test result analyses may be in a natural language form that can be understood by humans. Based on the above test result analysis, a test report may be constructed for the target application. Such a test report can visually present the issues existed in the target application from multiple aspects and effective suggestions to solve these issues.

[0026] In an aspect, the embodiments of the present disclosure propose to identify a user interface with a user interface global issue in a target application through pattern matching a user interface navigation map with each user interface global analysis rule in a set of user interface global analysis rules, respectively. Each user interface global analysis rule may target a type of user interface global issue. The set of user interface global analysis rules may include a page depth rule, a page loading time rule, a First Registration Experience (FRE) rule, etc. The page depth rule is intended to identify a user interface in the target application with high interactivity but hidden too deep in an interaction path. The page loading time rule is intended to identify a user interface in the target application with a long loading time. The first registration experience rule is intended to identify user interfaces in the target application that cause a first registration experience flow to be too long. Herein, a first registration experience flow refers to a flow that a user experiences when registering a software application for the first time. After identifying a user interface with a user interface global issue, a user interface global improvement suggestion for the user interface may be generated. For each user interface global analysis rule, a user interface global improvement suggestion corresponding to the user interface global analysis rule may be predefined. After a user interface with a user interface global issue is identified according to a user interface global analysis rule, a user interface global improvement suggestion for the user interface may be a predefined user interface global improvement suggestion corresponding to the user interface global analysis rule. Through the above manner, a developer or tester of the target application can have a clear understanding of which user interface or user interfaces in the target application need to be improved and how to improve them.

[0027] In another aspect, the embodiments of the present disclosure propose to identify a user interface component with a user interface component issue in a target application through pattern matching a page source with each user interface component analysis rule in a set of user interface component analysis rules, respectively. Each user interface component analysis rule may target a type of user interface component issue. The set of user interface component analysis rules may include a user interface component discoverability rule, a user interface component accessibility rule, a user interface component interactivity rule, etc. The user interface component discoverability rule is intended to identify a user interface component in the target application whose discoverability does not meet a requirement. Herein, the discoverability of a user interface component refers to how easily the user interface component can be discovered by a user. The user interface component accessibility rule is intended to identify a user interface component in the target application whose accessibility does not meet a requirement. Herein, the accessibility of a user interface component refers to how easily the user interface component can be accessed by a user, particularly a visually impaired person. The user interface component interactivity rule is intended to identify a user interface component in the target application whose interactivity does not meet a requirement. Herein, the interactivity of a user interface component refers to how easy it is for a user to interact with the user interface component. After identifying a user interface component with a user interface component issue, a user interface component improvement suggestion for the user interface component may be generated. A page source, etc., may be input to a Large Language Model (LLM). The Large Language Model may generate a user interface component improvement suggestion in a natural language form using its semantic understanding capability, inference capability, large data support capability, etc. Through the above manner, a developer or tester of the target application can have a clear understanding of which user interface component or user interface components in the target application need to be improved and how to improve them

[0028] In yet another aspect, the embodiments of the present disclosure propose to identify an error message in a test log, and generating an error reason, a fix suggestion, etc. for the error message through a large language model. Key information associated with the error message, etc. extracted from the test log may be input to the large language model. The large language model may generate the error reason, the fix suggestion, etc. in a natural language form for the error message using its semantic understanding capability, inference capability, large data support capability, etc. Through the above manner, a developer or tester of the target application can easily locate a vulnerability or fault existed in the target application, and improve the efficiency of resolving the vulnerability or fault.

[0029] In still another aspect, the embodiments of the present disclosure propose to automatically generate a test case and corresponding test data for a target application. The test case may include a series of test steps or actions to be performed on the target application. A user interface navigation map includes a plurality of interaction paths. A selection of one of the plurality of interaction paths may be received. Since the user interface navigation map is obtained through intelligent exploration testing, it is known what actions are performed to jump among various user interfaces in the interaction path. Accordingly, a series of actions corresponding to the selected interaction path may be obtained. A test case for the target application may be generated based on the series of actions. The test case may contain a step for inputting test data in a particular user interface. Herein, test data refers to data needed to be input into a user interface during testing. For example, for a user interface whose scenario category is a search scenario, test data needed to be input into the user interface may be a search keyword. The test data may be generated through a large language model to further enrich the test case. The large language model may generate test data for the user interface based on a scenario category of the user interface, etc. Through the above manner, a test case and corresponding test data that can explore an expected interaction path can be automatically generated, thus improving the efficiency of automated testing.

[0030] Various embodiments of the present disclosure will hereinafter be described in connection with the appended drawings.

[0031] FIG. 1 illustrates an exemplary architecture 100 of a system for test result analysis for automated testing according to an embodiment of the present disclosure

[0032] The architecture 100 includes an automated testing system 110 and a large language model 190. The automated testing system 110 may provide an automated testing service for a target application. The target application may include, e.g., a mobile application running on a mobile device, a computer application running on a desktop computer or a portable computer, etc. The automated testing service is, e.g., an intelligent exploration testing service. The large language model 190 may be a text generative model, such as a Generative Pre-trained Transformer (GPT). [0033] The automated testing system 110 includes a front-end 120, a back-end system 130, a test device set 180-1 to atest device set 180-M (M > 1), etc. The front-end 120 may interface with a user of the automated testing system 110, and may present, to the user, a chat user interface 122 associated with the large language model 190 and a back-end system interactive user interface 124 associated with the back-end system 130. The user of the automated testing system 110 may be a tester, a developer, etc. of the target application. The test device set 180-1 to the test device set 180-M may be interfaced with the back-end system 130. Each test device set may have various forms, such as a single test device, a test device pair consisting of two test devices, and a test device group consisting of more than two test devices, etc. Each test device set may be configured to run a target application. A single test device may be used to independently perform automated testing for the target application. The test device pair and the test device group may be used to collaboratively perform automated testing for the target application.

[0034] The back-end system 130 includes atest device set management unit 140 for managing the test device set 180-1 to the test device set 180-M.

[0035] The back-end system 130 also includes a test execution unit 150 for executing automated testing for the target application with the test device set 180-1 to the test device set 180-M. In particular, the test execution unit 150 includes an intelligent exploration test executor 152. The intelligent exploration test for the target application may be performed through the intelligent exploration test executor 152. The test execution unit 150 may output a raw test result for the target application.

[0036] The raw test result may include a user interface navigation map for the target application. The user interface navigation map may include a plurality of interaction paths. Each interaction path consists of a portion of a plurality' of user interfaces of the target application. FIG. 2 illustrates an exemplary user interface navigation map 200 according to an embodiment of the present disclosure. The user interface navigation map 200 includes 12 user interfaces, i.e., a user interface 1 to a user interface 12. The user interface 1 may be, e.g., a launching interface of the target application, i.e., a user interface that is presented when the target application is launched. The user interface navigation map 200 includes a plurality' of interaction paths. For example, applying an action A to the user interface 1 may trigger a user interface 2, applying an action B to the user interface 1 may trigger a user interface 3, and applying an action C to the user interface 1 may trigger a user interface 4. The interaction paths may be further explored. For example, applying an action D to the user interface 2 may trigger a user interface 5, applying an action E to the user interface 5 may trigger a user interface 6, applying an action F to the user interface 6 may trigger a user interface 9, and so on. In this case, the user interface 1, the user interface 2, the user interface 5, the user interface 6, and the user interface 9 may form an interaction path. In addition, some user interfaces may be switchable back and forth. For example, applying an action G to the user interface 5 may trigger a user interface 7, while applying an action H to the user interface 7 may return to the user interface 5. The action applied to each user interface may be determined automatically based on an understanding of the user interface, through a rule-based approach or through a model-based approach.

[0037] The raw test result may also include a page source of the target application. The page source may include a layout of each user interface in the target application, attributes of each user interface component in the target application, etc. The attributes of the user interface component may include name, type, shape, font color, background color, etc. of the user interface component. For a user interface component whose type is an image, a theme color of the user interface component may be calculated through a computer vision model, and the theme color may be added, as an attribute of the user interface component, to the page source. The page source may be in extensible Markup Language (XML) format.

[0038] The raw test result may also include a test log produced during the execution of automated testing on the target application through the automated testing system. The test log may include a test log produced by the target application, and a test log produced by the automated testing system, etc.

[0039] The back-end system 130 also includes a test result processing unit 160 for analyzing the raw test result of the target application output by the test execution unit 150. The test result processing unit 160 may generate test a result analysis for the raw test result either separately or in collaboration with the large language model 190. The test result analysis may include a user interface global improvement suggestion for a user interface with a user interface global issue in the target application, a user interface component improvement suggestion for a user interface component with a user interface component issue in the target application, an error reason and a fix suggestion for an error message in a test log, etc. These test result analyses may be in a natural language form that can be understood by humans. When collaborating with the large language model 190 to generate the test result analysis for the raw test result, a prompt may be created as an input to the large language model 190 through a prompt creator 165 in the test result processing unit 160. The large language model 190 may generate a corresponding test result analysis based on the prompt

[0040] The test result processing unit 160 includes a user interface global analyzer 161. The user interface global analyzer 161 may identify a user interface with a user interface global issue in the target application, and generate a user interface global improvement suggestion for the user interface. An exemplary' process for identifying a user interface with a user interface global issue in a target application and generating a user interface global improvement suggestion will be illustrated later in connection with FIG. 4. The test result processing unit 160 also includes a user interface component analyzer 162. The user interface component analyzer 162 may identify a user interface component with a user interface component issue in the target application, and generate a user interface component improvement suggestion for the user interface component in conjunction with the prompt creator 165 and the large language model 190. An exemplary process for identifying a user interface component with a user interface component issue in a target application and generating a user interface component improvement suggestion will be illustrated later in connection with FIG. 5. The test result processing unit 160 also includes an error message analyzer 163. The error message analyzer 163 may identify an error message in a test log, and generate an error reason and a fix suggestion for the error message in conjunction with the prompt creator 165 and the large language model 190. An exemplary process for identifying an error message in a test log and generating an error reason and a fix suggestion for the error message will be illustrated later in connection with FIG. 8. The test result processing unit 160 also includes a test case generator 164. The test case generator 164 may generate a test case. In addition, in conjunction with the prompt creator 165 and the large language model 190, test data for the test case may be generated. An exemplary process for generating a test case and test data will be illustrated later in conjunction with FIG. 11. The generated test case and test data may be provided to the test execution unit 150 to perform automated testing for the target application. Alternatively or additionally, the test case and the test data may be exported, e.g., through the back-end system interactive user interface 124, to a tester of the target application for performing further optimization thereof.

[0041] The back-end system 130 also includes a test report construction unit 170 for constructing a test report for the target application based on the test result analysis output by the test result processing unit 160. The test result processing unit 160 may output multiple types of test result analysis, such as a user interface global improvement suggestion for a user interface with a user interface global issue in the target application, a user interface component improvement suggestion for a user interface component with a user interface component issue in the target application, an error reason and a fix suggestion for an error message in the test log, etc. The test report construction unit 170 may construct a test report through combining these test result analyses together. Such a test report can visually present issues existed in the target application from a plurality of aspects, as well as effective suggestions for resolving these issues. Content of the test report may be presented in the chat user interface 122 in the front-end 120. Alternatively or additionally, a link to the test report may be presented in the back-end system interactive user interface 124 in the front-end 120. A tester or developer of the target application may download or read the test report through clicking on the link.

[0042] It should be appreciated that the architecture 100 shown in FIG. 1 is merely an example of the architecture of the system for test result analysis for automated testing. Depending on actual application requirements, the system for test result analysis for automated testing may have any other architecture and may include more or fewer modules.

[0043] FIG. 3 illustrates an exemplary process 300 for test result analysis for automated testing according to an embodiment of the present disclosure. The process 300 may be implemented by the test result analysis system of FIG. 1.

[0044] At 302, a raw test result generated through performing automated testing on a target application may be obtained. The target application may include a plurality of user interfaces. Each user interface may include at least one user interface component. The automated testing is, e.g., an intelligent exploration testing. The raw test result may include a user interface navigation map, a page source, a test log, etc., of the target application.

[0045] At 304, a user interface with a user interface issue in the target application may be identified. For example, a set of application analysis rules may be obtained. Each application analysis rule may target a type of user interface issue. A user interface with a user interface issue in the target application may be identified through pattern matching the raw test result with each application analysis rule in the set of application analysis rules, respectively. The set of application analysis rules may include a set of user interface global analysis rules. Each user interface global analysis rule may target a type of user interface global issue at a user interface level. A user interface with a user interface global issue in the target application may be identified through pattern matching the user interface navigation map from the raw test result with each user interface global analysis rule in a set of user interface global analysis rules, respectively. An exemplary process for identifying a user interface with a user interface global issue in a target application will be illustrated later in conjunction with FIG. 4 The set of application analysis rules may also include a set of user interface component analysis rules. Each user interface component analysis rule may target a type of user interface component issue at a user interface component level. A user interface component with a user interface component issue in the target application may be identified through pattern matching the page source from the raw test result with each user interface component analysis rule in the set of user interface component analysis rules, respectively. An exemplary process for identifying a user interface component with a user interface component issue in a target application will be illustrated later in conjunction with FIG. 5.

[0046] At 306, a user interface improvement suggestion for the user interface identified at 304 may be generated. The user interface improvement suggestion may include a user interface global improvement suggestion for a user interface with a user interface global issue. An exemplary process for generating a user interface global improvement suggestion will be illustrated later in conjunction with FIG. 4. The user interface improvement suggestion may also include a user interface component improvement suggestion for a user interface component with a user interface component issue. The exemplary process for generating a user interface component improvement suggestion will be illustrated later in conjunction with FIG. 5.

[0047] After the step 302, the process 300 may also proceed to a step 308. At 308, an error message in the raw test result may be identified. For example, an error message in the test log may be identified. At 310, an error reason and/or a fix suggestion for the error message may be generated. An exemplary process for identifying an error message in a test log and generating an error reason and a fix suggestion for the error message will be illustrated later in conjunction with FIG. 8.

[0048] After performing the step 306 and/orthe step 310, the process 300 may proceed to 312. At 312, a test report for the target application may be constructed based on the user interface improvement suggestion, the error reason, and/or the fix suggestion. The test report may be constructed through combining the user interface improvement suggestion, the error reason, and/or the fix suggestion together. Such a test report can visually present issues existed in the target application from a plurality of aspects, as well as effective suggestion for resolving those issues.

[0049] After the step 302, the process 300 may also proceed to a step 314. At 314, a test case for the target application may be generated. At 316, test data for the test case may be generated to further ennch the test case. An exemplary process for generating a test case and test data will be illustrated later in conjunction with FIG. 11.

[0050] It should be appreciated that the process for test result analysis for automated testing described above in conjunction with FIG. 3 is merely exemplary. Depending on actual application requirements, the steps in the process for test result analysis for automated testing may be replaced or modified in any manner, and the process may include more or fewer steps. Further, the specific order or hierarchy of steps in the process 300 is merely exemplary, and the process for test result analysis for automated testing may be performed in a different order than the order described.

[0051] FIG. 4 illustrates an exemplary process 400 for identifying a user interface with a user interface global issue in a target application and generating a user interface global improvement suggestion according to an embodiment of the present disclosure. The process 400 may correspond to one case of the steps 304 and 306 in FIG. 3.

[0052] A test execution unit 410 and an intelligent exploration test executor 412 may correspond to the test execution unit 150 and the intelligent exploration test executor 152 of FIG. 1, respectively. A user interface navigation map 414 output by the test execution unit 410 may be obtained. The user interface navigation map 414 may be provided to a user interface global analyzer 430 in a test result processing unit 420. The test result processing unit 420 and the user interface global analyzer 430 may correspond to the test result processing unit 160 and the user interface global analyzer 161 in FIG. 1, respectively. The user interface global analyzer 430 may have a set of preset user interface global analysis rules. Each user interface global analysis rule may target a type of user interface global issue at a user interface level. The set of user interface global analysis rules may include, e.g., a page depth rule 432, a page loading time rule 434, a first registration experience rule 436, etc. A user interface 438 with a user interface global issue in the target application may be identified through pattern matching the user interface navigation map 414 with the page depth rule 432, the page loading time rule 434, the first registration experience rule 436, etc., respectively.

[0053] The page depth rule 432 is intended to identify a user interface in the target application with high interactivity but hidden too deep in an interaction path. A user interface with high interactivity should be easily discovered by a user and thus should be located at the front of an interaction path. If a user interface has high interactivity but hidden too deep in an interaction path, the user interface is in need of improvement. According to the page depth rule 432, for each user interface in the target application, the number of interactive user interface components on the user interface may be counted. The interactive user interface component is a user interface component that can interact with the user, such as button, input box, selection box, etc. It may be determined whether the number of interactive user interface components is greater than a predefined number threshold. If it is determined that the number is greater than the number threshold, the user interface may be identified as a user interface with high interactivity. Subsequently, the depth of the user interface may be calculated. Herein, the depth of a user interface refers to the minimum number of jumps required to reach the user interface starting from a launching interface. The depth of the user interface may be calculated according to the user interface navigation map. Take the user interface navigation map 200 in FIG. 2 as an example. The user interface 1 is a launching interface, which may have a depth of 0. The user interface 5 may be reached by a minimum of two jumps starting from the user interface 1. For example, the user interface 5 may be reached by performing the action A and the action D. The user interface 5 may have a depth of 2. It may be determined whether the depth of the user interface is greater than a depth threshold. If it is determined that the depth is greater than the depth threshold, then it indicates that the user interface is hidden too deep. In this case, the user interface may be identified as a user interface with high interactivity but hidden too deep in the interaction path.

[0054] The page loading time rule 434 is intended to identify a user interface with a long loading time in the target application. According to the page loading time rule 434, for each user interface in the target application, the loading time of the user interface may be obtained. The loading time of each user interface may be obtained, e.g., from the test log. It may be determined whether the loading time is greater than a predefined time threshold. If it is determined that the loading time is greater than the time threshold, then the user interface may be identified as a user interface with a long loading time.

[0055] The first registration experience rule 436 is intended to identify user interfaces in the target application that cause a first registration experience flow to be too long. According to the first registration experience rule 436, an interaction path corresponding to the first registration experience flow may be identified from a plurality of interaction paths in the user interface navigation map 414. The length of the interaction path may be calculated. Herein, the length of an interaction path refers to the number of user interfaces that constitute the interaction path It may be determined whether the length is greater than a length threshold. If it is determined that the length is greater than the length threshold, then the user interfaces that constitute the interaction path may be identified as user interfaces that cause the first registration experience flow to be too long.

[0056] After identifying the user interface 438 with the user interface global issue, a user interface global improvement suggestion 440 for the user interface 438 may be generated. A set of predefined user interface global improvement suggestions corresponding to the set of user interface global analysis rules may be obtained. For each user interface global analysis rule, a user interface global improvement suggestion corresponding to the user interface global analysis rule may be predefined. For example, a user interface global improvement suggestion corresponding to the page depth rule may be “Please adjust the page to the front end of the interaction path”; a user interface global improvement suggestion corresponding to the page loading time rule may be “Please check the response time of the server; please optimize the number of elements to be loaded”; a user interface global improvement suggestion corresponding to the first registration experience rule may be “Please simplify the first registration experience flow”; etc. After identifying the user interface with the user interface global issue, a user interface global analysis rule for identifying the user interface may be determined. Subsequently, a user interface global improvement suggestion corresponding to the determined user interface global analysis rule may be extracted from the set of user interface global improvement suggestions. The extracted user interface global improvement suggestion may be used as a user interface global improvement suggestion 440 for the user interface 438.

[0057] Through the process 400, the user interface 438 with the user interface global issue in the target application can be identified, and the user interface component global improvement suggestion 440 for the user interface component 438 can be generated. This can enable a developer or tester of the target application to have a clear understanding of which user interface or user interfaces in the target application need to be improved and how to improve them.

[0058] It should be appreciated that the process described above in conjunction with FIG. 4 for identifying the user interface with the user interface global issue and generating the user interface global improvement suggestion is merely exemplary. Depending on actual application requirements, the steps in the process for identifying the user interface with the user interface global issue and generating the user interface global improvement suggestion may be replaced or modified in any manner, and the process may include more or fewer steps. For example, in the process 400, the user interface global improvement suggestion 440 is generated through the user interface global analyzer 430, but the embodiments of the present disclosure are not limited to this. In some embodiments, the user interface global improvement suggestion 440 may also be generated through a large language model. In addition, the user interface global analysis rules shown in FIG. 4 are merely exemplary . Depending on actual application requirements, the user interface global analyzer 430 may be preconfigured with more or fewer user interface global analysis rules.

[0059] FIG. 5 illustrates an exemplary process 500 for identifying a user interface component with a user interface component issue in a target application and generating a user interface component improvement suggestion according to an embodiment of the present disclosure. The process 500 may correspond to one case of the steps 304 and 306 in FIG. 3.

[0060] A test execution unit 510 and an intelligent exploration test executor 512 may correspond to the test execution unit 1 0 and the intelligent exploration test executor 152 in FIG. 1, respectively. A page source 514 output by the test execution unit 510 may be obtained. The page source 514 may include a layout of each user interface in the target application, attributes of each user interface component, etc. The attributes of the user interface component may include name, ty pe, shape, font color, background color, etc. of the user interface component. The page source 514 may be provided to a user interface component analyzer 530 in a test result processing unit 520. The test result processing unit 520 and the user interface component analyzer 530 may correspond to the test result processing unit 160 and the user interface component analyzer 162 in FIG. 1, respectively. The user interface component analyzer 162 may have a set of preset user interface component analysis rules. Each user interface component analysis rule may target a type of user interface component issues at a user interface component level. The set of user interface component analysis rules may include, e g., a user interface component discoverability rule 532, a user interface component accessibility rule 534, a user interface component interactivity rule 536, etc. A user interface component 538 with a user interface component issue in the target application may be identified through pattern matching the page source 514 with the user interface component discoverability rule 532, the user interface component accessibility rule 534, the user interface component interactivity rule 536, etc., respectively.

[0061] The user interface component discoverability rule 532 is intended to identify a user interface component in the target application whose discoverability does not meet a requirement. According to the user interface component discoverability rule 532, for each user interface component in the target application, an average revenue of the user interface component may be calculated. The average revenue of the user interface component may be calculated based on an interaction frequency of the user interface component and resources that can be obtained after each interaction. An interaction frequency of a user interface component refers to the number of interactions with the user interface component within predefined unit time dunng the execution of automated testing on the target application. Resources that can be obtained after an interaction may include a function that can be implemented, a page that can be reached, etc. after the interaction with the user interface component dunng the execution of the automated testing on the target application. An average revenue of a user interface component may be used to evaluate an importance of the user interface component. Subsequently, the discoverability of the user interface component may be evaluated according to the page source 514. The page source 514 includes attributes of individual user interface component, such as size, font color, background color, etc., of the user interface component. The discoverability of the user interface component may be evaluated through these attributes. Next, it may be determined whether the discoverability of the user interface component matches the average revenue of the user interface component. For example, when an average revenue of a user interface component is high, its discoverability may be strong, so that it can be easily discovered by a user. When an average revenue of a user interface component is low, its discoverability may be weak If a discoverability of a user interface component with a high average revenue is weak, or a discoverability of a user interface component with a low average revenue is strong, then the discoverability of the user interface component does not match its average revenue. If it is determined that the discoverability of a user interface component does not match its average revenue, then the user interface component may be identified as a user interface component whose discoverability does not meet the requirement.

[0062] The user interface component accessibility rule 534 is intended to identify a user interface component in the target application whose accessibility does not meet a requirement. The user interface component accessibility rule 534 defines conditions for a user interface component to have accessibility, such as a text being readable and having sufficient contrast, an image having a replacement word, a form element having a label, a link having a descriptive text, an interactive element being accessible through a keyboard, etc. For each user interface component in the target application, it may be evaluated, by the page source associated with the user interface component, whether the user interface component satisfies the conditions defined in the user interface component accessibility rule 534. The user interface component that does not satisfy the conditions may be identified as a user interface component whose accessibility does not meet the requirement.

[0063] The user interface component interactivity rule 536 is intended to identify a user interface component in the target application whose interactivity does not meet a requirement. The user interface component interactivity rule 536 defines conditions for a user interface component to have interactivity, such as a size requirement of the user interface component, a distance requirement between a location of the user interface component in the user interface and edges of the user interface, etc. For each user interface component in the target application, it may be evaluated, by the page source associated with the user interface component, whether the user interface component satisfies the conditions defined in the user interface component interactivity rule 536. The user interface component that does not satisfy the conditions may be identified as a user interface component whose interactivity does not meet the requirement.

[0064] After identifying the user interface component 538 with the user interface component issue, a user interface component improvement suggestion 552 for the user interface component may be generated. For example, a prompt 542 may be created based on the page source 514 and the user interface component 538 through a prompt creator 540. The prompt creator 540 may correspond to the prompt creator 165 in FIG. 1. Subsequently, the user interface component improvement suggestion 552 for the user interface component 538 may be generated based on the prompt 542 through a large language model 550. The large language model 550 may correspond to the large language model 190 in FIG. 1.

[0065] FIG. 6 illustrates an exemplary process 600 for creating a prompt for a user interface component improvement suggestion according to an embodiment of the present disclosure. The process 600 may be performed through the prompt creator 540 in FIG. 5.

[0066] At 602, key information associated with a user interface component may be extracted from a page source. The key information associated with the user interface component is, e.g., a page source of a user interface in which the user interface component is located.

[0067] At 604, a role of the large language model may be designated according to the page source and/or the user interface component. A set of roles may be predefined, and each role may correspond to an input of the prompt creator. The prompt creator may designate the role of the large language model according to the input it received. For example, when the prompt creator receives a page source and/or a user interface component, the role of the large language model may be designated as "user interface designer”; when the prompt creator receives a test log and/or an error message, the role of the large language model may be designated as “application developer”; when the prompt creator receives a test case, the role of the large language model may be designated as “application tester”; etc. At 604, the prompt creator receives the page source and the user interface component, and may designate the role of the large language model as “user interface designer”. The role may be expressed by a statement in natural language form, such as “I would like you to be a user interface designer”.

[0068] At 606, a response requirement for the large language model may be specified according to the role of the large language model. The response requirement may describe how the large language model should provide a response. The response requirement for the large language model may be a predefined statement in natural language form, and may correspond to the role of the large language model. For example, when the role of the large language model is “user interface designer”, the response requirement for the large language model may be “I would like you to provide a user interface component improvement suggestion”.

[0069] At 608, a prompt may be created based on the key information, the role of the large language model, the response requirement for the large language model, etc. For example, the prompt may be created through combining the key information, the role of the large language model, the response requirement for the large language model, etc. together.

[0070] Referring back to FIG. 5, after creating the prompt 542 through the prompt creator 540, the user interface component improvement suggestion 552 for the user interface component 538 with the user interface component issue may be generated based on the prompt 542 through the large language model 550. The large language model 550 may generate the user interface component improvement suggestion 552 based on the key information associated with the user interface component 538 in the page source 514, the role of the large language model 550, the response requirement for the large language model 550, and other information contained in the prompt 542, and using its semantic understanding capability, inference capability, large data support capability, etc. The user interface component improvement suggestion 552 is in natural language form. For example, when the user interface component 538 is a user interface component whose discoverability does not meet a requirement, such as when the user interface component 538 has a high average revenue but its discoverability is weak, the user interface component improvement suggestion 552 for the user interface component 538 may be a suggestion to increase the size of the user interface component 538. In addition, if the average revenue of the user interface component 538 is low, then the user interface component improvement suggestion 552 for the user interface component 538 may be to weaken or even remove the user interface component 538.

[0071] Through the process 500, the user interface component 538 with the user interface component issue in the target application can be identified, and the user interface component improvement suggestion 552 for the user interface component 538 can be generated. This can enable a developer or tester of the target application to have a clear understanding of which user interface component or user interface components of the target application need to be improved and how to improve them.

[0072] It should be appreciated that the process described above in conjunction with FIG. 5 and FIG. 6 for identifying the user interface component with the user interface component issue and generating the user interface component improvement suggestion is merely exemplary. Depending on actual application requirement, the steps in the process for identifying the user interface component with the user interface component issue and generating the user interface component improvement suggestion may be replaced or modified in any manner, and the process may include more or fewer steps. For example, in the process 500, the user interface component 538 with the user interface component issue in the target application is first identified through the user interface component analyzer 530, and then the user interface component improvement suggestion 552 for the user interface component 538 is generated through the large language model 550, but the embodiments of the present disclosure are not limited thereto. In some embodiments, the user interface component 538 with the user interface component issue in the target application may be identified and the user interface component improvement suggestion 552 for the user interface component 538 may be generated through the user interface component analyzer 530 without using the large language model 550. Further, the specific order or hierarchy of the steps in the process 500 is merely exemplary, and the process for generating the user interface component improvement suggestion may be performed in a different order than the order described. In addition, the user interface component analysis rules shown in FIG. 5 are merely exemplary. Depending on actual application requirements, the user interface component analyzer 530 may be preconfigured with more or fewer user interface component analysis rules.

[0073] FIG. 7 is an exemplary chat user interface 700 illustrating a prompt and a user interface component improvement suggestion according to an embodiment of the present disclosure. The chat user interface 700 is a chat user interface between a user and a robot The user may be a tester or developer of a target application. The robot may be a large language model.

[0074] A prompt 702 is input into the chat user interface 700 by the user in order to obtain a user interface component improvement suggestion from the robot. The prompt 702 may be created through the process 600. The prompt 702 is “I would like you to be a user interface designer. I will provide a page source for an Android application in XML format. I would like you to provide an improvement suggestion of a user interface component. The page source is: <?xml version- 1. O' encoding='UTF-8' standalone-yes' ? > ... </hierarchy>”. The prompt 702 contains a role of the large language model, i.e., “user interface designer”. The prompt 702 also contains a response requirement for the large language model, i.e. “I would like you to provide an improvement suggestion of a user interface component”. The prompt 702 also contains a piece of page source. It should be appreciated that for the sake of brevity of the example, only a portion of the page source is shown in the chat user interface 700. In actual applications, the prompt 702 usually contains a page source with more content.

[0075] A user interface component improvement suggestion 704 is generated based on the prompt 702 by the large language model. The user interface component improvement suggestion 704 is “Thanks for providing the page source of the Android application. The following are improvement suggestions for the user interface components: 1. Change the ‘Settings’ button to blue to make it easier to be found by users. 2. Increase the size of the ‘Notification’ icon to make it easier to be clicked by users”. The user interface component improvement suggestion 704 generated by the large language model may correspond to the role of the large language model, the response requirement for the large language model, etc., as indicated in the prompt 702.

[0076] FIG. 8 illustrates an exemplary process 800 for identifying an error message in a test log and generating an error reason and a fix suggestion for the error message according to an embodiment of the present disclosure. The process 800 may correspond to the steps 308 and 310 in FIG. 3.

[0077] A test execution unit 810 and an intelligent exploration test executor 812 may correspond to the test execution unit 150 and the intelligent exploration test executor 152 in FIG. 1, respectively. A test log 814 output by the test execution unit 810 may be obtained. The test log 814 may be produced during the execution of automated testing on the target application through the automated testing system. The test log 814 may include a test log produced by the target application, a test log produced by the automated testing system, etc. The test log 814 may be provided to an error message analyzer 830 in a test result processing unit 820. The test result processing unit 820 and the error message analyzer 830 may correspond to the test result processing unit 160 and the error message analyzer 163 in FIG. 1 , respectively. An error messages 832 in the test log 814 may be identified through the error message analyzer 830. The error message analyzer 830 may identify the error message 832 through performing a known log parsing technique.

[0078] Subsequently, an error reason 852 and/or a fix suggestion 854 for the error message 832 may be generated. The error reason 852 and the fix suggestion 854 may be in natural language form. For example, a prompt 842 may be created based on the test log 814 and the error message 832 through a prompt creator 840. The prompt creator 840 may correspond to the prompt creator 165 in FIG. 1. Subsequently, the error reason 852 and/or the fix suggestion 854 may be generated based on the prompt 842 through a large language model 850. The large language model 850 may correspond to the large language model 190 in FIG. 1.

[0079] FIG. 9 illustrates an exemplary process 900 for creating a prompt for an error reason and a fix suggestion according to an embodiment of the present disclosure. The process 900 may be performed through the prompt creator 840 in FIG. 8.

[0080] At 902, key information associated with an error message may be extracted from a test log. The key information associated with the error message may be, e.g., information associated with the target application, information whose time is within a predetermined time period around the time point when the error message occurred, etc.

[0081] At 904, a role of a large language model may be designated according to the test log and/or the error message. As described above in conjunction with FIG. 6, a set of roles may be predefined, and each role may correspond to an input to the prompt creator. The prompt creator may designate the role of the large language model according to the input it received. At 904, the prompt creator receives the test log and the error message, and it may designate the role of the large language model as “application developer”. The role may be expressed in a statement in natural language form, such as “I would like you to be an application developer”.

[0082] At 906, a response requirement for the large language model may be specified according to the role of the large language model. The response requirement may describe how the large language model should provide a response. The response requirement for the large language model may be a predefined statement in natural language form and may correspond to the role of the large language model. For example, when the role of the large language model is “application developer”, the response requirement for the large language model may be “I would like you to reply with only the error reason and the fix suggestion. Please put the error reason at the beginning of the reply”.

[0083] At 908, a prompt may be created based on the key information, the role of the large language model, the response requirement for the large language model, etc. For example, the prompt may be created through combining the key information, the role of the large language model, the response requirement for the large language model, etc., together.

[0084] Referring back to FIG. 8, after creating the prompt 842 through the prompt creator 840, the error reason 852 and the fix suggestion 854 for the error message 832 may be generated based on the prompt 842 through the large language model 850. The large language model 850 may generate the error reason 852 and the fix suggestion 854 based on the key information associated with the error message 832 in the test log 814, the role of the large language model 850, the response requirement for the large language model 850, and other information contained in the prompt 842, and using its semantic understanding capability, inference capability, large data support capability, etc.

[0085] Through the process 800, the error message 832 in the test log 814 can be i identified and the error reason 852 and the fix suggestions 854 for the error message 832 can be generated. This can enable a developer or tester of the target application to easily locate a vulnerability or fault existed in the target application, and improve the efficiency of resolving the vulnerability or fault.

[0086] It should be appreciated that the process described above in conjunction with FIG. 8 and FIG. 9 for identifying the error message in the test log and generating the error reason and the fix suggestion for the error message is merely exemplary. Depending on actual application requirement, the steps in the process for identifying the error message and generating the error reason and the fix suggestion may be replaced or modified in any manner, and the process may include more or fewer steps.

[0087] FIG. 10 is an exemplary chat user interface 1000 illustrating a prompt and an error reason and a fix suggestion according to an embodiment of the present disclosure. The chat user interface 1000 is a chat user interface between a user and a robot. The user may be a tester or developer of a target application. The robot may be a large language model.

[0088] A prompt 1002 is input into the chat user interface 1000 by the user in order to obtain an error reason and a fix suggestion from the robot. The prompt 1002 may be created through the process 900 The prompt 1002 is “I would like you to be an application developer. I will provide a section of test log. I would like you to reply only with an error reason and a fix suggestion. Please put the error reason at the beginning of the reply. The test log is: E/AndroidRuntime: FATAL EXCEPTION: ... (ActionBarContainer.java:61)”. The prompt 1002 contains a role of the large language model, i.e., “application developer”. The prompt 1002 also contains a response requirement for the large language model, i.e., “I would like you to reply only with an error reason and a fix suggestion. Please put the error reason at the beginning of the reply”. The prompt 1002 further contains a section of test log. It should be appreciated that for the sake of brevity of the example, only a portion of the test log is shown in the chat user interface 1000. In actual application, the prompt 1002 usually contains a test log with more content.

[0089] An error reason and fix suggestion 1004 is generated based on the prompt 1002 by the large language model The error reason and fix suggestion 1004 is “Error reason: the application encountered ajavalang reflect. InvocationTargetExcepti on; Fix suggestion: Please check whether the third-party library or plug-in used in the application is implemented and updated correctly”. The error reason and fix suggestion 1004 generated by the large language model may correspond to the role of the large language model, the response requirement for the large language model, etc., as indicated in the prompt 1002.

[0090] FIG. 11 illustrates an exemplary process 1100 for generating a test case and test data according to an embodiment of the present disclosure The process 1100 may correspond to the steps 314 and 316 in FIG. 3.

[0091] A test execution unit 1110 and an intelligent exploration test executor 1112 may correspond to the test execution unit 150 and the intelligent exploration test executor 152 in FIG. 1, respectively. A user interface navigation map 1114 provided by the test execution unit 1110 may be obtained. The user interface navigation map 1114 may include a plurality of interaction paths. The user interface navigation map 1114 may be provided to a test case generator 1130 in a test result processing unit 1120. The test result processing unit 1120 and the test case generator 1130 may correspond to the test result processing unit 160 and the test case generator 164 of FIG. 1, respectively. The test case generator 1130 may receive a path selection indication 1132, which may indicate a selection of one of the plurality of interaction paths in the user interface navigation map 1114. The selection of the interaction path may be made by a tester of a target application or may be made automatically by a machine learning model based on an understanding of the user interface navigation map 1114. Since the user interface navigation map 1114 is obtained through intelligent exploration testing, it is known what actions are performed to jump among various user interfaces in the interaction path. Accordingly, a series of actions corresponding to the selected interaction path may be obtained. A test case 1134 for the target application may be generated based on the series of actions through the test case generator 1130.

[0092] FIG. 12 illustrates an example of generating a test case according to an embodiment of the present disclosure. A user interface navigation map 1200 is shown at the top of FIG. 12. In the user interface navigation map 1200, a selected interaction path is shown by bolded lines, i.e., an interaction path consisting of a user interface 1, a user interface 2, a user interface 5, a user interface 6, and a user interface 9. Accordingly, a series of actions corresponding to the selected interaction path may be obtained The user interface 1 may be, e.g., a launching interface of the target application, i.e., the user interface presented when launching the target application. As an example, the target application may be launched through activating an application with a package name ‘‘com.microsoft.appmanager’’, thus the user interface 1 is triggered. The user interface 2 may be triggered through applying an action A to the user interface 1. The action A is, e.g., “Click a button with words ‘Link your phone and computer’”. The user interface 5 may be triggered through applying an action D to the user interface 2. The action D is, e.g., “Click a button with a word ‘Continue’”. The user interface 6 may be tnggered through applying an action E to the user interface 5. The action E is, e.g., “Input a program to be searched in an input box and click a button with a word ‘Search’”. The user interface 9 may be triggered through applying an action F to the user interface 6. The action F is, e.g., “Click a text view containing ‘Debug Page’”. A test case for the target application may be generated based on these actions. An example test case 1250 is shown at the bottom of FIG. 12. The test case 1250 contains five test steps including steps 1 to 5. The step 1 may correspond to the action used to launch the target application, i.e., “activate an application with a package name ‘com.microsoft.appmanager’”. The steps 2 to 5 may correspond to the action A, the action D, the action E, and the action F, respectively.

[0093] Referring back to FIG. 11, the test case 1134 may be generated through the test case generator 1130. The test case 1134 may contain a step for inputting test data in a particular user interface. The embodiments of the present disclosure propose that test data 1152 for the test case 1134 may be automatically generated to further enrich the test case 1134. A prompt 1142 may be created based on the test case 1134 through a prompt creator 1140. The prompt creator 1140 may correspond to the prompt creator 165 in FIG. 1. Subsequently, test data 1152 for the test case 1134 may be generated based on the prompt 1142 through a large language model 1150. The large language model 1150 may correspond to the large language model 190 in FIG. 1.

[0094] FIG. 13 illustrates an exemplary process 1300 for creating a prompt for test data according to an embodiment of the present disclosure. The process 1300 may be performed through the prompt creator 1140 in FIG. 11.

[0095] At 1302, a user interface for inputting test data may be determined according to a test case. A step that require to input test data may be determined first according to the test case. Subsequently, a user interface corresponding to the step may be identified. Take the test case 1250 in FIG. 12 as an example, the step 4 is “Input a program to be searched in an input box and click a button with a word “Search”’. This step is required to input test data in a user interface The user interface corresponding to this step is the user interface 5.

[0096] At 1304, a scenario category of the user interface for inputting test data may be obtained. The scenario category of the user interface may be identified through a machine learning model capable of performing a multi-class task during the execution of intelligent exploration testing on a target application. The scenario category may include, e g., a login scenario, a video playing scenario, an information filling scenario, a search scenario, a setting scenario, etc. Each scenario category may be further subdivided For example, the search scenario may also be subdivided into news search, video search, program search, etc.

[0097] At 1306, a role of a large language model may be designated according to the test case. As described above in conjunction with FIG. 6, a set of roles may be predefined, and each role may correspond to an input of the prompt creator. The prompt creator may designate the role of the large language model according to the input it received. At step 1306, the prompt creator receives a test case, and may designate the role of the large language model as “application tester”. The role may be expressed by a statement in natural language form, such as “I would like you to be an application tester”.

[0098] At 1308, a response requirement for the large language model may be specified according to the scenario category and/or the role of the large language model. The response requirement may describe how the large language model should provide a response. The response requirement for the large language model may be a predefined statement in natural language form, and may correspond to the scenario category of the user interface for inputting test data and/or the role of the large language model. For example, when the scenario category of the user interface for inputting test data is a search scenario and the role of the large language model is “application tester”, the response requirement may be “I hope you reply with a search keyword only”.

[0099] At 1310, a prompt may be created based on the scenario category, the role of the large language model, the response requirement for the large language model, etc. For example, the prompt may be created through combining the scenano category, the role of the large language model, the response requirement for the large language model, etc., together.

[00100] Referring back to FIG. 11, after creating the prompt 1142 through the prompt creator 1140, test data 1152 for the user interface component for inputting test data may be generated based on the prompt 1142 through the large language model 1150. The large language model 1150 may generate the test data 1152 based on the scenario category of the user interface for inputting test data, the role of the large language model 1150, the response requirement for the large language model 1150, and other information contained in the prompt 1142, and using its semantic understanding capability, inference capability, large data support capability, etc. The test data 1152 may be in natural language form. For example, when the scenario category of the user interface for inputting test data is a search scenario, the test data 1152 may be a search keyword. The generated test case 1134 and test data 1152 may be provided to a test execution unit in an automated testing system for performing automated testing for the target application. Alternatively or additionally, the test case 1134 and test data 1152 may be exported to a tester, e g., via a back- end system interactive user interface, for performing further optimization thereof.

[00101] Through the process 1100, the test case and corresponding test data that can explore the desired interaction path can be automatically generated, thereby improving the efficiency of automated testing.

[00102] It should be appreciated that the process for generating the test case and the test data described above in conj unction with FIG. 11 to FIG. 13 is merely exemplary. Depending on actual application requirements, the steps mthe process for generating the test case and the test data may be replaced or modified in any manner, and the process may include more or fewer steps.

[00103] FIG. 14 is an exemplary chat user interface 1400 illustrating a prompt and test data according to an embodiment of the present disclosure. The chat user interface 1400 is a chat user interface between a user and a robot. The user may be a tester or developer of a target application. The robot may be a large language model.

[00104] A prompt 1402 is input into the chat user interface 1400 by the user in order to obtain test data from the robot. The prompt 1402 may be created through the process 1300. The prompt 1402 is “I would like you to be an application tester. I will provide a scenario category of a user interface. I hope you reply with a search keyword only. The scenario category of the user interface is: program search”. The prompt 1402 contains a role of the large language model, i.e., “application tester”. The prompt 1402 also contains a response requirement for the large language model, i.e., “I hope you reply with a search keyword only”. The prompt 1402 further contains a scenario category of a user interface for inputting test data, i.e., “program search”.

[00105] The test data 1404 is generated based on the prompt 1402 by the large language model. The test data 1404 is “Browser”. The test data generated by the large language model may correspond to the scenario category of the user interface for inputting test data, the role of the large language model, the response requirement for the large language model, etc., as indicated in the prompt 1402. This test data may be input by the automated testing system into the corresponding user interface of the target application, in order to perform automated testing for the target application.

[00106] FIG. 15 is a flowchart of an exemplary method 1500 for test result analysis for automated testing according to an embodiment of the present disclosure

[00107] At 1510, a raw test result generated through performing automated testing on a target application may be obtained, the target application including a plurality of user interfaces, the raw test result including a user interface navigation map of the target application, the user interface navigation map including a plurality of interaction paths, and each interaction path consisting of a portion of the plurality of user interfaces.

[00108] At 1520, a set of application analysis rules may be obtained, each application analysis rule targeting a type of user interface issue.

[00109] At 1530, a user interface with a user interface issue in the target application may be identified through pattern matching the raw test result with each application analysis rule in the set of application analysis rules, respectively.

[00110] At 1540, a user interface improvement suggestion for the user interface may be generated.

[00111] At 1550, a selection of one of the plurality of interaction paths may be received.

[00112] At 1560, a series of actions corresponding to the selected interaction path may be obtained.

[00113] At 1570, a test case for the target application may be generated based on the series of actions.

[00114] In an implementation, the set of application analysis rules may include a set of user interface global analysis rules. Each user interface global analysis rule may target a type of user interface global issue at a user interface level. The identifying a user interface with a user interface issue in the target application may comprise: identifying a user interface with a user interface global issue in the target application through pattern matching the user interface navigation map with each user interface global analysis rule in the set of user interface global analysis rules, respectively.

[00115] The set of user interface global analysis rules may include a page depth rule. The page depth rule is used to identify a user interface with high interactivity but hidden too deep in an interaction path. The identifying a user interface with a user interface issue in the target application comprises, for each user interface in the target application: counting the number of interactive user interface components on the user interface; determining whether the number is greater than a number threshold; identifying the user interface as a user interface with high interactivity in response to determining that the number is greater than the number threshold; calculating a depth of the user interface; determining whether the depth is greater than a depth threshold; and in response to determining that the depth is greater than the depth threshold, identifying the user interface as a user interface with high interactivity but hidden too deep in the interaction path.

[00116] The set of user interface global analysis rules may include a page loading time rule. The page loading time rule may be used to identify a user interface with a long loading time. The identifying a user interface with a user interface global issue in the target application may comprise, for each user interface in the target application: obtaining the loading time of the user interface; determining whether the loading time is greater than a time threshold; and in response to determining that the loading time is greater than the time threshold, identifying the user interface as a user interface with a long loading time.

[00117] The set of user interface global analysis rules may include a first registration experience rule. The first registration experience rule may be used to identify user interfaces that cause a first registration experience flow to be too long. The identifying a user interface with a user interface global issue in the target application may comprise: identifying an interaction path corresponding to the first registration experience flow from the plurality of interaction paths; calculating the length of the interaction path; determining whether the length is greater than a length threshold; and in response to determining that the length is greater than the length threshold, identifying user interfaces that constitute the interaction path as user interfaces that cause the first registration experience flow to be too long.

[00118] The generating a user interface improvement suggestion for the user interface may comprise: obtaining a set of predefined user interface global improvement suggestions corresponding to the set of user interface global analysis rules; determining a user interface global analysis rule for identifying the user interface; and extracting a user interface global improvement suggestion corresponding to the determined user interface global analysis rule from the set of user interface global improvement suggestions.

[00119] In an implementation, each user interface in the plurality of user interfaces may include at least one user interface component. The raw test result may further include a page source of the target application. The page source may include a layout of each user interface and/or attributes of each user interface component.

[00120] The set of application analysis rules may include a set of user interface component analysis rules. Each user interface component analysis rule may target a type of user interface component issue at a user interface component level. The identifying a user interface with a user interface issue in the target application may comprise: identifying a user interface component with a user interface component issue in the target application through pattern matching the page source with each user interface component analysis rule in the set of user interface component analysis rules, respectively.

[00121] The user interface component analysis rule may include a user interface component discoverability rule. The user interface component discoverability rule may be used to identify a user interface component whose discoverability does not meet a requirement. The identifying a user interface component with a user interface component issue in the target application may comprise, for each user interface component in the target application: calculating an average revenue of the user interface component based on an interaction frequency of the user interface component and/or resources that can be obtained after each interaction; evaluating the discoverability of the user interface component according to the page source; determining whether the discoverability matches the average revenue; and

[00122] in response to determining that the discoverability does not match the average revenue, identifying the user interface component as a user interface component whose discoverability does not meet a requirement.

[00123] The user interface component analysis rule may include a user interface component accessibility rule or a user interface component interactivity rule. The user interface component accessibility rule may be used to identify a user interface component whose accessibility does not meet a requirement. The user interface component interactivity rule may be used to identify a user interface component whose interactivity does not meet a requirement.

[00124] The generating a user interface improvement suggestion for the user interface may comprise: creating a prompt based on the page source and the user interface component; and generating a user interface component improvement suggestion for the user interface component based on the prompt through a large language model.

[00125] The creating a prompt may comprise: extracting, from the page source, key information associated with the user interface component; designating a role of the large language model according to the page source and/or the user interface component; specifying a response requirement for the large language model according to the role; and creating the prompt based on the key information, the role, and the response requirement.

[00126] In an implementation, the raw test result may further include a test log for the target application. The method 1500 may further comprise: identifying an error message in the test log; and generating an error reason and/or a fix suggestion for the error message.

[00127] The generating an error reason and/or a fix suggestion for the error message may comprise: creating a prompt based on the test log and the error message; and generating the error reason and/or the fix suggestion based on the prompt through a large language model.

[00128] The creating a prompt may comprise: extracting key information associated with the error message from the test log; designating a role of the large language model according to the test log and/or the error message; specifying a response requirement for the large language model according to the role; and creating the prompt based on the key information, the role, and the response requirement.

[00129] In an implementation, the method 1500 may further comprise: creating a prompt based on the test case; and generating test data for the test case based on the prompt through a large language model.

[00130] The creating a prompt may comprise: determining a user interface for inputting test data according to the test case; obtaining a scenario category of the user interface for inputting test data; designating a role of the large language model according to the test case; specifying a response requirement for the large language model according to the scenario category and/or the role; and creating the prompt based on the scenario category; the role, and the response requirement.

[00131] It should be appreciated that the method 1500 may also comprise any other steps/processes for test result analysis for automated testing according to the embodiments of the present disclosure as described above.

[00132] FIG. 16 illustrates an exemplary apparatus 1600 for test result analysis for automated testing according to an embodiment of the present disclosure.

[00133] The apparatus 1600 may comprise: a raw test result obtaining module 1610, for obtaining a raw test result generated through performing automated testing on a target application, the target application including a plurality of user interfaces, the raw test result including a user interface navigation map of the target application, the user interface navigation map including a plurality of interaction paths, and each interaction path consisting of a portion of the plurality of user interfaces; an application analysis rule obtaining module 1620, for obtaining a set of application analysis rules, each application analysis rule targeting a type of user interface issue; a user interface identifying module 1630, for identifying a user interface with a user interface issue in the target application through pattern matching the raw test result with each application analysis rule in the set of application analysis rules, respectively; a user interface improvement suggestion generating module 1640, for generating a user interface improvement suggestion for the user interface; a path selection receiving module 1650, for receiving a selection of one of the plurality of interaction paths; an action obtaining module 1660, for obtaining a series of actions corresponding to the selected interaction path; and a test case generating module 1670, for generating a test case for the target application based on the series of actions. In addition, the apparatus 1600 may further comprise any other modules configured for test result analysis for automated testing according to the embodiments of the present disclosure as described above.

[00134] FIG. 17 illustrates another exemplary apparatus 1700 for test result analysis for automated testing according to an embodiment of the present disclosure

[00135] The apparatus 1700 may comprise: a processor 1710; and a memory 1720 storing computer-executable instructions. The computer-executable instructions, when executed, may cause the processor 1710 to: obtain a raw test result generated through performing automated testing on a target application, the target application including a plurality of user interfaces, the raw test result including a user interface navigation map of the target application, the user interface navigation map including a plurality of interaction paths, and each interaction path consisting of a portion of the plurality of user interfaces, obtain a set of application analysis rules, each application analysis rule targeting a type of user interface issue, identify a user interface with a user interface issue in the target application through pattern matching the raw test result with each application analysis rule in the set of application analysis rules, respectively, generate a user interface improvement suggestion for the user interface, receive a selection of one of the plurality of interaction paths, obtain a series of actions corresponding to the selected interaction path, and generate a test case for the target application based on the series of actions.

[00136] In an implementation, the raw test result may further include a test log for the target application. The computer-executable instructions, when executed, further cause the processor 1710 to: identify an error message in the test log; and generate an error reason and/or a fix suggestion for the error message.

[00137] It should be appreciated that the processor 1710 may also perform any other steps/processes of the method for test result analysis for automated testing according to the embodiments of the present disclosure as described above.

[00138] The embodiments of the present disclosure present a computer program product for test result analysis for automated testing, comprising a computer program, the computer program being executed by the processor to: obtain a raw test result generated through performing automated testing on a target application, the target application including a plurality of user interfaces, the raw test result including a user interface navigation map of the target application, the user interface navigation map including a plurality of interaction paths, and each interaction path consisting of a portion of the plurality of user interfaces; obtain a set of application analysis rules, each application analysis rule targeting a type of user interface issue; identify a user interface with a user interface issue in the target application through pattern matching the raw test result with each application analysis rule in the set of application analysis rules, respectively; generate a user interface improvement suggestion for the user interface; receive a selection of one of the plurality of interaction paths; obtain a series of actions corresponding to the selected interaction path; and generate a test case for the target application based on the series of actions. In addition, the computer program may be further performed to implement any other steps/processes of the method for test result analysis for automated testing according to the embodiments of the present disclosure as described above.

[00139] The embodiments of the present disclosure may be embodied in a computer-readable medium. The computer-readable medium may comprise instructions that, when executed, cause a processor to: obtain a raw test result generated through performing automated testing on a target application, the target application including a plurality of user interfaces, the raw test result including a user interface navigation map of the target application, the user interface navigation map including a plurality of interaction paths, and each interaction path consisting of a portion of the plurality of user interfaces; obtain a set of application analysis rules, each application analysis rule targeting a type of user interface issue; identify a user interface with a user interface issue in the target application through pattern matching the raw test result with each application analysis rule in the set of application analysis rules, respectively; generate a user interface improvement suggestion for the user interface; receive a selection of one of the plurality of interaction paths; obtain a series of actions corresponding to the selected interaction path; and generate a test case for the target application based on the series of actions. In addition, the instructions, when executed, may also cause the processor to perform any other steps/processes of the method for test result analysis for automated testing according to the embodiments of the present disclosure as described above.

[00140] It should be appreciated that all the operations in the methods described above are merely exemplary, and the present disclosure is not limited to any operations in the methods or sequence orders of these operations, and should cover all other equivalents under the same or similar concepts. In addition, the articles “a” and “an” as used in this specification and the appended claims should generally be construed to mean “one” or “one or more” unless specified otherwise or clear from the context to be directed to a singular form. [00141] It should also be appreciated that all the modules in the apparatuses described above may be implemented in various approaches. These modules may be implemented as hardware, software, or a combination thereof. Moreover, any of these modules may be further functionally divided into sub-modules or combined together.

[00142] Processors have been described in connection with vanous apparatuses and methods. These processors may be implemented using electronic hardware, computer software, or any combination thereof. Whether such processors are implemented as hardware or software will depend upon the particular application and overall design constraints imposed on the system. By way of example, a processor, any portion of a processor, or any combination of processors presented in the present disclosure may be implemented with a microprocessor, microcontroller, digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a state machine, gated logic, discrete hardware circuits, and other suitable processing components configured for performing the various functions described throughout the present disclosure. The functionality of a processor, any portion of a processor, or any combination of processors presented in the present disclosure may be implemented with software being executed by a microprocessor, microcontroller, DSP, or other suitable platform.

[00143] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, threads of execution, procedures, functions, etc. The software may reside on a computer-readable medium. A computer-readable medium may include, by way of example, memory such as a magnetic storage device (e g., hard disk, floppy disk, magnetic strip), an optical disk, a smart card, a flash memory device, random access memory (RAM), read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), a register, or a removable disk. Although memory is shown separate from the processors in the various aspects presented throughout the present disclosure, the memory may be internal to the processors, e.g., cache or register.

[00144] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents to the elements of the various aspects described throughout the present disclosure that are known or later come to be known to those of ordinary skilled in the art are expressly incorporated herein and intended to be encompassed by the claims.

Claims

1. A method for result analysis in natural language for intelligent exploration testing, comprising: obtaining an original test result produced through performing intelligent exploration testing on a target application, the target application including a plurality of user interfaces, each user interface including at least one user interface component; identifying a user interface or a user interface component to be improved in the target application through pattern matching the original test result with a user experience analysis rule; generating test result analysis in natural language corresponding to the user interface or the user interface component to be improved; and building a test report for the target application based on the test result analysis.

2. The method of claim 1, wherein the original test result includes a user interface navigation map of the target application, the user interface navigation map includes a plurality of interaction paths, and each interaction path consists of a portion of the plurality of user interfaces.

3. The method of claim 2, wherein the user expenence analysis rule is a user interface analysis rule, and the identifying a user interface or a user interface component to be improved in the target application compnses: identifying a user interface to be improved in the target application through pattern matching the user interface navigation map with the user interface analysis rule.

4. The method of claim 3, wherein the user interface analysis rule is a page depth rule, and the identifying a user interface to be improved in the target application comprises, for each user interface in the target application: counting the number of interactive user interface components on the user interface; determining whether the number is greater than a number threshold; in response to determining that the number is greater than the number threshold, calculating the depth of the user interface; determining whether the depth is greater than a depth threshold; and in response to determining that the depth is greater than the depth threshold, identifying the user interface as the user interface to be improved.

5. The method of claim 3, wherein the user interface analysis rule is a page loading time rule, and the identifying a user interface to be improved in the target application comprises, for each user interface in the target application: obtaining the loading time of the user interface; determining whether the loading time is greater than a time threshold; and in response to determining that the loading time is greater than the time threshold, identifying the user interface as the user interface to be improved.

6. The method of claim 3, wherein the user interface analysis rule is a first registration experience rule, and the identifying a user interface to be improved in the target application comprises: identifying an interaction path corresponding to a first registration expenence flow from the plurality of interaction paths; calculating the length of the interaction path; determining whether the length is greater than a length threshold; and in response to determining that the length is greater than the length threshold, identifying a user interface that constitutes the interaction path as the user interface to be improved.

7. The method of claim 1, wherein the original test result includes a page source, the page source includes a layout of each user interface in the target application and/or attributes of each user interface component in the target application.

8. The method of claim 7, wherein the user expenence analysis rule is a user interface component analysis rule, and the identifying a user interface or a user interface component to be improved in the target application comprises: identifying a user interface component to be improved in the target application through pattern matching the page source with the user interface component analysis rule.

9. The method of claim 8, wherein the user interface component analysis rule is a user interface component discoverability rule, and the identifying a user interface component to be improved in the target application comprises, for each user interface component in the target application: calculating an average revenue of the user interface component based on an interaction frequency of the user interface component and/or resources that can be obtained after each interaction; evaluating the discoverability of the user interface component according to the page source; determining whether the discoverability matches the average revenue; and in response to determining that the discoverability does not match the average revenue, identifying the user interface component as the user interface component to be improved.

10. The method of claim 8, wherein the user interface component analysis rule is a user interface component accessibility rule or a user interface component interactivity rule.

11. The method of claim 8, wherein the generating test result analysis in natural language corresponding to the user interface or the user interface component to be improved comprises: creating a prompt based on the page source and the user interface component to be improved; and generating, through a large language model, a user interface component improvement suggestion for the user interface component to be improved as test result analysis based on the prompt.

12. The method of claim 1, wherein the original test results include a test log for the target application, and the method further comprises: identifying an error message in the test log; and generating test result analysis in natural language corresponding to the error message.

13. The method of claim 2, further comprising: receiving a selection of one of the plurality of interaction paths; obtaining a series of actions corresponding to the selected interaction path; and generating a test case for the target application based on the series of actions.

14. An apparatus for result analysis in natural language for intelligent exploration testing, comprising: a processor; and a memory storing computer-executable instructions that, when executed, cause the processor to: obtain an original test result produced through performing intelligent exploration testing on a target application, the target application including a plurality of user interfaces, each user interface including at least one user interface component, identify a user interface or a user interface component to be improved in the target application through pattern matching the original test result with a user experience analysis rule, generate test result analysis in natural language corresponding to the user interface or the user interface component to be improved, and build a test report for the target application based on the test result analysis.

15. A computer-readable medium for result analysis in natural language for intelligent exploration testing, comprising instructions that, when executed, cause a processor to: obtaining an original test result produced through performing intelligent exploration testing on a target application, the target application including a plurality of user interfaces, each user interface including at least one user interface component; identifying a user interface or a user interface component to be improved in the target application through pattern matching the original test result with a user experience analysis rule; generating test result analysis in natural language corresponding to the user interface or the user interface component to be improved; and building a test report for the target application based on the test result analysis.