WO2023220902A1

WO2023220902A1 - User interface testing methods and apparatus

Info

Publication number: WO2023220902A1
Application number: PCT/CN2022/093164
Authority: WO
Inventors: 杨佼; 任伟博; 邓文奇
Original assignee: 北京小米移动软件有限公司
Priority date: 2022-05-16
Filing date: 2022-05-16
Publication date: 2023-11-23
Also published as: CN117413256A

Abstract

The present invention provides user interface testing methods and apparatus. One method comprises: by displaying a user interface to be tested, acquiring testing information of the user interface, the testing information being used for indicating an execution sequence of multiple operation behaviors and an interface screenshot corresponding to each operation behavior; according to each interface screenshot, determining a target control triggered by the corresponding operation behavior; and then sequentially triggering the target controls corresponding to the operation behaviors in the user interface according to the execution sequence. Therefore, the user interface can be automatically tested on the basis of the user interface screenshots, the labor cost is saved while the accuracy of user interface testing is ensured, and the user interface testing efficiency is improved. The present invention is applicable to various operating systems.

Description

User interface detection method and device

Technical field

The present invention relates to the field of computer technology, and in particular, to a user interface detection method and device.

Background technique

UI (User Interface) is the medium for interaction and information exchange between applications and users. In order to ensure that the user interface of the application meets the developer's expected requirements, UI testing is required, that is, the user interface needs to be detected.

UI testing is a technique for testing UI control elements on an application. For example, you can check whether the output results meet the expected test results by clicking on each UI control element on the user interface. In related technologies, manual UI testing is often required, that is, UI testing is completed by testers manually operating controls, which results in high labor costs.

Contents of the invention

An embodiment of the first aspect of the present invention proposes a user interface detection method, which includes: displaying a user interface to be detected; obtaining detection information of the user interface; wherein the detection information is used to indicate the execution of multiple operation behaviors sequence, and interface screenshots corresponding to each of the operation behaviors; determine the target control triggered by the corresponding operation behavior based on each of the interface screenshots; and trigger each of the operations in the user interface sequentially based on the execution sequence. The target control corresponding to the behavior.

Optionally, obtaining the detection information of the user interface includes: obtaining the detection information generated by the client from the server; wherein the detection information includes a test script and a picture library; the picture library includes a plurality of Interface screenshots; the test script is used to indicate the execution sequence of the operation behaviors corresponding to each of the interface screenshots in the picture library.

Optionally, determining the target control triggered by the corresponding operation behavior based on each interface screenshot includes: taking a screenshot of the user interface of the initial operation to obtain the target screenshot; comparing the target screenshot with the first operation Compare the interface screenshots corresponding to the behavior to use the control with display differences as the target control triggered by the first operation behavior; for any subsequent operation behavior except the first operation behavior, each time the target control is triggered once, Take a new screenshot of the user interface to update the target screenshot; compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so as to use the controls with display differences as the controls for the subsequent operation behavior. The target control to trigger.

Optionally, determining the target control triggered by the corresponding operation behavior based on each of the interface screenshots includes: using a trained object recognition model to perform object recognition on any of the interface screenshots to determine the target controls in the corresponding interface screenshots. The control on which the cursor is focused; the control on which the cursor is focused is determined as the target control.

Optionally, the object recognition model is trained using a training interface image; wherein the training interface image is marked with a cursor focus control.

Optionally, the method further includes: at least once the target control corresponding to the operation behavior triggers a response exception, sending exception prompt information to the server, or displaying the exception prompt information on the user interface.

The embodiment of the second aspect of the present invention proposes a user interface detection method, which includes: displaying the user interface; and taking screenshots of the user interface every time the corresponding operation behavior is completed in response to the remote control instruction to obtain the corresponding operation behavior of each operation. Interface screenshots; based on the execution sequence of each operation behavior and the interface screenshots corresponding to each operation behavior, detection information for detecting the user interface is generated.

Optionally, the method further includes: sending the detection information to a server, so as to send the detection information to a plurality of clients that are to detect the user interface.

The third embodiment of the present invention provides a user interface detection device, which includes: a first display module for displaying the user interface to be detected; an acquisition module for acquiring detection information of the user interface, wherein The detection information is used to indicate the execution sequence of multiple operation behaviors, as well as the interface screenshots corresponding to each operation behavior; the determination module is used to determine the target control triggered by the corresponding operation behavior based on each of the interface screenshots; first A processing module, configured to sequentially trigger the target controls corresponding to each of the operation behaviors in the user interface according to the execution sequence.

Optionally, the acquisition module is also configured to: obtain the detection information generated by the client from the server; wherein the detection information includes a test script and a picture library; the picture library includes multiple interface screenshots; The above-mentioned test script is used to instruct the execution sequence of the operation behaviors corresponding to each of the interface screenshots in the picture library.

Optionally, the determination module is also configured to: take a screenshot of the user interface of the initial operation to obtain a target screenshot; compare the target screenshot with the interface screenshot corresponding to the first operation behavior to display the existence The different controls are used as the target controls triggered by the first operation behavior; for any subsequent operation behavior except the first operation behavior, whenever the target control is triggered once, the user interface is re-screenshoted to update the Target screenshot: compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so that the control with a display difference is used as the target control triggered by the subsequent operation behavior.

Optionally, the determination module is also used to: use a trained object recognition model to perform object recognition on any of the interface screenshots to determine the control on which the cursor is focused in the corresponding interface screenshot; and determine the control on which the cursor is focused. , determined as the target control.

Optionally, the device further includes: a second processing module configured to send an exception prompt to the server at least once when a target control corresponding to the operation behavior triggers a response exception, or to display an exception prompt on the user interface. information.

The fourth embodiment of the present invention proposes a user interface detection device, which includes: a second display module for displaying the user interface; and a third processing module for processing the corresponding operation behavior whenever the corresponding operation behavior is completed in response to the remote control instruction. Screenshots are taken of the user interface to obtain interface screenshots corresponding to each operation; a generation module is used to generate a screenshot of the interface according to the execution order of each operation and the interface screenshots corresponding to each operation. Detection information for detection performed by the user interface.

Optionally, the device further includes: sending the detection information to a server, so as to send the detection information to multiple clients that are to detect the user interface.

A fifth embodiment of the present invention provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect, or to perform the method described in the second aspect.

The sixth aspect of the present invention provides a non-transient computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect, or to execute the second method. methods described in this aspect.

The seventh aspect embodiment of the present invention provides a computer program product, including a computer program, which when executed by a processor implements the method described in the first aspect, or implements the method described in the second aspect.

A method and device for detecting a user interface provided by an embodiment of the present invention realizes obtaining detection information of the user interface by displaying the user interface to be detected, where the detection information is used to indicate the execution sequence of multiple operation behaviors, and each Screenshots of the interface corresponding to the operation behavior, thereby determining the target control triggered by the corresponding operation behavior based on each interface screenshot, and then sequentially triggering the target control corresponding to each operation behavior in the user interface according to the execution sequence. As a result, the user interface can be automatically detected based on user interface screenshots, which not only ensures the accuracy of user interface detection, but also saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of the drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

Figure 1 is a schematic flow chart of a user interface detection method provided by an embodiment of the present invention;

Figure 2 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention;

Figure 3 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention;

Figure 4 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention;

Figure 5 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention;

Figure 6 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention;

Figure 7 is a schematic structural diagram of a user interface detection device provided by an embodiment of the present invention;

Figure 8 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention;

Figure 9 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention;

Figure 10 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention; and

Figure 11 is a structural block diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present invention and are not to be construed as limiting the present invention.

The user interface detection method and device according to the embodiment of the present invention will be described below with reference to the accompanying drawings.

FIG. 1 is a schematic flowchart of a user interface detection method provided by an embodiment of the present invention.

In related technologies, manual UI testing is often required, that is, UI testing is completed by testers manually operating controls, which results in high labor costs.

In response to this problem, embodiments of the present invention provide a user interface detection method to determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface in the detection information of the user interface to be detected, so as to determine the target control triggered by the user interface to be detected. The execution sequence in the detection information of the interface triggers the target controls corresponding to each operation behavior in the user interface sequentially, realizing automated detection of the user interface based on user interface screenshots and saving labor costs. As shown in Figure 1, the user interface The detection method includes the following steps:

Step 101: Display the user interface to be tested.

It should be noted that the user interface detection method provided in this embodiment can be executed by a user interface detection device. The user interface detection device may be an electronic device, or may be configured in the electronic device. Among them, the electronic device can be any stationary or mobile computing device capable of data processing, such as mobile computing devices such as laptops, smart phones, and wearable devices, or stationary computing devices such as desktop computers, or servers, or other types of Computing equipment, etc., are not limited in this embodiment.

In this embodiment, the user interface to be detected may be the user interface of any application program. Among them, an application program refers to a computer program that performs one or more specific tasks. It runs in user mode, can interact with users, and has a visual user interface.

In this embodiment, the number of user interfaces to be detected may be one or more, which is not limited in this embodiment. It should be noted that when there is one user interface to be detected, the user interface to be detected can be any user interface of any application; when there are multiple user interfaces to be detected, the multiple user interfaces can be Each user interface to be detected can be any multiple user interfaces of the same application, or any user interface of multiple different applications, or any multiple user interfaces of multiple different applications. This implementation There is no restriction on this in the example.

In this embodiment, the user interface detection device can display the user interface to be detected to obtain detection information of the user interface to be detected. As a possible implementation method, when there is one user interface to be detected, the user interface detection device can directly display the user interface to be detected; when there are multiple user interfaces to be detected, the user interface The detection device can display multiple user interfaces to be detected at the same time, or can display a set number of user interfaces to be detected each time in a set order until all user interfaces to be detected are displayed. In this embodiment There are no restrictions on this. Among them, this embodiment does not limit the determination of the setting order and the specific value of the setting number. Alternatively, the setting can be made based on manual experience. For example, the setting order can be set to the user's The operation sequence of the user interface operation is to set the number of settings to 3, or it can also be dynamically adjusted according to actual application requirements, which is not limited in this embodiment.

Step 102: Obtain detection information of the user interface, where the detection information is used to indicate the execution sequence of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior.

It is understandable that when a user operates the user interface, if there are multiple operations, there will be an execution sequence between the multiple operations, and each operation will have its corresponding interface screenshot, so it needs to be detected. The detection information of the user interface should be able to indicate the execution sequence of multiple operations, as well as the interface screenshots corresponding to each operation.

In this embodiment, since the detection information can be used to indicate the execution sequence of multiple operation behaviors, as well as the interface screenshots corresponding to each operation behavior, the detection device of the user interface obtains the detection information of the user interface, that is, obtains the user interface The execution sequence of multiple operations and the screenshots of the interface corresponding to each operation. Therefore, the user interface detection device can detect the user interface based on the execution order and screenshots of each interface in the acquired detection information of the user interface.

In a possible implementation of this embodiment, the user interface detection device can record the user interface to be detected while displaying the user interface to be detected, so that when the user operates the user interface to be detected, the user interface to be detected will be recorded. The user's operation behavior is recorded to obtain detection information of the user interface to be detected that can be used to indicate the execution order of the user's multiple operation behaviors, as well as interface screenshots corresponding to each operation behavior. Optionally, the detection information of the user interface to be detected can be presented in the form of scripts and pictures, where the script is used to indicate the execution sequence of multiple operation behaviors, and the picture is used to indicate the interface screenshot corresponding to each operation behavior.

It should be noted that when the detection information of the user interface to be detected is presented in the form of scripts and pictures, the user interface detection device obtains the detection information of the user interface to be detected, that is, the user interface to be detected is obtained. scripts and pictures, so that the user interface detection device can realize a new uninterrupted UI automated test by directly playing back the script.

Optionally, when obtaining the detection information of the user interface, the detection device of the user interface can use the method of taking a screenshot of each operation behavior to obtain the interface screenshot corresponding to each operation behavior. At this time, the execution order of each operation behavior can be based on the interface Determine the order of screenshots taken. For example, assuming that the user interface to be detected is the login interface of a certain application, the user interface detection device can record the login interface while displaying the login interface, so that when the user enters the login interface, When entering a username and password on the login interface to log in, you can take screenshots of the interface for clicking the username input box to enter the username, clicking the password input box to enter the password, and clicking the login button to log in, so as to obtain the corresponding information for each operation. Interface screenshot. Alternatively, you can use the method of taking a screenshot of a user interface to obtain an interface screenshot containing multiple operation behaviors. At this time, the execution order of each operation behavior can be determined by marking in the interface screenshot. For example, in the above login interface For example, after taking a screenshot of the login interface, you can determine the operation behavior corresponding to the screenshot of the login interface by marking 1 near the user name input box, 2 near the password input box, and 3 near the login button. The execution sequence is to click the user name input box to enter the user name, click the password input box to enter the password, click the login button to log in, etc., which are not limited in this embodiment.

Step 103: Determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface.

In this embodiment, a control can be understood as any user interface element of any application program. Users can trigger controls to complete one or more specific tasks. Optionally, the target control may include a text input box, a button, a drop-down menu, etc., which is not limited in this embodiment.

In this embodiment, the number of target controls triggered by the corresponding operation behavior may be one or more, and this is not limited in this embodiment.

In this embodiment, there is a corresponding relationship between interface screenshots and operation behaviors, so that the user interface detection device can determine the target control triggered by the corresponding operation behavior based on each interface screenshot. For example, in the above example of the login interface, since the operation behaviors corresponding to the screenshots of the login interface can include clicking on the username input box to enter the user name, clicking on the password input box to enter the password, and clicking on the login button to log in, the correspondence determined based on the screenshots of the login interface can be The target controls triggered by the operation behavior can include user name input box, password input box and login button. It should be noted that the operation behavior corresponding to each interface screenshot may include only one operation behavior or may include multiple operation behaviors, which is not limited in this embodiment.

Step 104: According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.

In this embodiment, after the user interface detection device obtains the detection information of the user interface and determines the target control triggered by each operation behavior, it can execute the sequence of multiple operation behaviors indicated by the detection information of the user interface. The detection of the user interface is completed by sequentially triggering the target controls corresponding to each operation behavior in the user interface. For example, in the above login interface example, since the execution sequence of the operation behaviors corresponding to the screenshots of the login interface is to click the user name input box to enter the user name, click the password input box to enter the password, and click the login button to log in, it is possible to follow this execution sequence. , sequentially trigger the user name input box, password input box and login button in the user interface to complete the detection of the user interface.

It can be understood that, after obtaining the detection information of the user interface, the target control triggered by the corresponding operation behavior can be determined based on the interface screenshot corresponding to each operation behavior indicated by the detection information of the user interface, so that the target control triggered by the corresponding operation behavior can be determined based on the detection information of the user interface. The execution sequence of multiple operation behaviors indicated by the detection information is implemented by sequentially triggering the target controls corresponding to each operation behavior in the user interface to realize automatic detection of the user interface based on user interface screenshots, saving labor costs.

The user interface detection method provided in this embodiment achieves acquisition of detection information of the user interface by displaying the user interface to be detected, where the detection information is used to indicate the execution order of multiple operation behaviors and the interface corresponding to each operation behavior. Take screenshots to determine the target controls triggered by the corresponding operation behaviors based on the screenshots of each interface, and then sequentially trigger the target controls corresponding to each operation behavior in the user interface based on the execution sequence. As a result, the user interface can be automatically detected based on user interface screenshots, which not only ensures the accuracy of user interface detection, but also saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems.

It can be seen from the above analysis that in the embodiment of the present invention, it is necessary to determine the target control triggered by the corresponding operation behavior based on each interface screenshot. In order to clearly explain how the present invention determines the target control triggered by the corresponding operation behavior based on each interface screenshot, the present invention A user interface detection method is also proposed.

FIG. 2 is a schematic flowchart of another user interface detection method provided by an embodiment of the present invention.

As shown in Figure 2, the user interface detection method may include the following steps:

Step 201: Display the user interface to be tested.

Step 202: Obtain detection information of the user interface, where the detection information is used to indicate the execution order of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior.

It should be noted that the execution process of steps 201-202 can refer to the execution process of the previous embodiment 101-102. The principles are the same and will not be described again here.

Step 203: Take a screenshot of the initial running user interface to obtain the target screenshot.

In this embodiment, the initially running user interface can be understood as any user interface in which the user who opens any application program for the first time has not performed any operation behavior. For example, in the login interface of a certain application, the user interface that is initially run is the interface in which the user has not entered the user name and password to log in.

In this embodiment, a screenshot can be taken of the initial running user interface to obtain the target screenshot. As a possible implementation method, you can use the screenshot tool that comes with the device, or the screenshot tool that has been downloaded by the device, or the screenshot function provided by the application, etc., for the user interface that is initially run in any application. Take a screenshot to get a screenshot of the user interface of the initial run and use it as the target screenshot.

Step 204: Compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so that the control with a display difference is used as the target control triggered by the first operation behavior.

It is understandable that the interface screenshot corresponding to the first operation behavior should be a user interface screenshot in which the first operation behavior is already being executed or has been completed. This is because if the first operation behavior has not yet been executed, the controls in the user interface will not change. Only when the first operation behavior is already being executed or has been completed, a certain control in the user interface will be significantly different from before the first operation behavior is executed.

In this embodiment, after the target screenshot is obtained, the target screenshot can be compared with the interface screenshot corresponding to the first operation behavior, so that the control with a display difference is used as the target control triggered by the first operation behavior. For example, in the login interface of an application, the target screenshot is the screenshot of the interface where the user has not entered the user name and password for the login operation. The first operation behavior is the user entering the user name, so the interface screenshot corresponding to the first operation behavior can be Take a screenshot of the interface where the user is inputting a user name in the user name input box of the login interface. At this time, compare the target screenshot with the screenshot of the interface corresponding to the first operation behavior to determine whether the control of the user name input box is before and after the first operation behavior is executed. There are significant differences, so that the user name input box control can be determined as the target control triggered by the first operation behavior.

Step 205: For any subsequent operation behavior except the first operation behavior, each time the target control is triggered, the user interface is re-screenshoted to update the target screenshot.

In this embodiment, after determining the target control triggered by the first operation behavior, the same method can be used to determine the target control of any subsequent operation behavior after the first operation behavior except for the first operation behavior. The target control to trigger. Specifically, for any subsequent operation behavior except the first operation behavior, each time the target control is triggered, the user interface can be re-screenshoted to update the target screenshot. For example, in the above example of the login interface, the first operation behavior is for the user to enter a user name, and the target control is the user name input box. Assume that any subsequent operation behavior after the first operation behavior of the user entering a user name is for the user to enter a password, then , you can trigger the target control of the user name input box once, that is, after the first operation behavior has been executed, you can re-take a screenshot of the user interface, and use the re-taken screenshot of the user interface as a new target screenshot to achieve the goal. Screenshot update.

It is understandable that since the same method is used to determine the target control triggered by any subsequent operation behavior after the first operation behavior, every time the target control is triggered, the user interface needs to be re-screenshoted to update the target. Take screenshots and compare them with interface screenshots corresponding to subsequent operations.

Step 206: Compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so that the control with display differences is used as the target control triggered by the subsequent operation behavior.

Similarly, the interface screenshot corresponding to the subsequent operation behavior should be the user interface screenshot in which the subsequent operation behavior is already being executed or has been completed. This is because if the subsequent operation behavior has not yet been executed, the controls in the user interface will not change. When the subsequent operation behavior is already being executed or has been completed, a certain control in the user interface will be significantly different from before the subsequent operation behavior is executed.

In this embodiment, after the target screenshot is updated, the updated target screenshot can be compared with the interface screenshot corresponding to the subsequent operation behavior, so that the control with a display difference is used as the target control triggered by the subsequent operation behavior. For example, in the above example of the login interface, the updated target screenshot can be understood as the screenshot of the interface where the user has entered the user name in the user name input box of the login interface. The subsequent operation behavior is the user entering the password. Therefore, the interface corresponding to the subsequent operation behavior The screenshot can be a screenshot of the interface where the user is inputting the password in the password input box of the login interface. At this time, comparing the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, it can be determined that the control of the password input box is executed in the subsequent operation behavior. There are significant differences before and after, so the password input box control can be determined as the target control triggered by subsequent operations.

Step 207: According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.

It should be noted that the execution process of this step can be referred to the execution process of step 104 in the above embodiment. The principles are the same and will not be described again here.

The user interface detection method provided in this embodiment is to obtain the target screenshot by taking a screenshot of the initial running user interface, and compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so as to use controls with display differences as The target control triggered by the first operation behavior, so that for any subsequent operation behavior except the first operation behavior, every time the target control is triggered, the user interface is re-screenshoted to update the target screenshot, and then the updated target screenshot is , compare it with the interface screenshot corresponding to the subsequent operation behavior, so as to use the control with display differences as the target control triggered by the subsequent operation behavior. Therefore, the target control triggered by each operation behavior can be determined by comparing the interface screenshot corresponding to the operation behavior with the target screenshot. It is easy to implement, operable, can effectively save labor costs, and is suitable for a variety of operating systems.

The previous embodiment provides a possible implementation method of determining the target control triggered by the corresponding operation behavior based on each interface screenshot. In order to more clearly explain the process of determining the target control triggered by the corresponding operation behavior based on each interface screenshot in the present invention, This embodiment provides another possible implementation method of determining the target control triggered by the corresponding operation behavior based on screenshots of each interface. Figure 3 is a flow chart of another user interface detection method provided by an embodiment of the present invention.

As shown in Figure 3, the user interface detection method may include the following steps:

Step 301: Display the user interface to be tested.

Step 302: Obtain detection information of the user interface, where the detection information is used to indicate the execution order of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior.

It should be noted that the execution process of steps 301-302 can refer to the execution process of the above-mentioned embodiments 101-102. The principles are the same and will not be described again.

Step 303: Use the trained object recognition model to perform object recognition on any interface screenshot to determine the control on which the cursor is focused in the corresponding interface screenshot.

In this embodiment, the control with the cursor focused can be understood as that in the user interface, when the user clicks on any control, the control obtains the cursor, which is the control with the cursor focused. For example, in the login interface of an application, when the user clicks the username input box of the login interface to enter the username, the username input box obtains the cursor, which is the control on which the cursor is focused.

In this embodiment, a trained object recognition model can be used to perform object recognition on any interface screenshot, thereby determining the control on which the cursor is focused in the corresponding interface screenshot. For example, assume that the screenshot of the login interface of an application is a screenshot of the user entering the user name in the user name input box of the login interface. Using a trained object recognition model to perform object recognition, it can be determined that the cursor focus in the screenshot of the interface is The control is the username input box. Optionally, the object recognition model may be trained using a training interface image, where a control with a cursor focus is marked in the training interface image.

It should be noted that the interface screenshots in this embodiment should be user interface screenshots when the operation behavior is being executed. This is because if the operation behavior has not yet been executed or has been completed, then a certain control in the user interface will no longer have the cursor focus. , only when the operation behavior is being executed, a certain control in the user interface will have the cursor focus.

Step 304: Determine the control on which the cursor is focused as the target control.

In this embodiment, after determining the control on which the cursor is focused in the interface screenshot, the control on which the cursor is focused can be determined as the target control. For example, in the above example of the login interface, since it is determined that the control focused by the cursor in the screenshot of the interface is the user name input box, the user name input box can be determined as the target control. Similarly, assume that the screenshot of the login interface of an application is a screenshot of the user entering the password in the password input box of the login interface. The trained object recognition model is used to perform object recognition on it, and it is determined that the cursor is focused on the screenshot of the interface. The control is a password input box, so the password input box can be determined as the target control.

Step 305: According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.

The user interface detection method provided by this embodiment adopts a trained object recognition model to perform object recognition on any interface screenshot to determine the control where the cursor is focused in the corresponding interface screenshot, thereby determining the control where the cursor is focused as the target. controls. As a result, the object recognition model can be used to determine the control that the cursor is focused on in the interface screenshot, thereby determining the control as the target control, saving labor costs and improving the efficiency of determining the target control.

In order to clearly explain the above embodiment, this embodiment provides another method for detecting a user interface. FIG. 4 is a schematic flowchart of another method for detecting a user interface provided by an embodiment of the present invention.

As shown in Figure 4, the user interface detection method may include the following steps:

Step 401: Display the user interface to be tested.

It should be noted that the execution process of this step can be referred to the execution process of step 101 in the above embodiment. The principles are the same and will not be described again here.

Step 402: Obtain the detection information generated by the client from the server.

In this embodiment, the detection information may include a test script and a picture library, where the picture library includes multiple interface screenshots, and the test script is used to indicate the execution order of the operation behaviors corresponding to each interface screenshot in the picture library.

In this embodiment, the detection information can be generated by the client of the user interface and sent to the server, so that the detection device of the user interface can obtain the detection information generated by the client from the server and perform subsequent operations.

Optionally, the detection device of the user interface can obtain the detection information generated by the client from the server through online real-time collection, or can obtain the detection information generated by the client from the server through offline manual collection, or can also obtain the detection information generated by the client through the network. The detection information generated by the client is obtained from the server through transmission or physical copy, etc. This embodiment does not limit this.

Step 403: Determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface.

Step 404: According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.

It should be noted that the execution process of steps 403-404 can refer to the execution process of the above-mentioned embodiments 103-104. The principles are the same and will not be described again here.

Step 405: At least one target control corresponding to the operation behavior triggers a response exception, and sends the exception prompt information to the server, or displays the exception prompt information on the user interface.

In this embodiment, after the target controls corresponding to each operation behavior are sequentially triggered in the user interface according to the execution sequence, the detection device of the user interface can provide feedback to the server or the user interface based on whether there is a response exception when the target control is triggered. Specifically, when at least one target control corresponding to the operation behavior triggers a response exception, the exception prompt information can be sent to the server, or the exception prompt information can be displayed on the user interface. For example, in the login interface of an application, the operation behaviors include clicking the user name input box to enter the user name, clicking the password input box to enter the password, and clicking the login button to log in. The target controls corresponding to the operation behaviors are the user name input box, password Input box and login button, so that when the user clicks on the user name input box and cannot enter the user name, an abnormal prompt message that the user name input box cannot be input normally can be sent to the server, or the user name input box cannot be displayed on the user interface. Exception prompt information for normal input.

Optionally, the exception prompt information sent to the server may be in text form, table form, picture form, etc., which is not limited in this embodiment. Similarly, the abnormal prompt information displayed on the user interface may be in the form of a pop-up window, a text prompt, a picture, etc., and this embodiment is not limited.

The user interface detection method provided in this embodiment, by displaying the user interface to be detected and obtaining the detection information generated by the client from the server, determines the target control triggered by the corresponding operation behavior based on the screenshots of each interface, thereby determining the execution sequence , sequentially trigger the target control corresponding to each operation behavior in the user interface, and then at least one target control corresponding to the operation behavior triggers a response exception, and sends the exception prompt information to the server, or displays the exception prompt information in the user interface. As a result, the user interface can be automatically detected based on the detection information generated by the client. While ensuring the accuracy of user interface detection, it saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems. .

It should be noted that the above embodiments are described from the perspective of the client that detects the user interface to be detected. In order to explain the detection process of the user interface more clearly, embodiments of the present invention provide a method from the perspective of the client that detects the user interface to be detected. Describing a possible implementation of a user interface detection method, FIG. 5 is a schematic flowchart of another user interface detection method provided by an embodiment of the present invention.

As shown in Figure 5, the user interface detection method may include the following steps:

Step 501: Display the user interface.

Similarly, the user interface detection method provided in this embodiment can be executed by a user interface detection device. The user interface detection device may be an electronic device, or may be configured in the electronic device. Among them, the electronic device can be any stationary or mobile computing device capable of data processing, such as mobile computing devices such as laptops, smart phones, and wearable devices, or stationary computing devices such as desktop computers, or servers, or other types of Computing equipment, etc., are not limited in this embodiment.

In this embodiment, the user interface may be the user interface of any application program. The number of user interfaces may be one or more, which is not limited in this embodiment. It should be noted that when there is one user interface, the user interface can be any user interface of any application; when there are multiple user interfaces, the multiple user interfaces to be detected should be the same user interface. Any number of user interfaces for an application.

In this embodiment, the user interface detection device can display the user interface so as to generate detection information for detecting the user interface. As a possible implementation method, when there is one user interface, the user interface detection device can directly display the user interface; when there are multiple user interfaces, the user interface detection device can display multiple users at the same time. The interface may also display a set number of user interfaces to be detected each time in a set order until all user interfaces are displayed. This is not limited in this embodiment. Among them, this embodiment does not limit the determination of the setting order and the specific value of the setting number. Alternatively, the setting can be made based on manual experience. For example, the setting order can be set to the user's The operation sequence of the user interface operation is to set the number of settings to 3, or it can also be dynamically adjusted according to actual application requirements, which is not limited in this embodiment.

Step 502: Whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface is taken to obtain an interface screenshot corresponding to each operation behavior.

In this embodiment, whenever the corresponding operation behavior is executed in response to the remote control command, the user interface detection device can obtain the interface screenshot corresponding to each operation behavior by taking a screenshot of the user interface. That is to say, the remote control command executes the corresponding operation behavior. Whenever the remote control command completes the execution of the corresponding operation behavior, the user interface detection device takes a screenshot of the user interface, so that the interface screenshot corresponding to each operation behavior can be obtained. For example, in the login interface of an application, the operation behaviors include clicking the user name input box to enter the user name, clicking the password input box to enter the password, and clicking the login button to log in. After the remote control command is executed, click the user name input box to enter the user name. In the case of the operation behavior of clicking the user name input box, the user interface detection device can obtain the interface screenshot corresponding to the operation behavior of clicking the user name input box and inputting the user name by taking a screenshot of the user interface. Similarly, when the remote control command is executed and the operation behavior of clicking on the password input box to enter the password is completed, the user interface detection device can obtain the interface corresponding to the operation behavior of clicking on the password input box to enter the password by taking a screenshot of the user interface. screenshot. And, in the case of the operation behavior of clicking the login button to log in after the remote control command is executed, the user interface detection device can obtain the interface screenshot corresponding to the operation behavior of clicking the login button to log in by taking a screenshot of the user interface.

Step 503: Generate detection information for detecting the user interface based on the execution order of each operation behavior and the interface screenshot corresponding to each operation behavior.

In this embodiment, after obtaining the interface screenshots corresponding to each operation behavior, since each interface screenshot has a time sequence, the execution order of each operation behavior can be determined based on the time sequence of each interface screenshot, and then the execution order of each operation behavior can be determined based on the time sequence of each operation behavior. The execution sequence of the behavior and the interface screenshots corresponding to each operation behavior generate detection information used to detect the user interface. Among them, the detection information is used to indicate the execution sequence of multiple operation behaviors, as well as the interface screenshots corresponding to each operation behavior. Optionally, the detection information may include a test script and a picture library, where the picture library includes multiple interface screenshots, and the test script is used to indicate the execution order of the operation behaviors corresponding to each interface screenshot in the picture library. For example, in the above example of the login interface, the chronological order of screenshots of each interface can be: first, the interface screenshot corresponding to the operation behavior of clicking the user name input box to enter the user name is obtained; secondly, the screenshot of the interface corresponding to the operation behavior of clicking the password input box to enter the password is obtained. A screenshot of the interface corresponding to the operation behavior. Finally, the screenshot of the interface corresponding to the operation behavior of clicking the login button to log in is obtained. Therefore, the execution order of each operation behavior is to first click the user name input box to enter the user name, and then click the password input box. Enter the password and finally click the login button to log in. From this, the detection information used to detect the user interface can be generated.

The user interface detection method provided by the embodiment of the present invention, by displaying the user interface, realizes that whenever the corresponding operation behavior is completed in response to the remote control instruction, a screenshot of the user interface is taken to obtain the interface screenshot corresponding to each operation behavior, so as to obtain the interface screenshot corresponding to each operation behavior. The execution sequence of each operation behavior and the interface screenshot corresponding to each operation behavior are used to generate detection information for detecting the user interface. Thus, whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface can be obtained to obtain the interface screenshot corresponding to each operation behavior, so that based on the execution sequence of each operation behavior and the corresponding operation behavior Take screenshots of the interface to generate detection information for testing the user interface.

In order to clearly explain the previous embodiment, this embodiment provides another method for detecting a user interface. FIG. 6 is a schematic flowchart of another method for detecting a user interface provided by an embodiment of the present invention.

As shown in Figure 6, the detection method of the user interface may include the following steps:

Step 601: Display the user interface.

Step 602: Whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface is taken to obtain an interface screenshot corresponding to each operation behavior.

Step 603: Generate detection information for detecting the user interface based on the execution order of each operation behavior and the interface screenshot corresponding to each operation behavior.

It should be noted that the execution process of steps 601-603 can refer to the execution process of steps 501-503 in the previous embodiment. The principles are the same and will not be described again here.

Step 604: Send the detection information to the server to send the detection information to multiple clients of the user interfaces to be detected.

In this embodiment, after generating the detection information for detecting the user interface, the detection device of the user interface may send the detection information to the server to send the detection information to multiple clients of the user interface to be detected. Optionally, the detection device of the user interface can send the detection information to the server through network transmission or physical copy.

The user interface detection method provided by the embodiment of the present invention, after generating detection information for detecting the user interface, sends the detection information to the server to send the detection information to multiple clients of the user interface to be detected. Thus, the generated detection information for detecting the user interface can be sent through the server to the client that detects the user interface to be detected, so as to perform user interface detection.

Corresponding to the user interface detection method provided by the above embodiments of Figures 1 to 4, the present invention also provides a user interface detection device, because the user interface detection device provided by the embodiment of the present invention is consistent with the above Figures 1 to 4 The detection method of the user interface provided by the embodiment corresponds to the detection method of the user interface. Therefore, the implementation of the detection method of the user interface is also applicable to the detection device of the user interface provided by the embodiment of the present invention, and will not be described in detail in the embodiment of the present invention.

FIG. 7 is a schematic structural diagram of a user interface detection device provided by an embodiment of the present invention.

As shown in FIG. 7 , the user interface detection device includes: a first display module 71 , an acquisition module 72 , a determination module 73 and a first processing module 74 .

The first display module 71 is used to display the user interface to be detected;

The acquisition module 72 is used to obtain detection information of the user interface, where the detection information is used to indicate the execution sequence of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior;

The determination module 73 is used to determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface;

The first processing module 74 is used to sequentially trigger the target controls corresponding to each operation behavior in the user interface according to the execution sequence.

Further, in a possible implementation of the embodiment of the present invention, the acquisition module 72 is also used to:

Obtain the detection information generated by the client from the server;

Among them, the detection information includes test scripts and image libraries;

The image gallery includes multiple interface screenshots;

The test script is used to instruct the execution sequence of the operation behaviors corresponding to the screenshots of each interface in the picture library.

Further, in a possible implementation of the embodiment of the present invention, the determination module 73 is also used to:

Take a screenshot of the user interface of the initial run to obtain the target screenshot;

Compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so as to use the control with display differences as the target control triggered by the first operation behavior;

For any subsequent operation behavior except the operation behavior, every time the target control is triggered, the user interface will be re-screenshoted to update the target screenshot;

Compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so that the control with display differences is used as the target control triggered by the subsequent operation behavior.

Use a trained object recognition model to perform object recognition on any interface screenshot to determine the control on which the cursor is focused in the corresponding interface screenshot;

Determine the control on which the cursor is focused as the target control.

Further, in a possible implementation manner of the embodiment of the present invention, the object recognition model is trained using a training interface image, where a control with a cursor focus is marked in the training interface image.

Based on the above embodiments, the embodiment of the present invention also provides a possible implementation of a user interface detection device. Figure 8 is a schematic structural diagram of another user interface detection device provided by the embodiment of the present invention. In the previous embodiment On the basis of , the user interface detection device also includes: a second processing module 75 .

The second processing module 75 is configured to trigger a response exception when the target control corresponding to at least one operation behavior occurs, and send exception prompt information to the server, or display the exception prompt information on the user interface.

The user interface detection device provided by the embodiment of the present invention achieves acquisition of detection information of the user interface by displaying the user interface to be detected, wherein the detection information is used to indicate the execution order of multiple operation behaviors, and the corresponding operation behaviors of each operation. Take screenshots of the interface to determine the target controls triggered by the corresponding operation behaviors based on the screenshots of each interface, and then sequentially trigger the target controls corresponding to each operation behavior in the user interface based on the execution sequence. As a result, the user interface can be automatically detected based on user interface screenshots, which not only ensures the accuracy of user interface detection, but also saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems.

Corresponding to the user interface detection method provided by the above embodiments of Figures 5 to 6, the present invention also provides a user interface detection device, because the user interface detection device provided by the embodiment of the present invention is consistent with the above Figures 5 to 6 The detection method of the user interface provided by the embodiment corresponds to the detection method of the user interface. Therefore, the implementation of the detection method of the user interface is also applicable to the detection device of the user interface provided by the embodiment of the present invention, and will not be described in detail in the embodiment of the present invention.

FIG. 9 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention.

As shown in FIG. 9 , the user interface detection device includes: a second display module 91 , a third processing module 92 and a generation module 93 .

The second display module 91 is used to display the user interface;

The third processing module 92 is configured to take a screenshot of the user interface every time the corresponding operation behavior is executed in response to the remote control command, so as to obtain the interface screenshot corresponding to each operation behavior;

The generation module 93 is configured to generate detection information for detecting the user interface based on the execution sequence of each operation behavior and the screenshots of the interface corresponding to each operation behavior.

Based on the previous embodiment, the embodiment of the present invention also provides a possible implementation of a user interface detection device. Figure 10 is a schematic structural diagram of another user interface detection device provided by the embodiment of the present invention. In the previous embodiment Based on the example, the user interface detection device also includes: a sending module 94 .

The sending module 94 is used to send detection information to the server, so as to send the detection information to multiple clients of the user interfaces to be detected.

The user interface detection device provided by the embodiment of the present invention, by displaying the user interface, realizes that whenever the corresponding operation behavior is executed in response to the remote control instruction, a screenshot of the user interface is taken to obtain the interface screenshot corresponding to each operation behavior, so as to obtain the interface screenshot corresponding to each operation behavior. The execution sequence of each operation behavior and the interface screenshot corresponding to each operation behavior are used to generate detection information for detecting the user interface. Thus, whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface can be obtained to obtain the interface screenshot corresponding to each operation behavior, so that based on the execution sequence of each operation behavior and the corresponding operation behavior Take screenshots of the interface to generate detection information for testing the user interface.

In order to implement the above embodiments, the present invention also proposes an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be used by the at least one processor. Instructions are executed by the at least one processor to enable the at least one processor to execute the user interface detection method provided by any of the foregoing embodiments.

In order to implement the above embodiments, the present invention also proposes a non-transient computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the user interface detection method provided in any of the foregoing embodiments. .

In order to implement the above embodiments, the present invention also proposes a computer program product, which includes a computer program. The computer program is executed by a processor for detecting the user interface provided by any of the foregoing embodiments.

Figure 11 is a structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device shown in FIG. 11 is only an example and should not impose any restrictions on the functions and scope of use of the embodiments of the present invention.

As shown in Figure 11, the electronic device 10 includes a processor 11, which can be loaded into a random access memory (RAM, Random Access Memory) 13 according to a program stored in a read-only memory (ROM, Read Only Memory) 12 or from a memory 16 perform various appropriate actions and processing according to the program in it. In the RAM 13, various programs and data required for the operation of the electronic device 10 are also stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via the bus 14. An input/output (I/O, Input/Output) interface 15 is also connected to the bus 14 .

The following components are connected to the I/O interface 15: a memory 16 including a hard disk, etc.; and a communication section 17 including a network interface card such as a LAN (Local Area Network) card, a modem, etc., which is executed via a network such as the Internet Communication processing; driver 18 is also connected to I/O interface 15 as needed.

In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication part 17. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the present invention are performed.

In an exemplary embodiment, a storage medium including instructions, such as a memory 16 including instructions, is also provided. The instructions can be executed by the processor 11 of the electronic device 10 to complete the above user interface detection method. Optionally, the storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage Equipment etc.

In the description of this specification, reference to the terms "one embodiment," "some embodiments," "an example," "specific examples," or "some examples" or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the invention. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine different embodiments or examples and features of different embodiments or examples described in this specification unless they are inconsistent with each other.

In addition, the terms “first” and “second” are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically limited.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments, or portions of code that include one or more executable instructions for implementing customized logical functions or steps of the process. , and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed out of the order shown or discussed, including in a substantially simultaneous manner or in the reverse order, depending on the functionality involved, which shall It should be understood by those skilled in the art to which embodiments of the present invention belong.

The logic and/or steps represented in the flowcharts or otherwise described herein, for example, may be considered a sequenced list of executable instructions for implementing the logical functions, and may be embodied in any computer-readable medium, For use by, or in combination with, instruction execution systems, devices or devices (such as computer-based systems, systems including processors or other systems that can fetch instructions from and execute instructions from the instruction execution system, device or device) or equipment. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wires (electronic device), portable computer disk cartridges (magnetic device), random access memory (RAM), Read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, and subsequently edited, interpreted, or otherwise suitable as necessary. process to obtain the program electronically and then store it in computer memory.

It should be understood that various parts of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if it is implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: discrete logic gate circuits with logic functions for implementing data signals; Logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

Those of ordinary skill in the art can understand that all or part of the steps involved in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The program can be stored in a computer-readable storage medium. When executed, one of the steps of the method embodiment or a combination thereof is included.

In addition, each functional unit in various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

The storage media mentioned above can be read-only memory, magnetic disks or optical disks, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above-mentioned embodiments are illustrative and should not be construed as limitations of the present invention. Those of ordinary skill in the art can make modifications to the above-mentioned embodiments within the scope of the present invention. The embodiments are subject to changes, modifications, substitutions and variations.

Claims

A user interface detection method, characterized by including the following steps:

Display the user interface to be tested;

Obtain detection information of the user interface, wherein the detection information is used to indicate the execution order of multiple operation behaviors, and interface screenshots corresponding to each operation behavior;

According to the screenshots of each interface, determine the target control triggered by the corresponding operation behavior;

According to the execution sequence, the target controls corresponding to each operation behavior are sequentially triggered in the user interface.
The method according to claim 1, characterized in that said obtaining the detection information of the user interface includes:

Obtain the detection information generated by the client from the server;

Wherein, the detection information includes test scripts and picture libraries;

The picture library includes multiple interface screenshots;

The test script is used to instruct the execution sequence of the operation behaviors corresponding to each of the interface screenshots in the picture library.
The method according to claim 1 or 2, characterized in that determining the target control triggered by the corresponding operation behavior based on each interface screenshot includes:

Take a screenshot of the user interface of the initial run to obtain the target screenshot;

Compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so that the control with a display difference is used as the target control triggered by the first operation behavior;

For any subsequent operation behavior after the first operation behavior, each time the target control is triggered, the user interface is re-screenshoted to update the target screenshot;

The updated target screenshot is compared with the interface screenshot corresponding to the subsequent operation behavior, so that the control with a display difference is used as the target control triggered by the subsequent operation behavior.
The method according to claim 1 or 2, characterized in that determining the target control triggered by the corresponding operation behavior based on each interface screenshot includes:

Using a trained object recognition model, perform object recognition on any of the interface screenshots to determine the control on which the cursor is focused in the corresponding interface screenshot;

The control on which the cursor is focused is determined as the target control.
The method of claim 4, wherein the object recognition model is trained using a training interface image, wherein the training interface image is marked with a cursor focus control.
The method according to any one of claims 1-5, characterized in that the method further includes:

At least once, a target control corresponding to the operation behavior triggers a response exception, and the exception prompt information is sent to the server, or the exception prompt information is displayed on the user interface.
A user interface detection method, characterized by including the following steps:

Display user interface;

Whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface is taken to obtain an interface screenshot corresponding to each operation behavior;

According to the execution order of each operation behavior and the interface screenshot corresponding to each operation behavior, detection information for detecting the user interface is generated.
The method of claim 7, further comprising:

The detection information is sent to a server to send the detection information to multiple clients that are to detect the user interface.
A user interface detection device, characterized by including:

The first display module is used to display the user interface to be tested;

An acquisition module, used to obtain detection information of the user interface, wherein the detection information is used to indicate the execution sequence of multiple operation behaviors, and interface screenshots corresponding to each operation behavior;

The determination module is used to determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface;

The first processing module is configured to sequentially trigger target controls corresponding to each operation behavior in the user interface according to the execution sequence.
The device according to claim 9, characterized in that the acquisition module is also used to:

Obtain the detection information generated by the client from the server;

Wherein, the detection information includes test scripts and image libraries;

The picture library includes multiple interface screenshots;

The test script is used to instruct the execution sequence of the operation behaviors corresponding to each of the interface screenshots in the picture library.
The device according to claim 8 or 9, characterized in that the determining module is also used to:

Take a screenshot of the user interface of the initial run to obtain the target screenshot;

Compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so that the control with a display difference is used as the target control triggered by the first operation behavior;

For any subsequent operation behavior after the first operation behavior, each time the target control is triggered, the user interface is re-screenshoted to update the target screenshot;

The updated target screenshot is compared with the interface screenshot corresponding to the subsequent operation behavior, so that the control with a display difference is used as the target control triggered by the subsequent operation behavior.
The device according to claim 8 or 9, characterized in that the determining module is also used to:

Using a trained object recognition model, perform object recognition on any of the interface screenshots to determine the control on which the cursor is focused in the corresponding interface screenshot;

The control on which the cursor is focused is determined as the target control.
The device according to claim 12, wherein the object recognition model is trained using a training interface image, wherein a cursor focus control is marked in the training interface image.
The device according to any one of claims 8-13, characterized in that the device further includes:

The second processing module is configured to trigger a response exception at least once on the target control corresponding to the operation behavior, send exception prompt information to the server, or display the exception prompt information on the user interface.
A user interface detection device, characterized by including:

The second display module is used to display the user interface;

The third processing module is used to take screenshots of the user interface every time the corresponding operation behavior is completed in response to the remote control command, so as to obtain the interface screenshot corresponding to each operation behavior;

The generation module is configured to generate detection information for detecting the user interface based on the execution sequence of each operation behavior and the screenshots of the interface corresponding to each operation behavior.
The device according to claim 15, characterized in that the device further includes:

A sending module, configured to send the detection information to a server, so as to send the detection information to multiple clients that need to detect the user interface.
An electronic device, characterized by including:

at least one processor; and

a memory communicatively connected to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor performs the method described in any one of claims 1-6. Method, or perform the method according to any one of claims 7-8.
A non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to cause the computer to execute the method described in any one of claims 1-6, or to execute claim 7 The method described in any one of -8.
A computer program product, characterized in that it includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6, or implements any one of claims 7-8 method described in the item.