WO2023220902A1 - User interface testing methods and apparatus - Google Patents

User interface testing methods and apparatus Download PDF

Info

Publication number
WO2023220902A1
WO2023220902A1 PCT/CN2022/093164 CN2022093164W WO2023220902A1 WO 2023220902 A1 WO2023220902 A1 WO 2023220902A1 CN 2022093164 W CN2022093164 W CN 2022093164W WO 2023220902 A1 WO2023220902 A1 WO 2023220902A1
Authority
WO
WIPO (PCT)
Prior art keywords
user interface
interface
operation behavior
screenshot
target
Prior art date
Application number
PCT/CN2022/093164
Other languages
French (fr)
Chinese (zh)
Inventor
杨佼
任伟博
邓文奇
Original Assignee
北京小米移动软件有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京小米移动软件有限公司 filed Critical 北京小米移动软件有限公司
Priority to PCT/CN2022/093164 priority Critical patent/WO2023220902A1/en
Priority to CN202280004099.5A priority patent/CN117413256A/en
Publication of WO2023220902A1 publication Critical patent/WO2023220902A1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software

Definitions

  • the present invention relates to the field of computer technology, and in particular, to a user interface detection method and device.
  • UI User Interface
  • UI testing is required, that is, the user interface needs to be detected.
  • UI testing is a technique for testing UI control elements on an application. For example, you can check whether the output results meet the expected test results by clicking on each UI control element on the user interface.
  • manual UI testing is often required, that is, UI testing is completed by testers manually operating controls, which results in high labor costs.
  • An embodiment of the first aspect of the present invention proposes a user interface detection method, which includes: displaying a user interface to be detected; obtaining detection information of the user interface; wherein the detection information is used to indicate the execution of multiple operation behaviors sequence, and interface screenshots corresponding to each of the operation behaviors; determine the target control triggered by the corresponding operation behavior based on each of the interface screenshots; and trigger each of the operations in the user interface sequentially based on the execution sequence.
  • the target control corresponding to the behavior.
  • obtaining the detection information of the user interface includes: obtaining the detection information generated by the client from the server; wherein the detection information includes a test script and a picture library; the picture library includes a plurality of Interface screenshots; the test script is used to indicate the execution sequence of the operation behaviors corresponding to each of the interface screenshots in the picture library.
  • determining the target control triggered by the corresponding operation behavior based on each interface screenshot includes: taking a screenshot of the user interface of the initial operation to obtain the target screenshot; comparing the target screenshot with the first operation Compare the interface screenshots corresponding to the behavior to use the control with display differences as the target control triggered by the first operation behavior; for any subsequent operation behavior except the first operation behavior, each time the target control is triggered once, Take a new screenshot of the user interface to update the target screenshot; compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so as to use the controls with display differences as the controls for the subsequent operation behavior.
  • the target control to trigger.
  • determining the target control triggered by the corresponding operation behavior based on each of the interface screenshots includes: using a trained object recognition model to perform object recognition on any of the interface screenshots to determine the target controls in the corresponding interface screenshots.
  • the control on which the cursor is focused; the control on which the cursor is focused is determined as the target control.
  • the object recognition model is trained using a training interface image; wherein the training interface image is marked with a cursor focus control.
  • the method further includes: at least once the target control corresponding to the operation behavior triggers a response exception, sending exception prompt information to the server, or displaying the exception prompt information on the user interface.
  • the embodiment of the second aspect of the present invention proposes a user interface detection method, which includes: displaying the user interface; and taking screenshots of the user interface every time the corresponding operation behavior is completed in response to the remote control instruction to obtain the corresponding operation behavior of each operation. Interface screenshots; based on the execution sequence of each operation behavior and the interface screenshots corresponding to each operation behavior, detection information for detecting the user interface is generated.
  • the method further includes: sending the detection information to a server, so as to send the detection information to a plurality of clients that are to detect the user interface.
  • the third embodiment of the present invention provides a user interface detection device, which includes: a first display module for displaying the user interface to be detected; an acquisition module for acquiring detection information of the user interface, wherein The detection information is used to indicate the execution sequence of multiple operation behaviors, as well as the interface screenshots corresponding to each operation behavior; the determination module is used to determine the target control triggered by the corresponding operation behavior based on each of the interface screenshots; first A processing module, configured to sequentially trigger the target controls corresponding to each of the operation behaviors in the user interface according to the execution sequence.
  • the acquisition module is also configured to: obtain the detection information generated by the client from the server; wherein the detection information includes a test script and a picture library; the picture library includes multiple interface screenshots; The above-mentioned test script is used to instruct the execution sequence of the operation behaviors corresponding to each of the interface screenshots in the picture library.
  • the determination module is also configured to: take a screenshot of the user interface of the initial operation to obtain a target screenshot; compare the target screenshot with the interface screenshot corresponding to the first operation behavior to display the existence
  • the different controls are used as the target controls triggered by the first operation behavior; for any subsequent operation behavior except the first operation behavior, whenever the target control is triggered once, the user interface is re-screenshoted to update the Target screenshot: compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so that the control with a display difference is used as the target control triggered by the subsequent operation behavior.
  • the determination module is also used to: use a trained object recognition model to perform object recognition on any of the interface screenshots to determine the control on which the cursor is focused in the corresponding interface screenshot; and determine the control on which the cursor is focused. , determined as the target control.
  • the object recognition model is trained using a training interface image; wherein the training interface image is marked with a cursor focus control.
  • the device further includes: a second processing module configured to send an exception prompt to the server at least once when a target control corresponding to the operation behavior triggers a response exception, or to display an exception prompt on the user interface. information.
  • a second processing module configured to send an exception prompt to the server at least once when a target control corresponding to the operation behavior triggers a response exception, or to display an exception prompt on the user interface. information.
  • the fourth embodiment of the present invention proposes a user interface detection device, which includes: a second display module for displaying the user interface; and a third processing module for processing the corresponding operation behavior whenever the corresponding operation behavior is completed in response to the remote control instruction.
  • Screenshots are taken of the user interface to obtain interface screenshots corresponding to each operation; a generation module is used to generate a screenshot of the interface according to the execution order of each operation and the interface screenshots corresponding to each operation. Detection information for detection performed by the user interface.
  • the device further includes: sending the detection information to a server, so as to send the detection information to multiple clients that are to detect the user interface.
  • a fifth embodiment of the present invention provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be executed by the at least one processor.
  • the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect, or to perform the method described in the second aspect.
  • the sixth aspect of the present invention provides a non-transient computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect, or to execute the second method. methods described in this aspect.
  • the seventh aspect embodiment of the present invention provides a computer program product, including a computer program, which when executed by a processor implements the method described in the first aspect, or implements the method described in the second aspect.
  • a method and device for detecting a user interface realizes obtaining detection information of the user interface by displaying the user interface to be detected, where the detection information is used to indicate the execution sequence of multiple operation behaviors, and each Screenshots of the interface corresponding to the operation behavior, thereby determining the target control triggered by the corresponding operation behavior based on each interface screenshot, and then sequentially triggering the target control corresponding to each operation behavior in the user interface according to the execution sequence.
  • the user interface can be automatically detected based on user interface screenshots, which not only ensures the accuracy of user interface detection, but also saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems.
  • Figure 1 is a schematic flow chart of a user interface detection method provided by an embodiment of the present invention.
  • Figure 2 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention.
  • Figure 3 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention.
  • Figure 4 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention.
  • Figure 5 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention.
  • Figure 6 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention.
  • Figure 7 is a schematic structural diagram of a user interface detection device provided by an embodiment of the present invention.
  • Figure 8 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention.
  • Figure 9 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention.
  • Figure 10 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention.
  • Figure 11 is a structural block diagram of an electronic device provided by an embodiment of the present invention.
  • FIG. 1 is a schematic flowchart of a user interface detection method provided by an embodiment of the present invention.
  • embodiments of the present invention provide a user interface detection method to determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface in the detection information of the user interface to be detected, so as to determine the target control triggered by the user interface to be detected.
  • the execution sequence in the detection information of the interface triggers the target controls corresponding to each operation behavior in the user interface sequentially, realizing automated detection of the user interface based on user interface screenshots and saving labor costs.
  • the user interface The detection method includes the following steps:
  • Step 101 Display the user interface to be tested.
  • the user interface detection method provided in this embodiment can be executed by a user interface detection device.
  • the user interface detection device may be an electronic device, or may be configured in the electronic device.
  • the electronic device can be any stationary or mobile computing device capable of data processing, such as mobile computing devices such as laptops, smart phones, and wearable devices, or stationary computing devices such as desktop computers, or servers, or other types of Computing equipment, etc., are not limited in this embodiment.
  • the user interface to be detected may be the user interface of any application program.
  • an application program refers to a computer program that performs one or more specific tasks. It runs in user mode, can interact with users, and has a visual user interface.
  • the number of user interfaces to be detected may be one or more, which is not limited in this embodiment. It should be noted that when there is one user interface to be detected, the user interface to be detected can be any user interface of any application; when there are multiple user interfaces to be detected, the multiple user interfaces can be Each user interface to be detected can be any multiple user interfaces of the same application, or any user interface of multiple different applications, or any multiple user interfaces of multiple different applications. This implementation There is no restriction on this in the example.
  • the user interface detection device can display the user interface to be detected to obtain detection information of the user interface to be detected.
  • the user interface detection device can directly display the user interface to be detected; when there are multiple user interfaces to be detected, the user interface The detection device can display multiple user interfaces to be detected at the same time, or can display a set number of user interfaces to be detected each time in a set order until all user interfaces to be detected are displayed.
  • this embodiment does not limit the determination of the setting order and the specific value of the setting number.
  • the setting can be made based on manual experience.
  • the setting order can be set to the user's
  • the operation sequence of the user interface operation is to set the number of settings to 3, or it can also be dynamically adjusted according to actual application requirements, which is not limited in this embodiment.
  • Step 102 Obtain detection information of the user interface, where the detection information is used to indicate the execution sequence of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior.
  • the detection device of the user interface obtains the detection information of the user interface, that is, obtains the user interface The execution sequence of multiple operations and the screenshots of the interface corresponding to each operation. Therefore, the user interface detection device can detect the user interface based on the execution order and screenshots of each interface in the acquired detection information of the user interface.
  • the user interface detection device can record the user interface to be detected while displaying the user interface to be detected, so that when the user operates the user interface to be detected, the user interface to be detected will be recorded.
  • the user's operation behavior is recorded to obtain detection information of the user interface to be detected that can be used to indicate the execution order of the user's multiple operation behaviors, as well as interface screenshots corresponding to each operation behavior.
  • the detection information of the user interface to be detected can be presented in the form of scripts and pictures, where the script is used to indicate the execution sequence of multiple operation behaviors, and the picture is used to indicate the interface screenshot corresponding to each operation behavior.
  • the user interface detection device obtains the detection information of the user interface to be detected, that is, the user interface to be detected is obtained. scripts and pictures, so that the user interface detection device can realize a new uninterrupted UI automated test by directly playing back the script.
  • the detection device of the user interface can use the method of taking a screenshot of each operation behavior to obtain the interface screenshot corresponding to each operation behavior.
  • the execution order of each operation behavior can be based on the interface Determine the order of screenshots taken. For example, assuming that the user interface to be detected is the login interface of a certain application, the user interface detection device can record the login interface while displaying the login interface, so that when the user enters the login interface, When entering a username and password on the login interface to log in, you can take screenshots of the interface for clicking the username input box to enter the username, clicking the password input box to enter the password, and clicking the login button to log in, so as to obtain the corresponding information for each operation. Interface screenshot.
  • the execution order of each operation behavior can be determined by marking in the interface screenshot.
  • the execution sequence is to click the user name input box to enter the user name, click the password input box to enter the password, click the login button to log in, etc., which are not limited in this embodiment.
  • Step 103 Determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface.
  • a control can be understood as any user interface element of any application program. Users can trigger controls to complete one or more specific tasks.
  • the target control may include a text input box, a button, a drop-down menu, etc., which is not limited in this embodiment.
  • the number of target controls triggered by the corresponding operation behavior may be one or more, and this is not limited in this embodiment.
  • the user interface detection device can determine the target control triggered by the corresponding operation behavior based on each interface screenshot.
  • the correspondence determined based on the screenshots of the login interface can be
  • the target controls triggered by the operation behavior can include user name input box, password input box and login button. It should be noted that the operation behavior corresponding to each interface screenshot may include only one operation behavior or may include multiple operation behaviors, which is not limited in this embodiment.
  • Step 104 According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.
  • the user interface detection device after the user interface detection device obtains the detection information of the user interface and determines the target control triggered by each operation behavior, it can execute the sequence of multiple operation behaviors indicated by the detection information of the user interface.
  • the detection of the user interface is completed by sequentially triggering the target controls corresponding to each operation behavior in the user interface. For example, in the above login interface example, since the execution sequence of the operation behaviors corresponding to the screenshots of the login interface is to click the user name input box to enter the user name, click the password input box to enter the password, and click the login button to log in, it is possible to follow this execution sequence. , sequentially trigger the user name input box, password input box and login button in the user interface to complete the detection of the user interface.
  • the target control triggered by the corresponding operation behavior can be determined based on the interface screenshot corresponding to each operation behavior indicated by the detection information of the user interface, so that the target control triggered by the corresponding operation behavior can be determined based on the detection information of the user interface.
  • the execution sequence of multiple operation behaviors indicated by the detection information is implemented by sequentially triggering the target controls corresponding to each operation behavior in the user interface to realize automatic detection of the user interface based on user interface screenshots, saving labor costs.
  • the user interface detection method achieves acquisition of detection information of the user interface by displaying the user interface to be detected, where the detection information is used to indicate the execution order of multiple operation behaviors and the interface corresponding to each operation behavior. Take screenshots to determine the target controls triggered by the corresponding operation behaviors based on the screenshots of each interface, and then sequentially trigger the target controls corresponding to each operation behavior in the user interface based on the execution sequence.
  • the user interface can be automatically detected based on user interface screenshots, which not only ensures the accuracy of user interface detection, but also saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems.
  • FIG. 2 is a schematic flowchart of another user interface detection method provided by an embodiment of the present invention.
  • the user interface detection method may include the following steps:
  • Step 201 Display the user interface to be tested.
  • Step 202 Obtain detection information of the user interface, where the detection information is used to indicate the execution order of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior.
  • steps 201-202 can refer to the execution process of the previous embodiment 101-102.
  • the principles are the same and will not be described again here.
  • Step 203 Take a screenshot of the initial running user interface to obtain the target screenshot.
  • the initially running user interface can be understood as any user interface in which the user who opens any application program for the first time has not performed any operation behavior.
  • the user interface that is initially run is the interface in which the user has not entered the user name and password to log in.
  • a screenshot can be taken of the initial running user interface to obtain the target screenshot.
  • Step 204 Compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so that the control with a display difference is used as the target control triggered by the first operation behavior.
  • the interface screenshot corresponding to the first operation behavior should be a user interface screenshot in which the first operation behavior is already being executed or has been completed. This is because if the first operation behavior has not yet been executed, the controls in the user interface will not change. Only when the first operation behavior is already being executed or has been completed, a certain control in the user interface will be significantly different from before the first operation behavior is executed.
  • the target screenshot can be compared with the interface screenshot corresponding to the first operation behavior, so that the control with a display difference is used as the target control triggered by the first operation behavior.
  • the target screenshot is the screenshot of the interface where the user has not entered the user name and password for the login operation.
  • the first operation behavior is the user entering the user name
  • the interface screenshot corresponding to the first operation behavior can be Take a screenshot of the interface where the user is inputting a user name in the user name input box of the login interface.
  • Step 205 For any subsequent operation behavior except the first operation behavior, each time the target control is triggered, the user interface is re-screenshoted to update the target screenshot.
  • the same method can be used to determine the target control of any subsequent operation behavior after the first operation behavior except for the first operation behavior.
  • the target control to trigger.
  • the user interface can be re-screenshoted to update the target screenshot.
  • the first operation behavior is for the user to enter a user name
  • the target control is the user name input box.
  • any subsequent operation behavior after the first operation behavior of the user entering a user name is for the user to enter a password
  • Step 206 Compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so that the control with display differences is used as the target control triggered by the subsequent operation behavior.
  • the interface screenshot corresponding to the subsequent operation behavior should be the user interface screenshot in which the subsequent operation behavior is already being executed or has been completed. This is because if the subsequent operation behavior has not yet been executed, the controls in the user interface will not change. When the subsequent operation behavior is already being executed or has been completed, a certain control in the user interface will be significantly different from before the subsequent operation behavior is executed.
  • the updated target screenshot can be compared with the interface screenshot corresponding to the subsequent operation behavior, so that the control with a display difference is used as the target control triggered by the subsequent operation behavior.
  • the updated target screenshot can be understood as the screenshot of the interface where the user has entered the user name in the user name input box of the login interface.
  • the subsequent operation behavior is the user entering the password. Therefore, the interface corresponding to the subsequent operation behavior
  • the screenshot can be a screenshot of the interface where the user is inputting the password in the password input box of the login interface.
  • comparing the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior it can be determined that the control of the password input box is executed in the subsequent operation behavior. There are significant differences before and after, so the password input box control can be determined as the target control triggered by subsequent operations.
  • Step 207 According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.
  • step 104 the execution process of this step can be referred to the execution process of step 104 in the above embodiment.
  • the principles are the same and will not be described again here.
  • the user interface detection method provided in this embodiment is to obtain the target screenshot by taking a screenshot of the initial running user interface, and compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so as to use controls with display differences as The target control triggered by the first operation behavior, so that for any subsequent operation behavior except the first operation behavior, every time the target control is triggered, the user interface is re-screenshoted to update the target screenshot, and then the updated target screenshot is , compare it with the interface screenshot corresponding to the subsequent operation behavior, so as to use the control with display differences as the target control triggered by the subsequent operation behavior. Therefore, the target control triggered by each operation behavior can be determined by comparing the interface screenshot corresponding to the operation behavior with the target screenshot. It is easy to implement, operable, can effectively save labor costs, and is suitable for a variety of operating systems.
  • the previous embodiment provides a possible implementation method of determining the target control triggered by the corresponding operation behavior based on each interface screenshot.
  • This embodiment provides another possible implementation method of determining the target control triggered by the corresponding operation behavior based on screenshots of each interface.
  • Figure 3 is a flow chart of another user interface detection method provided by an embodiment of the present invention.
  • the user interface detection method may include the following steps:
  • Step 301 Display the user interface to be tested.
  • Step 302 Obtain detection information of the user interface, where the detection information is used to indicate the execution order of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior.
  • steps 301-302 can refer to the execution process of the above-mentioned embodiments 101-102. The principles are the same and will not be described again.
  • Step 303 Use the trained object recognition model to perform object recognition on any interface screenshot to determine the control on which the cursor is focused in the corresponding interface screenshot.
  • control with the cursor focused can be understood as that in the user interface, when the user clicks on any control, the control obtains the cursor, which is the control with the cursor focused.
  • the control obtains the cursor, which is the control with the cursor focused.
  • the username input box obtains the cursor, which is the control on which the cursor is focused.
  • a trained object recognition model can be used to perform object recognition on any interface screenshot, thereby determining the control on which the cursor is focused in the corresponding interface screenshot.
  • the screenshot of the login interface of an application is a screenshot of the user entering the user name in the user name input box of the login interface.
  • the object recognition model may be trained using a training interface image, where a control with a cursor focus is marked in the training interface image.
  • the interface screenshots in this embodiment should be user interface screenshots when the operation behavior is being executed. This is because if the operation behavior has not yet been executed or has been completed, then a certain control in the user interface will no longer have the cursor focus. , only when the operation behavior is being executed, a certain control in the user interface will have the cursor focus.
  • Step 304 Determine the control on which the cursor is focused as the target control.
  • the control on which the cursor is focused can be determined as the target control.
  • the user name input box can be determined as the target control.
  • the screenshot of the login interface of an application is a screenshot of the user entering the password in the password input box of the login interface.
  • the trained object recognition model is used to perform object recognition on it, and it is determined that the cursor is focused on the screenshot of the interface.
  • the control is a password input box, so the password input box can be determined as the target control.
  • Step 305 According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.
  • step 104 the execution process of this step can be referred to the execution process of step 104 in the above embodiment.
  • the principles are the same and will not be described again here.
  • the user interface detection method provided by this embodiment adopts a trained object recognition model to perform object recognition on any interface screenshot to determine the control where the cursor is focused in the corresponding interface screenshot, thereby determining the control where the cursor is focused as the target. controls.
  • the object recognition model can be used to determine the control that the cursor is focused on in the interface screenshot, thereby determining the control as the target control, saving labor costs and improving the efficiency of determining the target control.
  • FIG. 4 is a schematic flowchart of another method for detecting a user interface provided by an embodiment of the present invention.
  • the user interface detection method may include the following steps:
  • Step 401 Display the user interface to be tested.
  • Step 402 Obtain the detection information generated by the client from the server.
  • the detection information may include a test script and a picture library, where the picture library includes multiple interface screenshots, and the test script is used to indicate the execution order of the operation behaviors corresponding to each interface screenshot in the picture library.
  • the detection information can be generated by the client of the user interface and sent to the server, so that the detection device of the user interface can obtain the detection information generated by the client from the server and perform subsequent operations.
  • the detection device of the user interface can obtain the detection information generated by the client from the server through online real-time collection, or can obtain the detection information generated by the client from the server through offline manual collection, or can also obtain the detection information generated by the client through the network.
  • the detection information generated by the client is obtained from the server through transmission or physical copy, etc. This embodiment does not limit this.
  • Step 403 Determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface.
  • Step 404 According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.
  • steps 403-404 can refer to the execution process of the above-mentioned embodiments 103-104.
  • the principles are the same and will not be described again here.
  • Step 405 At least one target control corresponding to the operation behavior triggers a response exception, and sends the exception prompt information to the server, or displays the exception prompt information on the user interface.
  • the detection device of the user interface can provide feedback to the server or the user interface based on whether there is a response exception when the target control is triggered.
  • the exception prompt information can be sent to the server, or the exception prompt information can be displayed on the user interface.
  • the operation behaviors include clicking the user name input box to enter the user name, clicking the password input box to enter the password, and clicking the login button to log in.
  • the target controls corresponding to the operation behaviors are the user name input box, password Input box and login button, so that when the user clicks on the user name input box and cannot enter the user name, an abnormal prompt message that the user name input box cannot be input normally can be sent to the server, or the user name input box cannot be displayed on the user interface. Exception prompt information for normal input.
  • the exception prompt information sent to the server may be in text form, table form, picture form, etc., which is not limited in this embodiment.
  • the abnormal prompt information displayed on the user interface may be in the form of a pop-up window, a text prompt, a picture, etc., and this embodiment is not limited.
  • the user interface detection method by displaying the user interface to be detected and obtaining the detection information generated by the client from the server, determines the target control triggered by the corresponding operation behavior based on the screenshots of each interface, thereby determining the execution sequence , sequentially trigger the target control corresponding to each operation behavior in the user interface, and then at least one target control corresponding to the operation behavior triggers a response exception, and sends the exception prompt information to the server, or displays the exception prompt information in the user interface.
  • the user interface can be automatically detected based on the detection information generated by the client. While ensuring the accuracy of user interface detection, it saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems. .
  • FIG. 5 is a schematic flowchart of another user interface detection method provided by an embodiment of the present invention.
  • the user interface detection method may include the following steps:
  • Step 501 Display the user interface.
  • the user interface detection method provided in this embodiment can be executed by a user interface detection device.
  • the user interface detection device may be an electronic device, or may be configured in the electronic device.
  • the electronic device can be any stationary or mobile computing device capable of data processing, such as mobile computing devices such as laptops, smart phones, and wearable devices, or stationary computing devices such as desktop computers, or servers, or other types of Computing equipment, etc., are not limited in this embodiment.
  • the user interface may be the user interface of any application program.
  • the number of user interfaces may be one or more, which is not limited in this embodiment. It should be noted that when there is one user interface, the user interface can be any user interface of any application; when there are multiple user interfaces, the multiple user interfaces to be detected should be the same user interface. Any number of user interfaces for an application.
  • the user interface detection device can display the user interface so as to generate detection information for detecting the user interface.
  • the user interface detection device can directly display the user interface; when there are multiple user interfaces, the user interface detection device can display multiple users at the same time.
  • the interface may also display a set number of user interfaces to be detected each time in a set order until all user interfaces are displayed. This is not limited in this embodiment. Among them, this embodiment does not limit the determination of the setting order and the specific value of the setting number.
  • the setting can be made based on manual experience. For example, the setting order can be set to the user's
  • the operation sequence of the user interface operation is to set the number of settings to 3, or it can also be dynamically adjusted according to actual application requirements, which is not limited in this embodiment.
  • Step 502 Whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface is taken to obtain an interface screenshot corresponding to each operation behavior.
  • the user interface detection device can obtain the interface screenshot corresponding to each operation behavior by taking a screenshot of the user interface. That is to say, the remote control command executes the corresponding operation behavior.
  • the user interface detection device takes a screenshot of the user interface, so that the interface screenshot corresponding to each operation behavior can be obtained.
  • the operation behaviors include clicking the user name input box to enter the user name, clicking the password input box to enter the password, and clicking the login button to log in. After the remote control command is executed, click the user name input box to enter the user name.
  • the user interface detection device can obtain the interface screenshot corresponding to the operation behavior of clicking the user name input box and inputting the user name by taking a screenshot of the user interface.
  • the user interface detection device can obtain the interface corresponding to the operation behavior of clicking on the password input box to enter the password by taking a screenshot of the user interface. screenshot.
  • the user interface detection device can obtain the interface screenshot corresponding to the operation behavior of clicking the login button to log in by taking a screenshot of the user interface.
  • Step 503 Generate detection information for detecting the user interface based on the execution order of each operation behavior and the interface screenshot corresponding to each operation behavior.
  • the execution order of each operation behavior can be determined based on the time sequence of each interface screenshot, and then the execution order of each operation behavior can be determined based on the time sequence of each operation behavior.
  • the execution sequence of the behavior and the interface screenshots corresponding to each operation behavior generate detection information used to detect the user interface.
  • the detection information is used to indicate the execution sequence of multiple operation behaviors, as well as the interface screenshots corresponding to each operation behavior.
  • the detection information may include a test script and a picture library, where the picture library includes multiple interface screenshots, and the test script is used to indicate the execution order of the operation behaviors corresponding to each interface screenshot in the picture library.
  • the chronological order of screenshots of each interface can be: first, the interface screenshot corresponding to the operation behavior of clicking the user name input box to enter the user name is obtained; secondly, the screenshot of the interface corresponding to the operation behavior of clicking the password input box to enter the password is obtained. A screenshot of the interface corresponding to the operation behavior. Finally, the screenshot of the interface corresponding to the operation behavior of clicking the login button to log in is obtained. Therefore, the execution order of each operation behavior is to first click the user name input box to enter the user name, and then click the password input box. Enter the password and finally click the login button to log in. From this, the detection information used to detect the user interface can be generated.
  • the user interface detection method provided by the embodiment of the present invention, by displaying the user interface, realizes that whenever the corresponding operation behavior is completed in response to the remote control instruction, a screenshot of the user interface is taken to obtain the interface screenshot corresponding to each operation behavior, so as to obtain the interface screenshot corresponding to each operation behavior.
  • the execution sequence of each operation behavior and the interface screenshot corresponding to each operation behavior are used to generate detection information for detecting the user interface.
  • a screenshot of the user interface can be obtained to obtain the interface screenshot corresponding to each operation behavior, so that based on the execution sequence of each operation behavior and the corresponding operation behavior Take screenshots of the interface to generate detection information for testing the user interface.
  • FIG. 6 is a schematic flowchart of another method for detecting a user interface provided by an embodiment of the present invention.
  • the detection method of the user interface may include the following steps:
  • Step 601 Display the user interface.
  • Step 602 Whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface is taken to obtain an interface screenshot corresponding to each operation behavior.
  • Step 603 Generate detection information for detecting the user interface based on the execution order of each operation behavior and the interface screenshot corresponding to each operation behavior.
  • steps 601-603 can refer to the execution process of steps 501-503 in the previous embodiment. The principles are the same and will not be described again here.
  • Step 604 Send the detection information to the server to send the detection information to multiple clients of the user interfaces to be detected.
  • the detection device of the user interface may send the detection information to the server to send the detection information to multiple clients of the user interface to be detected.
  • the detection device of the user interface can send the detection information to the server through network transmission or physical copy.
  • the user interface detection method provided by the embodiment of the present invention after generating detection information for detecting the user interface, sends the detection information to the server to send the detection information to multiple clients of the user interface to be detected.
  • the generated detection information for detecting the user interface can be sent through the server to the client that detects the user interface to be detected, so as to perform user interface detection.
  • the present invention also provides a user interface detection device, because the user interface detection device provided by the embodiment of the present invention is consistent with the above Figures 1 to 4
  • the detection method of the user interface provided by the embodiment corresponds to the detection method of the user interface. Therefore, the implementation of the detection method of the user interface is also applicable to the detection device of the user interface provided by the embodiment of the present invention, and will not be described in detail in the embodiment of the present invention.
  • FIG. 7 is a schematic structural diagram of a user interface detection device provided by an embodiment of the present invention.
  • the user interface detection device includes: a first display module 71 , an acquisition module 72 , a determination module 73 and a first processing module 74 .
  • the first display module 71 is used to display the user interface to be detected
  • the acquisition module 72 is used to obtain detection information of the user interface, where the detection information is used to indicate the execution sequence of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior;
  • the determination module 73 is used to determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface;
  • the first processing module 74 is used to sequentially trigger the target controls corresponding to each operation behavior in the user interface according to the execution sequence.
  • the acquisition module 72 is also used to:
  • the detection information includes test scripts and image libraries
  • the image gallery includes multiple interface screenshots
  • test script is used to instruct the execution sequence of the operation behaviors corresponding to the screenshots of each interface in the picture library.
  • the determination module 73 is also used to:
  • the user interface will be re-screenshoted to update the target screenshot
  • the determination module 73 is also used to:
  • the object recognition model is trained using a training interface image, where a control with a cursor focus is marked in the training interface image.
  • the embodiment of the present invention also provides a possible implementation of a user interface detection device.
  • Figure 8 is a schematic structural diagram of another user interface detection device provided by the embodiment of the present invention.
  • the user interface detection device also includes: a second processing module 75 .
  • the second processing module 75 is configured to trigger a response exception when the target control corresponding to at least one operation behavior occurs, and send exception prompt information to the server, or display the exception prompt information on the user interface.
  • the user interface detection device achieves acquisition of detection information of the user interface by displaying the user interface to be detected, wherein the detection information is used to indicate the execution order of multiple operation behaviors, and the corresponding operation behaviors of each operation. Take screenshots of the interface to determine the target controls triggered by the corresponding operation behaviors based on the screenshots of each interface, and then sequentially trigger the target controls corresponding to each operation behavior in the user interface based on the execution sequence.
  • the user interface can be automatically detected based on user interface screenshots, which not only ensures the accuracy of user interface detection, but also saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems.
  • the present invention also provides a user interface detection device, because the user interface detection device provided by the embodiment of the present invention is consistent with the above Figures 5 to 6
  • the detection method of the user interface provided by the embodiment corresponds to the detection method of the user interface. Therefore, the implementation of the detection method of the user interface is also applicable to the detection device of the user interface provided by the embodiment of the present invention, and will not be described in detail in the embodiment of the present invention.
  • FIG. 9 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention.
  • the user interface detection device includes: a second display module 91 , a third processing module 92 and a generation module 93 .
  • the second display module 91 is used to display the user interface
  • the third processing module 92 is configured to take a screenshot of the user interface every time the corresponding operation behavior is executed in response to the remote control command, so as to obtain the interface screenshot corresponding to each operation behavior;
  • the generation module 93 is configured to generate detection information for detecting the user interface based on the execution sequence of each operation behavior and the screenshots of the interface corresponding to each operation behavior.
  • the embodiment of the present invention also provides a possible implementation of a user interface detection device.
  • Figure 10 is a schematic structural diagram of another user interface detection device provided by the embodiment of the present invention.
  • the user interface detection device also includes: a sending module 94 .
  • the sending module 94 is used to send detection information to the server, so as to send the detection information to multiple clients of the user interfaces to be detected.
  • the user interface detection device by displaying the user interface, realizes that whenever the corresponding operation behavior is executed in response to the remote control instruction, a screenshot of the user interface is taken to obtain the interface screenshot corresponding to each operation behavior, so as to obtain the interface screenshot corresponding to each operation behavior.
  • the execution sequence of each operation behavior and the interface screenshot corresponding to each operation behavior are used to generate detection information for detecting the user interface.
  • a screenshot of the user interface can be obtained to obtain the interface screenshot corresponding to each operation behavior, so that based on the execution sequence of each operation behavior and the corresponding operation behavior Take screenshots of the interface to generate detection information for testing the user interface.
  • the present invention also proposes an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be used by the at least one processor. Instructions are executed by the at least one processor to enable the at least one processor to execute the user interface detection method provided by any of the foregoing embodiments.
  • the present invention also proposes a non-transient computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the user interface detection method provided in any of the foregoing embodiments. .
  • the present invention also proposes a computer program product, which includes a computer program.
  • the computer program is executed by a processor for detecting the user interface provided by any of the foregoing embodiments.
  • FIG. 11 is a structural block diagram of an electronic device provided by an embodiment of the present invention.
  • the electronic device shown in FIG. 11 is only an example and should not impose any restrictions on the functions and scope of use of the embodiments of the present invention.
  • the electronic device 10 includes a processor 11, which can be loaded into a random access memory (RAM, Random Access Memory) 13 according to a program stored in a read-only memory (ROM, Read Only Memory) 12 or from a memory 16 perform various appropriate actions and processing according to the program in it.
  • RAM random access memory
  • ROM Read Only Memory
  • RAM 13 various programs and data required for the operation of the electronic device 10 are also stored.
  • the processor 11, the ROM 12 and the RAM 13 are connected to each other via the bus 14.
  • An input/output (I/O, Input/Output) interface 15 is also connected to the bus 14 .
  • the following components are connected to the I/O interface 15: a memory 16 including a hard disk, etc.; and a communication section 17 including a network interface card such as a LAN (Local Area Network) card, a modem, etc., which is executed via a network such as the Internet Communication processing; driver 18 is also connected to I/O interface 15 as needed.
  • a network interface card such as a LAN (Local Area Network) card, a modem, etc.
  • driver 18 is also connected to I/O interface 15 as needed.
  • embodiments of the present invention may be implemented as computer software programs.
  • embodiments of the present invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program can be downloaded and installed from the network via the communication part 17.
  • the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the present invention are performed.
  • a storage medium including instructions such as a memory 16 including instructions.
  • the instructions can be executed by the processor 11 of the electronic device 10 to complete the above user interface detection method.
  • the storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage Equipment etc.
  • references to the terms “one embodiment,” “some embodiments,” “an example,” “specific examples,” or “some examples” or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the invention. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine different embodiments or examples and features of different embodiments or examples described in this specification unless they are inconsistent with each other.
  • first and second are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as “first” and “second” may explicitly or implicitly include at least one of these features.
  • “plurality” means at least two, such as two, three, etc., unless otherwise expressly and specifically limited.
  • a "computer-readable medium” may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
  • Non-exhaustive list of computer readable media include the following: electrical connections with one or more wires (electronic device), portable computer disk cartridges (magnetic device), random access memory (RAM), Read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM).
  • the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, and subsequently edited, interpreted, or otherwise suitable as necessary. process to obtain the program electronically and then store it in computer memory.
  • various parts of the present invention may be implemented in hardware, software, firmware, or a combination thereof.
  • various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if it is implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: discrete logic gate circuits with logic functions for implementing data signals; Logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.
  • the program can be stored in a computer-readable storage medium.
  • the program can be stored in a computer-readable storage medium.
  • each functional unit in various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module.
  • the above integrated modules can be implemented in the form of hardware or software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
  • the storage media mentioned above can be read-only memory, magnetic disks or optical disks, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

The present invention provides user interface testing methods and apparatus. One method comprises: by displaying a user interface to be tested, acquiring testing information of the user interface, the testing information being used for indicating an execution sequence of multiple operation behaviors and an interface screenshot corresponding to each operation behavior; according to each interface screenshot, determining a target control triggered by the corresponding operation behavior; and then sequentially triggering the target controls corresponding to the operation behaviors in the user interface according to the execution sequence. Therefore, the user interface can be automatically tested on the basis of the user interface screenshots, the labor cost is saved while the accuracy of user interface testing is ensured, and the user interface testing efficiency is improved. The present invention is applicable to various operating systems.

Description

用户界面的检测方法及装置User interface detection method and device 技术领域Technical field
本发明涉及计算机技术领域,尤其涉及一种用户界面的检测方法及装置。The present invention relates to the field of computer technology, and in particular, to a user interface detection method and device.
背景技术Background technique
UI(User Interface,用户界面)是应用程序和用户之间进行交互和信息交换的媒介。为了确保应用程序的用户界面能够符合开发人员的预期要求,需要进行UI测试,即需要对用户界面进行检测。UI (User Interface) is the medium for interaction and information exchange between applications and users. In order to ensure that the user interface of the application meets the developer's expected requirements, UI testing is required, that is, the user interface needs to be detected.
UI测试是一种对应用程序上的UI控件元素进行测试的技术。例如,可以通过点击用户界面上的各UI控件元素的方式来检测输出结果是否符合预期测试结果。相关技术中,往往需要人工手动进行UI测试,即通过测试人员手动操作控件来完成UI测试,人力成本较高。UI testing is a technique for testing UI control elements on an application. For example, you can check whether the output results meet the expected test results by clicking on each UI control element on the user interface. In related technologies, manual UI testing is often required, that is, UI testing is completed by testers manually operating controls, which results in high labor costs.
发明内容Contents of the invention
本发明第一方面实施例提出了一种用户界面的检测方法,包括:展示待检测的用户界面;获取所述用户界面的检测信息;其中,所述检测信息用于指示多次操作行为的执行顺序,以及各次所述操作行为对应的界面截图;根据各所述界面截图,确定对应操作行为所触发的目标控件;依据所述执行顺序,在所述用户界面中顺序触发各次所述操作行为对应的目标控件。An embodiment of the first aspect of the present invention proposes a user interface detection method, which includes: displaying a user interface to be detected; obtaining detection information of the user interface; wherein the detection information is used to indicate the execution of multiple operation behaviors sequence, and interface screenshots corresponding to each of the operation behaviors; determine the target control triggered by the corresponding operation behavior based on each of the interface screenshots; and trigger each of the operations in the user interface sequentially based on the execution sequence. The target control corresponding to the behavior.
可选地,所述获取所述用户界面的检测信息,包括:从服务器获取客户端生成的所述检测信息;其中,所述检测信息包括测试脚本和图片库;所述图片库中包括多个界面截图;所述测试脚本,用于指示所述图片库中各所述界面截图所对应操作行为的执行顺序。Optionally, obtaining the detection information of the user interface includes: obtaining the detection information generated by the client from the server; wherein the detection information includes a test script and a picture library; the picture library includes a plurality of Interface screenshots; the test script is used to indicate the execution sequence of the operation behaviors corresponding to each of the interface screenshots in the picture library.
可选地,所述根据各所述界面截图,确定对应操作行为所触发的目标控件,包括:对初始运行的所述用户界面进行截图,以得到目标截图;将所述目标截图,与首次操作行为对应的界面截图进行比较,以将存在显示差异的控件作为所述首次操作行为所触发的目标控件;针对除首次操作行为之后的任意一次后续操作行为,每当执行一次对目标控件的触发,对所述用户界面重新截图,以更新所述目标截图;将更新的所述目标截图,与所述后续操作行为对应的界面截图进行比较,以将存在显示差异的控件作为所述后续操作行为所触发的目标控件。Optionally, determining the target control triggered by the corresponding operation behavior based on each interface screenshot includes: taking a screenshot of the user interface of the initial operation to obtain the target screenshot; comparing the target screenshot with the first operation Compare the interface screenshots corresponding to the behavior to use the control with display differences as the target control triggered by the first operation behavior; for any subsequent operation behavior except the first operation behavior, each time the target control is triggered once, Take a new screenshot of the user interface to update the target screenshot; compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so as to use the controls with display differences as the controls for the subsequent operation behavior. The target control to trigger.
可选地,所述根据各所述界面截图,确定对应操作行为所触发的目标控件,包括:采用经过训练的对象识别模型,对任意的所述界面截图进行对象识别,以确定对应界面截图中光标聚焦的控件;将所述光标聚焦的控件,确定为所述目标控件。Optionally, determining the target control triggered by the corresponding operation behavior based on each of the interface screenshots includes: using a trained object recognition model to perform object recognition on any of the interface screenshots to determine the target controls in the corresponding interface screenshots. The control on which the cursor is focused; the control on which the cursor is focused is determined as the target control.
可选地,所述对象识别模型是采用训练界面图像进行训练得到;其中,所述训练界面图像中标注有光标聚焦的控件。Optionally, the object recognition model is trained using a training interface image; wherein the training interface image is marked with a cursor focus control.
可选地,所述方法还包括:至少一次所述操作行为对应的目标控件触发存在响应异常,向服务器发送异常提示信息,或者,在所述用户界面展示异常提示信息。Optionally, the method further includes: at least once the target control corresponding to the operation behavior triggers a response exception, sending exception prompt information to the server, or displaying the exception prompt information on the user interface.
本发明第二方面实施例提出了一种用户界面的检测方法,包括:展示用户界面;每当响应遥控指令执行对应的操作行为完毕,对所述用户界面进行截图,以得到各次操作行为对应的界面截图;根据各次操作行为的执行顺序,和各次所述操作行为对应的界面截图,生成用于对所述用户界面进行检测的检测信息。The embodiment of the second aspect of the present invention proposes a user interface detection method, which includes: displaying the user interface; and taking screenshots of the user interface every time the corresponding operation behavior is completed in response to the remote control instruction to obtain the corresponding operation behavior of each operation. Interface screenshots; based on the execution sequence of each operation behavior and the interface screenshots corresponding to each operation behavior, detection information for detecting the user interface is generated.
可选地,所述方法还包括:向服务器发送所述检测信息,以将所述检测信息发送至多个待检测所述用户界面的客户端。Optionally, the method further includes: sending the detection information to a server, so as to send the detection information to a plurality of clients that are to detect the user interface.
本发明第三方面实施例提出了一种用户界面的检测装置,包括:第一展示模块,用于展示待检测的用户界面;获取模块,用于获取所述用户界面的检测信息,其中,所述检测信息用于指示多次操作行为的执行顺序,以及各次所述操作行为对应的界面截图;确定模块,用于根据各所述界面截图,确定对应操作行为所触发的目标控件;第一处理模块,用于依据所述执行顺序,在所述用户界面中顺序触发各次所述操作行为对应的目标控件。The third embodiment of the present invention provides a user interface detection device, which includes: a first display module for displaying the user interface to be detected; an acquisition module for acquiring detection information of the user interface, wherein The detection information is used to indicate the execution sequence of multiple operation behaviors, as well as the interface screenshots corresponding to each operation behavior; the determination module is used to determine the target control triggered by the corresponding operation behavior based on each of the interface screenshots; first A processing module, configured to sequentially trigger the target controls corresponding to each of the operation behaviors in the user interface according to the execution sequence.
可选地,所述获取模块,还用于:从服务器获取客户端生成的所述检测信息;其中,所述检测信息包括测试脚本和图片库;所述图片库中包括多个界面截图;所述测试脚本,用于指示所述图片库中各所述界面截图所对应操作行为的执行顺序。Optionally, the acquisition module is also configured to: obtain the detection information generated by the client from the server; wherein the detection information includes a test script and a picture library; the picture library includes multiple interface screenshots; The above-mentioned test script is used to instruct the execution sequence of the operation behaviors corresponding to each of the interface screenshots in the picture library.
可选地,所述确定模块,还用于:对初始运行的所述用户界面进行截图,以得到目标截图;将所述目标截图,与首次操作行为对应的界面截图进行比较,以将存在显示差异的控件作为所述首次操作行为所触发的目标控件;针对除首次操作行为之后的任意一次后续操作行为,每当执行一次对目标控件的触发,对所述用户界面重新截图,以更新所述目标截图;将更新的所述目标截图,与所述后续操作行为对 应的界面截图进行比较,以将存在显示差异的控件作为所述后续操作行为所触发的目标控件。Optionally, the determination module is also configured to: take a screenshot of the user interface of the initial operation to obtain a target screenshot; compare the target screenshot with the interface screenshot corresponding to the first operation behavior to display the existence The different controls are used as the target controls triggered by the first operation behavior; for any subsequent operation behavior except the first operation behavior, whenever the target control is triggered once, the user interface is re-screenshoted to update the Target screenshot: compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so that the control with a display difference is used as the target control triggered by the subsequent operation behavior.
可选地,所述确定模块,还用于:采用经过训练的对象识别模型,对任意的所述界面截图进行对象识别,以确定对应界面截图中光标聚焦的控件;将所述光标聚焦的控件,确定为所述目标控件。Optionally, the determination module is also used to: use a trained object recognition model to perform object recognition on any of the interface screenshots to determine the control on which the cursor is focused in the corresponding interface screenshot; and determine the control on which the cursor is focused. , determined as the target control.
可选地,所述对象识别模型是采用训练界面图像进行训练得到;其中,所述训练界面图像中标注有光标聚焦的控件。Optionally, the object recognition model is trained using a training interface image; wherein the training interface image is marked with a cursor focus control.
可选地,所述装置,还包括:第二处理模块,用于至少一次所述操作行为对应的目标控件触发存在响应异常,向服务器发送异常提示信息,或者,在所述用户界面展示异常提示信息。Optionally, the device further includes: a second processing module configured to send an exception prompt to the server at least once when a target control corresponding to the operation behavior triggers a response exception, or to display an exception prompt on the user interface. information.
本发明第四方面实施例提出了一种用户界面的检测装置,包括:第二展示模块,用于展示用户界面;第三处理模块,用于每当响应遥控指令执行对应的操作行为完毕,对所述用户界面进行截图,以得到各次操作行为对应的界面截图;生成模块,用于根据各次操作行为的执行顺序,和各次所述操作行为对应的界面截图,生成用于对所述用户界面进行检测的检测信息。The fourth embodiment of the present invention proposes a user interface detection device, which includes: a second display module for displaying the user interface; and a third processing module for processing the corresponding operation behavior whenever the corresponding operation behavior is completed in response to the remote control instruction. Screenshots are taken of the user interface to obtain interface screenshots corresponding to each operation; a generation module is used to generate a screenshot of the interface according to the execution order of each operation and the interface screenshots corresponding to each operation. Detection information for detection performed by the user interface.
可选地,所述装置,还包括:向服务器发送所述检测信息,以将所述检测信息发送至多个待检测所述用户界面的客户端。Optionally, the device further includes: sending the detection information to a server, so as to send the detection information to multiple clients that are to detect the user interface.
本发明第五方面实施例提出了一种电子设备,包括:至少一个处理器;以及与所述至少一个处理器通信连接的存储器;其中,所述存储器存储有可被所述至少一个处理器执行的指令,所述指令被所述至少一个处理器执行,以使所述至少一个处理器能够执行第一方面所述的方法,或者,执行第二方面所述的方法。A fifth embodiment of the present invention provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect, or to perform the method described in the second aspect.
本发明第六方面实施例提出了一种存储有计算机指令的非瞬时计算机可读存储介质,其中,所述计算机指令用于使所述计算机执行第一方面所述的方法,或者,执行第二方面所述的方法。The sixth aspect of the present invention provides a non-transient computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method described in the first aspect, or to execute the second method. methods described in this aspect.
本发明第七方面实施例提出了一种计算机程序产品,包括计算机程序,所述计算机程序在被处理器执行时实现第一方面所述的方法,或者,实现第二方面所述的方法。The seventh aspect embodiment of the present invention provides a computer program product, including a computer program, which when executed by a processor implements the method described in the first aspect, or implements the method described in the second aspect.
本发明实施例提供的一种用户界面的检测方法及装置,通过展示待检测的用户界面,实现获取用户界面的检测信息,其中,检测信息用于指示多次操作行为的执行顺序,以及各次操作行为对应的界面截图,从而根据各界面截图,确定对应操作行为所触发的目标控件,进而依据执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件。由此,可实现基于用户界面截图对用户界面进行自动化检测,在保证用户界面检测的准确度的同时,节省了人力成本,提高了用户界面检测的效率,可适用于多种操作系统。A method and device for detecting a user interface provided by an embodiment of the present invention realizes obtaining detection information of the user interface by displaying the user interface to be detected, where the detection information is used to indicate the execution sequence of multiple operation behaviors, and each Screenshots of the interface corresponding to the operation behavior, thereby determining the target control triggered by the corresponding operation behavior based on each interface screenshot, and then sequentially triggering the target control corresponding to each operation behavior in the user interface according to the execution sequence. As a result, the user interface can be automatically detected based on user interface screenshots, which not only ensures the accuracy of user interface detection, but also saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems.
本发明附加的方面和优点将在下面的描述中部分给出,部分将从下面的描述中变得明显,或通过本发明的实践了解到。Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
附图说明Description of the drawings
本发明上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解,其中:The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
图1为本发明实施例所提供的一种用户界面的检测方法的流程示意图;Figure 1 is a schematic flow chart of a user interface detection method provided by an embodiment of the present invention;
图2为本发明实施例所提供的另一种用户界面的检测方法的流程示意图;Figure 2 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention;
图3为本发明实施例所提供的另一种用户界面的检测方法的流程示意图;Figure 3 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention;
图4为本发明实施例所提供的另一种用户界面的检测方法的流程示意图;Figure 4 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention;
图5为本发明实施例所提供的另一种用户界面的检测方法的流程示意图;Figure 5 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention;
图6为本发明实施例所提供的另一种用户界面的检测方法的流程示意图;Figure 6 is a schematic flow chart of another user interface detection method provided by an embodiment of the present invention;
图7为本发明实施例所提供的一种用户界面的检测装置的结构示意图;Figure 7 is a schematic structural diagram of a user interface detection device provided by an embodiment of the present invention;
图8为本发明实施例所提供的另一种用户界面的检测装置的结构示意图;Figure 8 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention;
图9为本发明实施例所提供的另一种用户界面的检测装置的结构示意图;Figure 9 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention;
图10为本发明实施例所提供的另一种用户界面的检测装置的结构示意图;以及Figure 10 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention; and
图11为本发明实施例提供的一种电子设备的结构框图。Figure 11 is a structural block diagram of an electronic device provided by an embodiment of the present invention.
具体实施方式Detailed ways
下面详细描述本发明的实施例,所述实施例的示例在附图中示出,其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的,旨在用于解释本发明,而不能理解为对本发明的限制。Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present invention and are not to be construed as limiting the present invention.
下面参考附图描述本发明实施例的用户界面的检测方法及装置。The user interface detection method and device according to the embodiment of the present invention will be described below with reference to the accompanying drawings.
图1为本发明实施例所提供的一种用户界面的检测方法的流程示意图。FIG. 1 is a schematic flowchart of a user interface detection method provided by an embodiment of the present invention.
相关技术中,往往需要人工手动进行UI测试,即通过测试人员手动操作控件来完成UI测试,人力成本较高。In related technologies, manual UI testing is often required, that is, UI testing is completed by testers manually operating controls, which results in high labor costs.
针对这一问题,本发明实施例提供了用户界面的检测方法,以实现根据待检测的用户界面的检测信息中的各界面截图,确定对应操作行为所触发的目标控件,从而依据待检测的用户界面的检测信息中的执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件,实现基于用户界面截图对用户界面进行自动化检测,节省人力成本,如图1所示,该用户界面的检测方法包括以下步骤:In response to this problem, embodiments of the present invention provide a user interface detection method to determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface in the detection information of the user interface to be detected, so as to determine the target control triggered by the user interface to be detected. The execution sequence in the detection information of the interface triggers the target controls corresponding to each operation behavior in the user interface sequentially, realizing automated detection of the user interface based on user interface screenshots and saving labor costs. As shown in Figure 1, the user interface The detection method includes the following steps:
步骤101,展示待检测的用户界面。Step 101: Display the user interface to be tested.
需要说明的是,本实施例提供的用户界面的检测方法,可以由用户界面的检测装置执行。该用户界面的检测装置可以为电子设备,也可以被配置在电子设备中。其中,电子设备,可以是任意能够进行数据处理的静止或者移动计算设备,例如笔记本电脑、智能手机、可穿戴设备等移动计算设备,或者台式计算机等静止的计算设备,或者服务器,或者其它类型的计算设备等,本实施例中对此不作限制。It should be noted that the user interface detection method provided in this embodiment can be executed by a user interface detection device. The user interface detection device may be an electronic device, or may be configured in the electronic device. Among them, the electronic device can be any stationary or mobile computing device capable of data processing, such as mobile computing devices such as laptops, smart phones, and wearable devices, or stationary computing devices such as desktop computers, or servers, or other types of Computing equipment, etc., are not limited in this embodiment.
在本实施例中,待检测的用户界面可以为任一应用程序的用户界面。其中,应用程序是指为完成某项或多项特定工作的计算机程序,它运行在用户模式,可以和用户进行交互,具有可视的用户界面。In this embodiment, the user interface to be detected may be the user interface of any application program. Among them, an application program refers to a computer program that performs one or more specific tasks. It runs in user mode, can interact with users, and has a visual user interface.
在本实施例中,待检测的用户界面的数量可以为一个或者多个,本实施例中对此不作限制。需要说明的是,在待检测的用户界面为一个的情况下,该待检测的用户界面可以为任一应用程序的任意一个用户界面;在待检测的用户界面为多个的情况下,该多个待检测的用户界面可以是同一个应用程序的任意多个用户界面,也可以是多个不同应用程序的任意一个用户界面,还可以是多个不同应用程序的任意多个用户界面,本实施例中对此不作限制。In this embodiment, the number of user interfaces to be detected may be one or more, which is not limited in this embodiment. It should be noted that when there is one user interface to be detected, the user interface to be detected can be any user interface of any application; when there are multiple user interfaces to be detected, the multiple user interfaces can be Each user interface to be detected can be any multiple user interfaces of the same application, or any user interface of multiple different applications, or any multiple user interfaces of multiple different applications. This implementation There is no restriction on this in the example.
在本实施例中,用户界面的检测装置可以对待检测的用户界面进行展示,以获取待检测的用户界面的检测信息。作为一种可能的实现方式,在待检测的用户界面为一个的情况下,用户界面的检测装置可以直接展示该待检测的用户界面;在待检测的用户界面为多个的情况下,用户界面的检测装置可以同时展示多个待检测的用户界面,也可以按照设定次序每次展示设定个数个待检测的用户界面,直至所有的待检测的用户界面都展示完毕为止,本实施例中对此不作限制。其中,对于设定次序的确定和对于设定个数的具体取值本实施例并没有作出限定,可选地,可以根据人工经验进行设定,例如,可以将设定次序设定为用户对用户界面进行操作的操作顺序,将设定个数设定为3个,或者,也可以根据实际应用需求进行动态调整,本实施例中对此不作限制。In this embodiment, the user interface detection device can display the user interface to be detected to obtain detection information of the user interface to be detected. As a possible implementation method, when there is one user interface to be detected, the user interface detection device can directly display the user interface to be detected; when there are multiple user interfaces to be detected, the user interface The detection device can display multiple user interfaces to be detected at the same time, or can display a set number of user interfaces to be detected each time in a set order until all user interfaces to be detected are displayed. In this embodiment There are no restrictions on this. Among them, this embodiment does not limit the determination of the setting order and the specific value of the setting number. Alternatively, the setting can be made based on manual experience. For example, the setting order can be set to the user's The operation sequence of the user interface operation is to set the number of settings to 3, or it can also be dynamically adjusted according to actual application requirements, which is not limited in this embodiment.
步骤102,获取用户界面的检测信息,其中,检测信息用于指示多次操作行为的执行顺序,以及各次操作行为对应的界面截图。Step 102: Obtain detection information of the user interface, where the detection information is used to indicate the execution sequence of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior.
可以理解的是,用户在对用户界面进行操作时,如果存在多次操作行为,则该多次操作行为之间会存在执行顺序,并且每次操作行为都有其对应的界面截图,从而待检测的用户界面的检测信息应能够指示多次操作行为的执行顺序,以及各次操作行为对应的界面截图。It is understandable that when a user operates the user interface, if there are multiple operations, there will be an execution sequence between the multiple operations, and each operation will have its corresponding interface screenshot, so it needs to be detected. The detection information of the user interface should be able to indicate the execution sequence of multiple operations, as well as the interface screenshots corresponding to each operation.
在本实施例中,由于检测信息可以用于指示多次操作行为的执行顺序,以及各次操作行为对应的界面截图,从而用户界面的检测装置获取到用户界面的检测信息,即获取到用户界面的多次操作行为的执行顺序和各次操作行为对应的界面截图。由此,用户界面的检测装置可以根据获取到的用户界面的检测信息中的执行顺序和各界面截图,对用户界面进行检测。In this embodiment, since the detection information can be used to indicate the execution sequence of multiple operation behaviors, as well as the interface screenshots corresponding to each operation behavior, the detection device of the user interface obtains the detection information of the user interface, that is, obtains the user interface The execution sequence of multiple operations and the screenshots of the interface corresponding to each operation. Therefore, the user interface detection device can detect the user interface based on the execution order and screenshots of each interface in the acquired detection information of the user interface.
在本实施例的一种可能的实现方式中,用户界面的检测装置可以在展示待检测的用户界面的同时,对待检测的用户界面进行录制,从而当用户对待检测的用户界面进行操作时,将用户的操作行为进行记录,以得到可以用于指示用户多次操作行为的执行顺序,以及各次操作行为对应的界面截图的待检测的用户界面的检测信息。可选地,待检测的用户界面的检测信息可以以脚本和图片的形式进行呈现,其中,脚本用于指示多次操作行为的执行顺序,图片用于指示各次操作行为对应的界面截图。In a possible implementation of this embodiment, the user interface detection device can record the user interface to be detected while displaying the user interface to be detected, so that when the user operates the user interface to be detected, the user interface to be detected will be recorded. The user's operation behavior is recorded to obtain detection information of the user interface to be detected that can be used to indicate the execution order of the user's multiple operation behaviors, as well as interface screenshots corresponding to each operation behavior. Optionally, the detection information of the user interface to be detected can be presented in the form of scripts and pictures, where the script is used to indicate the execution sequence of multiple operation behaviors, and the picture is used to indicate the interface screenshot corresponding to each operation behavior.
需要说明的是,在待检测的用户界面的检测信息以脚本和图片的形式进行呈现的情况下,用户界面的检测装置获取到待检测的用户界面的检测信息,即获取到待检测的用户界面的脚本和图片,由此,用户界面的检测装置可通过对脚本进行直接回放的方式来实现全新的无人干预的UI自动化测试。It should be noted that when the detection information of the user interface to be detected is presented in the form of scripts and pictures, the user interface detection device obtains the detection information of the user interface to be detected, that is, the user interface to be detected is obtained. scripts and pictures, so that the user interface detection device can realize a new uninterrupted UI automated test by directly playing back the script.
可选地,用户界面的检测装置在获取用户界面的检测信息时,可以采用一个操作行为截一次图的方式来得到每次操作行为对应的界面截图,此时各操作行为的执行顺序可以依据界面截图的截图顺序来确定,例如,假设待检测的用户界面为某一应用程序的登录界面,用户界面的检测装置可以在展示该登录界面的同时,对该登录界面进行录制,从而当用户在该登录界面输入用户名和密码进行登录操作时,可以分别对点击用户名输入框输入用户名、点击密码输入框输入密码和点击登录按钮进行登录的操作行为进行界面截图,以得到每次操作行为对应的界面截图。或者,可以采用一个用户界面截一次图的方式来得到包含多次操作行为的界面截图,此时各操作行为的执行顺序可以通过在界面截图中进行标注的方式来确定,例如,在上述登录界面例子中,可以在对登录界面进行界面截图后,通过在用户名输入框附近标注1,在密码输入框附近标注2以及在登录按钮附近标注3的方式来确定与登录界面截图对应的操作行为的执行顺序为点击用户名输入框输入用户名、点击密码输入框输入密码和点击登录按钮进行登录,等等,本实施例中对此不作限制。Optionally, when obtaining the detection information of the user interface, the detection device of the user interface can use the method of taking a screenshot of each operation behavior to obtain the interface screenshot corresponding to each operation behavior. At this time, the execution order of each operation behavior can be based on the interface Determine the order of screenshots taken. For example, assuming that the user interface to be detected is the login interface of a certain application, the user interface detection device can record the login interface while displaying the login interface, so that when the user enters the login interface, When entering a username and password on the login interface to log in, you can take screenshots of the interface for clicking the username input box to enter the username, clicking the password input box to enter the password, and clicking the login button to log in, so as to obtain the corresponding information for each operation. Interface screenshot. Alternatively, you can use the method of taking a screenshot of a user interface to obtain an interface screenshot containing multiple operation behaviors. At this time, the execution order of each operation behavior can be determined by marking in the interface screenshot. For example, in the above login interface For example, after taking a screenshot of the login interface, you can determine the operation behavior corresponding to the screenshot of the login interface by marking 1 near the user name input box, 2 near the password input box, and 3 near the login button. The execution sequence is to click the user name input box to enter the user name, click the password input box to enter the password, click the login button to log in, etc., which are not limited in this embodiment.
步骤103,根据各界面截图,确定对应操作行为所触发的目标控件。Step 103: Determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface.
在本实施例中,控件可以理解为任一应用程序的任一用户界面元素。用户可以通过触发控件来完成某项或多项特定工作。可选地,目标控件可以包括文本输入框、按钮、下拉式菜单等,本实施例中对此不作限制。In this embodiment, a control can be understood as any user interface element of any application program. Users can trigger controls to complete one or more specific tasks. Optionally, the target control may include a text input box, a button, a drop-down menu, etc., which is not limited in this embodiment.
在本实施例中,对应操作行为所触发的目标控件的数量可以为一个或者多个,本实施例中对此不作限制。In this embodiment, the number of target controls triggered by the corresponding operation behavior may be one or more, and this is not limited in this embodiment.
在本实施例中,界面截图与操作行为之间存在着对应关系,从而用户界面的检测装置可以根据各界面截图,确定对应操作行为所触发的目标控件。例如,在上述登录界面例子中,由于与登录界面截图对应的操作行为可以包括点击用户名输入框输入用户名、点击密码输入框输入密码和点击登录按钮进行登录,从而基于登录界面截图确定的对应操作行为所触发的目标控件可以包括用户名输入框、密码输入框和登录按钮。需要说明的是,各界面截图对应的操作行为可以只包括一种操作行为,也可以包括多种操作行为,本实施例中对此不作限制。In this embodiment, there is a corresponding relationship between interface screenshots and operation behaviors, so that the user interface detection device can determine the target control triggered by the corresponding operation behavior based on each interface screenshot. For example, in the above example of the login interface, since the operation behaviors corresponding to the screenshots of the login interface can include clicking on the username input box to enter the user name, clicking on the password input box to enter the password, and clicking on the login button to log in, the correspondence determined based on the screenshots of the login interface can be The target controls triggered by the operation behavior can include user name input box, password input box and login button. It should be noted that the operation behavior corresponding to each interface screenshot may include only one operation behavior or may include multiple operation behaviors, which is not limited in this embodiment.
步骤104,依据执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件。Step 104: According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.
在本实施例中,用户界面的检测装置在获取到用户界面的检测信息和确定好各操作行为所触发的目标控件之后,可以依据用户界面的检测信息所指示的多次操作行为的执行顺序,通过在用户界面中顺序触发各次操作行为对应的目标控件来完成对用户界面进行检测。例如,在上述登录界面例子中,由于与登录界面截图对应的操作行为的执行顺序为点击用户名输入框输入用户名、点击密码输入框输入密码和点击登录按钮进行登录,从而可以依据该执行顺序,在用户界面中顺序触发用户名输入框、密码输入框和登录按钮,完成对用户界面进行检测。In this embodiment, after the user interface detection device obtains the detection information of the user interface and determines the target control triggered by each operation behavior, it can execute the sequence of multiple operation behaviors indicated by the detection information of the user interface. The detection of the user interface is completed by sequentially triggering the target controls corresponding to each operation behavior in the user interface. For example, in the above login interface example, since the execution sequence of the operation behaviors corresponding to the screenshots of the login interface is to click the user name input box to enter the user name, click the password input box to enter the password, and click the login button to log in, it is possible to follow this execution sequence. , sequentially trigger the user name input box, password input box and login button in the user interface to complete the detection of the user interface.
可以理解的是,由于在获取到用户界面的检测信息后,可以基于用户界面的检测信息所指示的各次操作行为对应的界面截图确定对应操作行为所触发的目标控件,从而可以依据用户界面的检测信息所指示的多次操作行为的执行顺序,通过在用户界面中顺序触发各次操作行为对应的目标控件来实现基于用户界面截图对用户界面进行自动化检测,节省了人力成本。It can be understood that, after obtaining the detection information of the user interface, the target control triggered by the corresponding operation behavior can be determined based on the interface screenshot corresponding to each operation behavior indicated by the detection information of the user interface, so that the target control triggered by the corresponding operation behavior can be determined based on the detection information of the user interface. The execution sequence of multiple operation behaviors indicated by the detection information is implemented by sequentially triggering the target controls corresponding to each operation behavior in the user interface to realize automatic detection of the user interface based on user interface screenshots, saving labor costs.
本实施例提供的用户界面的检测方法,通过展示待检测的用户界面,实现获取用户界面的检测信息,其中,检测信息用于指示多次操作行为的执行顺序,以及各次操作行为对应的界面截图,从而根据各界面截图,确定对应操作行为所触发的目标控件,进而依据执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件。由此,可实现基于用户界面截图对用户界面进行自动化检测,在保证用户界面检测的准确度的同时,节省了人力成本,提高了用户界面检测的效率,可适用于多种操作系统。The user interface detection method provided in this embodiment achieves acquisition of detection information of the user interface by displaying the user interface to be detected, where the detection information is used to indicate the execution order of multiple operation behaviors and the interface corresponding to each operation behavior. Take screenshots to determine the target controls triggered by the corresponding operation behaviors based on the screenshots of each interface, and then sequentially trigger the target controls corresponding to each operation behavior in the user interface based on the execution sequence. As a result, the user interface can be automatically detected based on user interface screenshots, which not only ensures the accuracy of user interface detection, but also saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems.
通过上述分析可知,本发明实施例中需要根据各界面截图,确定对应操作行为所触发的目标控件,为了清楚说明本发明是如何根据各界面截图确定对应操作行为所触发的目标控件的,本发明还提出一种用户界面的检测方法。It can be seen from the above analysis that in the embodiment of the present invention, it is necessary to determine the target control triggered by the corresponding operation behavior based on each interface screenshot. In order to clearly explain how the present invention determines the target control triggered by the corresponding operation behavior based on each interface screenshot, the present invention A user interface detection method is also proposed.
图2为本发明实施例所提供的另一种用户界面的检测方法的流程示意图。FIG. 2 is a schematic flowchart of another user interface detection method provided by an embodiment of the present invention.
如图2所示,该用户界面的检测方法可以包括以下步骤:As shown in Figure 2, the user interface detection method may include the following steps:
步骤201,展示待检测的用户界面。Step 201: Display the user interface to be tested.
步骤202,获取用户界面的检测信息,其中,检测信息用于指示多次操作行为的执行顺序,以及各次操作行为对应的界面截图。Step 202: Obtain detection information of the user interface, where the detection information is used to indicate the execution order of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior.
需要说明的是,步骤201-202的执行过程可以参见上一实施例101-102的执行过程,原理相同,在此不再赘述。It should be noted that the execution process of steps 201-202 can refer to the execution process of the previous embodiment 101-102. The principles are the same and will not be described again here.
步骤203,对初始运行的用户界面进行截图,以得到目标截图。Step 203: Take a screenshot of the initial running user interface to obtain the target screenshot.
在本实施例中,初始运行的用户界面可以理解为任一应用程序首次打开的用户尚未进行任何操作行为的任一用户界面。例如,在某一应用程序的登录界面中,初始运行的用户界面即为用户尚未在该登录界面输入用户名和密码进行登录操作的界面。In this embodiment, the initially running user interface can be understood as any user interface in which the user who opens any application program for the first time has not performed any operation behavior. For example, in the login interface of a certain application, the user interface that is initially run is the interface in which the user has not entered the user name and password to log in.
在本实施例中,可以针对初始运行的用户界面进行截图,以得到目标截图。作为一种可能的实现方式,可以针对任一应用程序中初始运行的用户界面,采用设备自带的截图工具,或者设备已下载的截图工具,或者该应用程序提供的截图功能,等等,进行截图,从而得到初始运行的用户界面截图,并将其作为目标截图。In this embodiment, a screenshot can be taken of the initial running user interface to obtain the target screenshot. As a possible implementation method, you can use the screenshot tool that comes with the device, or the screenshot tool that has been downloaded by the device, or the screenshot function provided by the application, etc., for the user interface that is initially run in any application. Take a screenshot to get a screenshot of the user interface of the initial run and use it as the target screenshot.
步骤204,将目标截图,与首次操作行为对应的界面截图进行比较,以将存在显示差异的控件作为首次操作行为所触发的目标控件。Step 204: Compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so that the control with a display difference is used as the target control triggered by the first operation behavior.
可以理解的是,首次操作行为对应的界面截图应该是首次操作行为已经正在执行或者已经执行完毕的用户界面截图,这是因为如果首次操作行为尚未执行,那么用户界面中的控件不会发生变化,只有当首次操作行为已经正在执行或者已经执行完毕时,用户界面中的某一控件才会与首次操作行为执行前存在显著差异。It is understandable that the interface screenshot corresponding to the first operation behavior should be a user interface screenshot in which the first operation behavior is already being executed or has been completed. This is because if the first operation behavior has not yet been executed, the controls in the user interface will not change. Only when the first operation behavior is already being executed or has been completed, a certain control in the user interface will be significantly different from before the first operation behavior is executed.
在本实施例中,在得到目标截图之后,可以将目标截图与首次操作行为对应的界面截图进行比较,从而将存在显示差异的控件作为首次操作行为所触发的目标控件。例如,在某一应用程序的登录界面中,目标截图为用户尚未在该登录界面输入用户名和密码进行登录操作的界面截图,首次操作行为为用户输入用户名,从而首次操作行为对应的界面截图可以为用户正在该登录界面的用户名输入框输入用户名的界面截图,此时,将目标截图与首次操作行为对应的界面截图进行比较,可以确定用户名输入框这一控件在首次操作行为执行前后存在显著差异,从而可以将用户名输入框这一控件确定为首次操作行为所触发的目标控件。In this embodiment, after the target screenshot is obtained, the target screenshot can be compared with the interface screenshot corresponding to the first operation behavior, so that the control with a display difference is used as the target control triggered by the first operation behavior. For example, in the login interface of an application, the target screenshot is the screenshot of the interface where the user has not entered the user name and password for the login operation. The first operation behavior is the user entering the user name, so the interface screenshot corresponding to the first operation behavior can be Take a screenshot of the interface where the user is inputting a user name in the user name input box of the login interface. At this time, compare the target screenshot with the screenshot of the interface corresponding to the first operation behavior to determine whether the control of the user name input box is before and after the first operation behavior is executed. There are significant differences, so that the user name input box control can be determined as the target control triggered by the first operation behavior.
步骤205,针对除首次操作行为之后的任意一次后续操作行为,每当执行一次对目标控件的触发,对用户界面重新截图,以更新目标截图。Step 205: For any subsequent operation behavior except the first operation behavior, each time the target control is triggered, the user interface is re-screenshoted to update the target screenshot.
在本实施例中,在确定好首次操作行为所触发的目标控件之后,可以针对除首次操作行为之后的任意一次后续操作行为,采用相同的方式来确定首次操作行为之后的任意一次后续操作行为所触发的目标控件。具体来说,可以针对除首次操作行为之后的任意一次后续操作行为,每当执行一次对目标控件的触发,对用户界面重新截图,以更新目标截图。例如,在上述登录界面例子中,首次操作行为为用户输入用户名,目标控件为用户名输入框,假设除用户输入用户名这一首次操作行为之后的任意一次后续操作行为为用户输入密码,那么,可以在执行一次对用户名输入框这一目标控件的触发,即首次操作行为已经执行完毕之后,对用户界面重新截图,以将重新截图得到的用户界面截图作为新的目标截图,实现对目标截图的更新。In this embodiment, after determining the target control triggered by the first operation behavior, the same method can be used to determine the target control of any subsequent operation behavior after the first operation behavior except for the first operation behavior. The target control to trigger. Specifically, for any subsequent operation behavior except the first operation behavior, each time the target control is triggered, the user interface can be re-screenshoted to update the target screenshot. For example, in the above example of the login interface, the first operation behavior is for the user to enter a user name, and the target control is the user name input box. Assume that any subsequent operation behavior after the first operation behavior of the user entering a user name is for the user to enter a password, then , you can trigger the target control of the user name input box once, that is, after the first operation behavior has been executed, you can re-take a screenshot of the user interface, and use the re-taken screenshot of the user interface as a new target screenshot to achieve the goal. Screenshot update.
可以理解的是,由于是采用相同的方式来确定首次操作行为之后的任意一次后续操作行为所触发的目标控件,从而每当执行一次对目标控件的触发,需要对用户界面重新截图,以更新目标截图,进行与后续操作行为对应的界面截图的比较。It is understandable that since the same method is used to determine the target control triggered by any subsequent operation behavior after the first operation behavior, every time the target control is triggered, the user interface needs to be re-screenshoted to update the target. Take screenshots and compare them with interface screenshots corresponding to subsequent operations.
步骤206,将更新的目标截图,与后续操作行为对应的界面截图进行比较,以将存在显示差异的控件作为后续操作行为所触发的目标控件。Step 206: Compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so that the control with display differences is used as the target control triggered by the subsequent operation behavior.
类似地,后续操作行为对应的界面截图应该是后续操作行为已经正在执行或者已经执行完毕的用户界面截图,这是因为如果后续操作行为尚未执行,那么用户界面中的控件不会发生变化,只有当后续操作行为已经正在执行或者已经执行完毕时,用户界面中的某一控件才会与后续操作行为执行前存在显著差异。Similarly, the interface screenshot corresponding to the subsequent operation behavior should be the user interface screenshot in which the subsequent operation behavior is already being executed or has been completed. This is because if the subsequent operation behavior has not yet been executed, the controls in the user interface will not change. When the subsequent operation behavior is already being executed or has been completed, a certain control in the user interface will be significantly different from before the subsequent operation behavior is executed.
在本实施例中,在对目标截图进行更新之后,可以将更新的目标截图与后续操作行为对应的界面截图进行比较,从而将存在显示差异的控件作为后续操作行为所触发的目标控件。例如,在上述登录界面例子中,更新的目标截图可以理解为用户在该登录界面的用户名输入框输入完用户名的界面截图,后续操作行为为用户输入密码,从而,后续操作行为对应的界面截图可以为用户正在该登录界面的密码输入框输入密码的界面截图,此时,将更新的目标截图与后续操作行为对应的界面截图进行比较,可以确定密码输入框这一控件在后续操作行为执行前后存在显著差异,从而可以将密码输入框这一控件确定为后续操作行为所触发的目标控件。In this embodiment, after the target screenshot is updated, the updated target screenshot can be compared with the interface screenshot corresponding to the subsequent operation behavior, so that the control with a display difference is used as the target control triggered by the subsequent operation behavior. For example, in the above example of the login interface, the updated target screenshot can be understood as the screenshot of the interface where the user has entered the user name in the user name input box of the login interface. The subsequent operation behavior is the user entering the password. Therefore, the interface corresponding to the subsequent operation behavior The screenshot can be a screenshot of the interface where the user is inputting the password in the password input box of the login interface. At this time, comparing the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, it can be determined that the control of the password input box is executed in the subsequent operation behavior. There are significant differences before and after, so the password input box control can be determined as the target control triggered by subsequent operations.
步骤207,依据执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件。Step 207: According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.
需要说明的是,本步骤的执行过程可以参见上述实施例中步骤104的执行过程,原理相同,在此不再赘述。It should be noted that the execution process of this step can be referred to the execution process of step 104 in the above embodiment. The principles are the same and will not be described again here.
本实施例提供的用户界面的检测方法,通过对初始运行的用户界面进行截图,以得到目标截图,实现将目标截图,与首次操作行为对应的界面截图进行比较,以将存在显示差异的控件作为首次操作行为所触发的目标控件,从而针对除首次操作行为之后的任意一次后续操作行为,每当执行一次对目标控件的触发,对用户界面重新截图,以更新目标截图,进而将更新的目标截图,与后续操作行为对应的界面截图进行比较,以将存在显示差异的控件作为后续操作行为所触发的目标控件。由此,可采用将操作行为对应的界面截图与目标截图进行比较的方式来确定各操作行为所触发的目标控件,易实施、可操作,能有效节省人工成本,适用于多种操作系统。The user interface detection method provided in this embodiment is to obtain the target screenshot by taking a screenshot of the initial running user interface, and compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so as to use controls with display differences as The target control triggered by the first operation behavior, so that for any subsequent operation behavior except the first operation behavior, every time the target control is triggered, the user interface is re-screenshoted to update the target screenshot, and then the updated target screenshot is , compare it with the interface screenshot corresponding to the subsequent operation behavior, so as to use the control with display differences as the target control triggered by the subsequent operation behavior. Therefore, the target control triggered by each operation behavior can be determined by comparing the interface screenshot corresponding to the operation behavior with the target screenshot. It is easy to implement, operable, can effectively save labor costs, and is suitable for a variety of operating systems.
上一实施例提供了一种根据各界面截图确定对应操作行为所触发的目标控件可能的实现方式,为了更加清楚地说明本发明中根据各界面截图确定对应操作行为所触发的目标控件的过程,本实施例提供了另一种根据各界面截图确定对应操作行为所触发的目标控件可能的实现方式,图3为本发明实施例所提供的另一种用户界面的检测方法的流程示意图。The previous embodiment provides a possible implementation method of determining the target control triggered by the corresponding operation behavior based on each interface screenshot. In order to more clearly explain the process of determining the target control triggered by the corresponding operation behavior based on each interface screenshot in the present invention, This embodiment provides another possible implementation method of determining the target control triggered by the corresponding operation behavior based on screenshots of each interface. Figure 3 is a flow chart of another user interface detection method provided by an embodiment of the present invention.
如图3所示,该用户界面的检测方法可以包括以下步骤:As shown in Figure 3, the user interface detection method may include the following steps:
步骤301,展示待检测的用户界面。Step 301: Display the user interface to be tested.
步骤302,获取用户界面的检测信息,其中,检测信息用于指示多次操作行为的执行顺序,以及各次操作行为对应的界面截图。Step 302: Obtain detection information of the user interface, where the detection information is used to indicate the execution order of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior.
需要说明的是,步骤301-302的执行过程可以参见上述实施例101-102的执行过程,原理相同,在 此不再赘述。It should be noted that the execution process of steps 301-302 can refer to the execution process of the above-mentioned embodiments 101-102. The principles are the same and will not be described again.
步骤303,采用经过训练的对象识别模型,对任意的界面截图进行对象识别,以确定对应界面截图中光标聚焦的控件。Step 303: Use the trained object recognition model to perform object recognition on any interface screenshot to determine the control on which the cursor is focused in the corresponding interface screenshot.
在本实施例中,光标聚焦的控件可以理解为在用户界面中,当用户点击任一控件时,该控件获取到光标,即为光标聚焦的控件。例如,在某一应用程序的登录界面中,当用户点击该登录界面的用户名输入框进行用户名输入操作时,用户名输入框获取到光标,即为光标聚焦的控件。In this embodiment, the control with the cursor focused can be understood as that in the user interface, when the user clicks on any control, the control obtains the cursor, which is the control with the cursor focused. For example, in the login interface of an application, when the user clicks the username input box of the login interface to enter the username, the username input box obtains the cursor, which is the control on which the cursor is focused.
在本实施例中,可以采用经过训练的对象识别模型,对任意的界面截图进行对象识别,从而确定对应界面截图中光标聚焦的控件。例如,假设某一应用程序的登录界面截图为用户正在该登录界面的用户名输入框输入用户名的界面截图,采用经过训练的对象识别模型对其进行对象识别,可以确定该界面截图中光标聚焦的控件为用户名输入框。可选地,对象识别模型可以是采用训练界面图像进行训练得到的,其中,训练界面图像中标注有光标聚焦的控件。In this embodiment, a trained object recognition model can be used to perform object recognition on any interface screenshot, thereby determining the control on which the cursor is focused in the corresponding interface screenshot. For example, assume that the screenshot of the login interface of an application is a screenshot of the user entering the user name in the user name input box of the login interface. Using a trained object recognition model to perform object recognition, it can be determined that the cursor focus in the screenshot of the interface is The control is the username input box. Optionally, the object recognition model may be trained using a training interface image, where a control with a cursor focus is marked in the training interface image.
需要说明的是,本实施例中的界面截图应该是操作行为正在执行的用户界面截图,这是因为如果操作行为尚未执行或者已经执行完毕,那么用户界面中的某一控件将不再有光标聚焦,只有当操作行为正在执行时,用户界面中的某一控件才会有光标聚焦。It should be noted that the interface screenshots in this embodiment should be user interface screenshots when the operation behavior is being executed. This is because if the operation behavior has not yet been executed or has been completed, then a certain control in the user interface will no longer have the cursor focus. , only when the operation behavior is being executed, a certain control in the user interface will have the cursor focus.
步骤304,将光标聚焦的控件,确定为目标控件。Step 304: Determine the control on which the cursor is focused as the target control.
在本实施例中,在确定好界面截图中光标聚焦的控件之后,可以将光标聚焦的控件,确定为目标控件。例如,在上述登录界面例子中,由于确定的该界面截图中光标聚焦的控件为用户名输入框,可以将用户名输入框确定为目标控件。类似地,假设某一应用程序的登录界面截图为用户正在该登录界面的密码输入框输入密码的界面截图,采用经过训练的对象识别模型对其进行对象识别,确定的该界面截图中光标聚焦的控件为密码输入框,从而可以将密码输入框确定为目标控件。In this embodiment, after determining the control on which the cursor is focused in the interface screenshot, the control on which the cursor is focused can be determined as the target control. For example, in the above example of the login interface, since it is determined that the control focused by the cursor in the screenshot of the interface is the user name input box, the user name input box can be determined as the target control. Similarly, assume that the screenshot of the login interface of an application is a screenshot of the user entering the password in the password input box of the login interface. The trained object recognition model is used to perform object recognition on it, and it is determined that the cursor is focused on the screenshot of the interface. The control is a password input box, so the password input box can be determined as the target control.
步骤305,依据执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件。Step 305: According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.
需要说明的是,本步骤的执行过程可以参见上述实施例中步骤104的执行过程,原理相同,在此不再赘述。It should be noted that the execution process of this step can be referred to the execution process of step 104 in the above embodiment. The principles are the same and will not be described again here.
本实施例提供的用户界面的检测方法,通过采用经过训练的对象识别模型,对任意的界面截图进行对象识别,以确定对应界面截图中光标聚焦的控件,从而将光标聚焦的控件,确定为目标控件。由此,可实现采用对象识别模型确定界面截图中光标聚焦的控件,从而将该控件确定为目标控件,节省了人力成本,提高了确定目标控件的效率。The user interface detection method provided by this embodiment adopts a trained object recognition model to perform object recognition on any interface screenshot to determine the control where the cursor is focused in the corresponding interface screenshot, thereby determining the control where the cursor is focused as the target. controls. As a result, the object recognition model can be used to determine the control that the cursor is focused on in the interface screenshot, thereby determining the control as the target control, saving labor costs and improving the efficiency of determining the target control.
为了清楚说明上述实施例,本实施例提供了另一种用户界面的检测方法,图4为本发明实施例所提供的另一种用户界面的检测方法的流程示意图。In order to clearly explain the above embodiment, this embodiment provides another method for detecting a user interface. FIG. 4 is a schematic flowchart of another method for detecting a user interface provided by an embodiment of the present invention.
如图4所示,该用户界面的检测方法可以包括以下步骤:As shown in Figure 4, the user interface detection method may include the following steps:
步骤401,展示待检测的用户界面。Step 401: Display the user interface to be tested.
需要说明的是,本步骤的执行过程可以参见上述实施例中步骤101的执行过程,原理相同,在此不再赘述。It should be noted that the execution process of this step can be referred to the execution process of step 101 in the above embodiment. The principles are the same and will not be described again here.
步骤402,从服务器获取客户端生成的检测信息。Step 402: Obtain the detection information generated by the client from the server.
在本实施例中,检测信息可以包括测试脚本和图片库,其中,图片库中包括多个界面截图,测试脚本用于指示图片库中各界面截图所对应操作行为的执行顺序。In this embodiment, the detection information may include a test script and a picture library, where the picture library includes multiple interface screenshots, and the test script is used to indicate the execution order of the operation behaviors corresponding to each interface screenshot in the picture library.
在本实施例中,检测信息可以由用户界面的客户端进行生成,并发送至服务器,以使用户界面的检测装置能够从服务器获取到客户端生成的检测信息,并执行后续操作。In this embodiment, the detection information can be generated by the client of the user interface and sent to the server, so that the detection device of the user interface can obtain the detection information generated by the client from the server and perform subsequent operations.
可选地,用户界面的检测装置可以采用线上实时采集的方式从服务器获取客户端生成的检测信息,也可以采用线下人工采集的方式从服务器获取客户端生成的检测信息,还可以通过网络传输或者物理拷贝的方式从服务器获取客户端生成的检测信息,等等,本实施例中对此不作限制。Optionally, the detection device of the user interface can obtain the detection information generated by the client from the server through online real-time collection, or can obtain the detection information generated by the client from the server through offline manual collection, or can also obtain the detection information generated by the client through the network. The detection information generated by the client is obtained from the server through transmission or physical copy, etc. This embodiment does not limit this.
步骤403,根据各界面截图,确定对应操作行为所触发的目标控件。Step 403: Determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface.
步骤404,依据执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件。Step 404: According to the execution sequence, target controls corresponding to each operation behavior are sequentially triggered in the user interface.
需要说明的是,步骤403-404的执行过程可以参见上述实施例103-104的执行过程,原理相同,在此不再赘述。It should be noted that the execution process of steps 403-404 can refer to the execution process of the above-mentioned embodiments 103-104. The principles are the same and will not be described again here.
步骤405,至少一次操作行为对应的目标控件触发存在响应异常,向服务器发送异常提示信息,或者,在用户界面展示异常提示信息。Step 405: At least one target control corresponding to the operation behavior triggers a response exception, and sends the exception prompt information to the server, or displays the exception prompt information on the user interface.
在本实施例中,在依据执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件之后,用户界面的检测装置可以依据目标控件触发是否存在响应异常来向服务器或者用户界面进行反馈。具体来说,可以在至少一次操作行为对应的目标控件触发存在响应异常的情况下,向服务器发送异常提示信息,或者,在用户界面展示异常提示信息。例如,在某一应用程序的登录界面中,操作行为包括点击用户名输入框输入用户名、点击密码输入框输入密码和点击登录按钮进行登录,操作行为对应的目标控件为用 户名输入框、密码输入框和登录按钮,从而在用户点击用户名输入框无法进行用户名输入的情况下,可以向服务器发送用户名输入框无法正常输入的异常提示信息,或者,在用户界面展示用户名输入框无法正常输入的异常提示信息。In this embodiment, after the target controls corresponding to each operation behavior are sequentially triggered in the user interface according to the execution sequence, the detection device of the user interface can provide feedback to the server or the user interface based on whether there is a response exception when the target control is triggered. Specifically, when at least one target control corresponding to the operation behavior triggers a response exception, the exception prompt information can be sent to the server, or the exception prompt information can be displayed on the user interface. For example, in the login interface of an application, the operation behaviors include clicking the user name input box to enter the user name, clicking the password input box to enter the password, and clicking the login button to log in. The target controls corresponding to the operation behaviors are the user name input box, password Input box and login button, so that when the user clicks on the user name input box and cannot enter the user name, an abnormal prompt message that the user name input box cannot be input normally can be sent to the server, or the user name input box cannot be displayed on the user interface. Exception prompt information for normal input.
可选地,向服务器发送的异常提示信息可以是文字形式,也可以是表格形式,还可以是图片形式,等等,本实施例中对此不作限制。类似地,在用户界面展示的异常提示信息可以是弹窗形式,也可以是文字提示形式,还可以是图片形式,等等,本实施例中对此也不作限制。Optionally, the exception prompt information sent to the server may be in text form, table form, picture form, etc., which is not limited in this embodiment. Similarly, the abnormal prompt information displayed on the user interface may be in the form of a pop-up window, a text prompt, a picture, etc., and this embodiment is not limited.
本实施例提供的用户界面的检测方法,通过展示待检测的用户界面,以及从服务器获取客户端生成的检测信息,实现根据各界面截图,确定对应操作行为所触发的目标控件,从而依据执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件,进而至少一次操作行为对应的目标控件触发存在响应异常,向服务器发送异常提示信息,或者,在用户界面展示异常提示信息。由此,可实现基于客户端生成的检测信息对用户界面进行自动化检测,在保证用户界面检测的准确度的同时,节省了人力成本,提高了用户界面检测的效率,可适用于多种操作系统。The user interface detection method provided in this embodiment, by displaying the user interface to be detected and obtaining the detection information generated by the client from the server, determines the target control triggered by the corresponding operation behavior based on the screenshots of each interface, thereby determining the execution sequence , sequentially trigger the target control corresponding to each operation behavior in the user interface, and then at least one target control corresponding to the operation behavior triggers a response exception, and sends the exception prompt information to the server, or displays the exception prompt information in the user interface. As a result, the user interface can be automatically detected based on the detection information generated by the client. While ensuring the accuracy of user interface detection, it saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems. .
需要说明的是,上述实施例是从对待检测用户界面进行检测的客户端的角度进行描述,为了更加清楚地说明用户界面的检测过程,本发明实施例提供了从待检测用户界面的客户端的角度进行描述的一种用户界面的检测方法可能的实现方式,图5是本发明实施例所提供的另一种用户界面的检测方法的流程示意图。It should be noted that the above embodiments are described from the perspective of the client that detects the user interface to be detected. In order to explain the detection process of the user interface more clearly, embodiments of the present invention provide a method from the perspective of the client that detects the user interface to be detected. Describing a possible implementation of a user interface detection method, FIG. 5 is a schematic flowchart of another user interface detection method provided by an embodiment of the present invention.
如图5所示,该用户界面的检测方法可以包括以下步骤:As shown in Figure 5, the user interface detection method may include the following steps:
步骤501,展示用户界面。Step 501: Display the user interface.
类似地,本实施例提供的用户界面的检测方法,可以由用户界面的检测装置执行。该用户界面的检测装置可以为电子设备,也可以被配置在电子设备中。其中,电子设备,可以是任意能够进行数据处理的静止或者移动计算设备,例如笔记本电脑、智能手机、可穿戴设备等移动计算设备,或者台式计算机等静止的计算设备,或者服务器,或者其它类型的计算设备等,本实施例中对此不作限制。Similarly, the user interface detection method provided in this embodiment can be executed by a user interface detection device. The user interface detection device may be an electronic device, or may be configured in the electronic device. Among them, the electronic device can be any stationary or mobile computing device capable of data processing, such as mobile computing devices such as laptops, smart phones, and wearable devices, or stationary computing devices such as desktop computers, or servers, or other types of Computing equipment, etc., are not limited in this embodiment.
在本实施例中,用户界面可以为任一应用程序的用户界面。用户界面的数量可以为一个或者多个,本实施例中对此不作限制。需要说明的是,在用户界面为一个的情况下,该用户界面可以为任一应用程序的任意一个用户界面;在用户界面为多个的情况下,该多个待检测的用户界面应该为同一个应用程序的任意多个用户界面。In this embodiment, the user interface may be the user interface of any application program. The number of user interfaces may be one or more, which is not limited in this embodiment. It should be noted that when there is one user interface, the user interface can be any user interface of any application; when there are multiple user interfaces, the multiple user interfaces to be detected should be the same user interface. Any number of user interfaces for an application.
在本实施例中,用户界面的检测装置可以对用户界面进行展示,以便于生成用于对用户界面进行检测的检测信息。作为一种可能的实现方式,在用户界面为一个的情况下,用户界面的检测装置可以直接展示该用户界面;在用户界面为多个的情况下,用户界面的检测装置可以同时展示多个用户界面,也可以按照设定次序每次展示设定个数个待检测的用户界面,直至所有的用户界面都展示完毕为止,本实施例中对此不作限制。其中,对于设定次序的确定和对于设定个数的具体取值本实施例并没有作出限定,可选地,可以根据人工经验进行设定,例如,可以将设定次序设定为用户对用户界面进行操作的操作顺序,将设定个数设定为3个,或者,也可以根据实际应用需求进行动态调整,本实施例中对此不作限制。In this embodiment, the user interface detection device can display the user interface so as to generate detection information for detecting the user interface. As a possible implementation method, when there is one user interface, the user interface detection device can directly display the user interface; when there are multiple user interfaces, the user interface detection device can display multiple users at the same time. The interface may also display a set number of user interfaces to be detected each time in a set order until all user interfaces are displayed. This is not limited in this embodiment. Among them, this embodiment does not limit the determination of the setting order and the specific value of the setting number. Alternatively, the setting can be made based on manual experience. For example, the setting order can be set to the user's The operation sequence of the user interface operation is to set the number of settings to 3, or it can also be dynamically adjusted according to actual application requirements, which is not limited in this embodiment.
步骤502,每当响应遥控指令执行对应的操作行为完毕,对用户界面进行截图,以得到各次操作行为对应的界面截图。Step 502: Whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface is taken to obtain an interface screenshot corresponding to each operation behavior.
在本实施例中,每当响应遥控指令执行对应的操作行为完毕,用户界面的检测装置可以通过对用户界面进行截图来得到各次操作行为对应的界面截图。也就是说,遥控指令执行对应的操作行为,每当遥控指令执行完毕对应的操作行为,用户界面的检测装置就对用户界面进行截图,从而可以得到各次操作行为对应的界面截图。例如,在某一应用程序的登录界面中,操作行为包括点击用户名输入框输入用户名、点击密码输入框输入密码和点击登录按钮进行登录,从而在遥控指令执行完毕点击用户名输入框输入用户名这一操作行为的情况下,用户界面的检测装置可以通过对用户界面进行截图来得到得到点击用户名输入框输入用户名这一操作行为对应的界面截图。类似地,在遥控指令执行完毕点击密码输入框输入密码这一操作行为的情况下,用户界面的检测装置可以通过对用户界面进行截图来得到得到点击密码输入框输入密码这一操作行为对应的界面截图。以及,在遥控指令执行完毕点击登录按钮进行登录这一操作行为的情况下,用户界面的检测装置可以通过对用户界面进行截图来得到得到点击登录按钮进行登录这一操作行为对应的界面截图。In this embodiment, whenever the corresponding operation behavior is executed in response to the remote control command, the user interface detection device can obtain the interface screenshot corresponding to each operation behavior by taking a screenshot of the user interface. That is to say, the remote control command executes the corresponding operation behavior. Whenever the remote control command completes the execution of the corresponding operation behavior, the user interface detection device takes a screenshot of the user interface, so that the interface screenshot corresponding to each operation behavior can be obtained. For example, in the login interface of an application, the operation behaviors include clicking the user name input box to enter the user name, clicking the password input box to enter the password, and clicking the login button to log in. After the remote control command is executed, click the user name input box to enter the user name. In the case of the operation behavior of clicking the user name input box, the user interface detection device can obtain the interface screenshot corresponding to the operation behavior of clicking the user name input box and inputting the user name by taking a screenshot of the user interface. Similarly, when the remote control command is executed and the operation behavior of clicking on the password input box to enter the password is completed, the user interface detection device can obtain the interface corresponding to the operation behavior of clicking on the password input box to enter the password by taking a screenshot of the user interface. screenshot. And, in the case of the operation behavior of clicking the login button to log in after the remote control command is executed, the user interface detection device can obtain the interface screenshot corresponding to the operation behavior of clicking the login button to log in by taking a screenshot of the user interface.
步骤503,根据各次操作行为的执行顺序,和各次操作行为对应的界面截图,生成用于对用户界面进行检测的检测信息。Step 503: Generate detection information for detecting the user interface based on the execution order of each operation behavior and the interface screenshot corresponding to each operation behavior.
在本实施例中,在得到各次操作行为对应的界面截图之后,由于各界面截图存在时间顺序,从而可以基于各界面截图的时间顺序确定各次操作行为的执行顺序,进而可以根据各次操作行为的执行顺序,和各次操作行为对应的界面截图,生成用于对用户界面进行检测的检测信息。其中,检测信息用于指示多次操作行为的执行顺序,以及各次操作行为对应的界面截图。可选地,检测信息可以包括测试脚本和 图片库,其中,图片库中包括多个界面截图,测试脚本用于指示图片库中各界面截图所对应操作行为的执行顺序。例如,在上述登录界面例子中,各界面截图的时间顺序可以为:首先,得到了点击用户名输入框输入用户名这一操作行为对应的界面截图,其次,得到了点击密码输入框输入密码这一操作行为对应的界面截图,最后,得到了点击登录按钮进行登录这一操作行为对应的界面截图,从而各次操作行为的执行顺序为先点击用户名输入框输入用户名,再点击密码输入框输入密码,最后点击登录按钮进行登录,由此,可生成用于对用户界面进行检测的检测信息。In this embodiment, after obtaining the interface screenshots corresponding to each operation behavior, since each interface screenshot has a time sequence, the execution order of each operation behavior can be determined based on the time sequence of each interface screenshot, and then the execution order of each operation behavior can be determined based on the time sequence of each operation behavior. The execution sequence of the behavior and the interface screenshots corresponding to each operation behavior generate detection information used to detect the user interface. Among them, the detection information is used to indicate the execution sequence of multiple operation behaviors, as well as the interface screenshots corresponding to each operation behavior. Optionally, the detection information may include a test script and a picture library, where the picture library includes multiple interface screenshots, and the test script is used to indicate the execution order of the operation behaviors corresponding to each interface screenshot in the picture library. For example, in the above example of the login interface, the chronological order of screenshots of each interface can be: first, the interface screenshot corresponding to the operation behavior of clicking the user name input box to enter the user name is obtained; secondly, the screenshot of the interface corresponding to the operation behavior of clicking the password input box to enter the password is obtained. A screenshot of the interface corresponding to the operation behavior. Finally, the screenshot of the interface corresponding to the operation behavior of clicking the login button to log in is obtained. Therefore, the execution order of each operation behavior is to first click the user name input box to enter the user name, and then click the password input box. Enter the password and finally click the login button to log in. From this, the detection information used to detect the user interface can be generated.
本发明实施例提供的用户界面的检测方法,通过展示用户界面,实现每当响应遥控指令执行对应的操作行为完毕,对用户界面进行截图,以得到各次操作行为对应的界面截图,从而根据各次操作行为的执行顺序,和各次操作行为对应的界面截图,生成用于对用户界面进行检测的检测信息。由此,可实现每当响应遥控指令执行对应的操作行为完毕,对用户界面进行截图,以得到各次操作行为对应的界面截图,从而基于各次操作行为的执行顺序和各次操作行为对应的界面截图,生成用于对用户界面进行检测的检测信息。The user interface detection method provided by the embodiment of the present invention, by displaying the user interface, realizes that whenever the corresponding operation behavior is completed in response to the remote control instruction, a screenshot of the user interface is taken to obtain the interface screenshot corresponding to each operation behavior, so as to obtain the interface screenshot corresponding to each operation behavior. The execution sequence of each operation behavior and the interface screenshot corresponding to each operation behavior are used to generate detection information for detecting the user interface. Thus, whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface can be obtained to obtain the interface screenshot corresponding to each operation behavior, so that based on the execution sequence of each operation behavior and the corresponding operation behavior Take screenshots of the interface to generate detection information for testing the user interface.
为了清楚说明上一实施例,本实施例提供了另一种用户界面的检测方法,图6为本发明实施例所提供的另一种用户界面的检测方法的流程示意图。In order to clearly explain the previous embodiment, this embodiment provides another method for detecting a user interface. FIG. 6 is a schematic flowchart of another method for detecting a user interface provided by an embodiment of the present invention.
如图6所示,该用户界面的检测方法可以包括以下步骤:As shown in Figure 6, the detection method of the user interface may include the following steps:
步骤601,展示用户界面。Step 601: Display the user interface.
步骤602,每当响应遥控指令执行对应的操作行为完毕,对用户界面进行截图,以得到各次操作行为对应的界面截图。Step 602: Whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface is taken to obtain an interface screenshot corresponding to each operation behavior.
步骤603,根据各次操作行为的执行顺序,和各次操作行为对应的界面截图,生成用于对用户界面进行检测的检测信息。Step 603: Generate detection information for detecting the user interface based on the execution order of each operation behavior and the interface screenshot corresponding to each operation behavior.
需要说明的是,步骤601-603的执行过程可以参见上一实施例501-503的执行过程,原理相同,在此不再赘述。It should be noted that the execution process of steps 601-603 can refer to the execution process of steps 501-503 in the previous embodiment. The principles are the same and will not be described again here.
步骤604,向服务器发送检测信息,以将检测信息发送至多个待检测用户界面的客户端。Step 604: Send the detection information to the server to send the detection information to multiple clients of the user interfaces to be detected.
在本实施例中,在生成用于对用户界面进行检测的检测信息之后,用户界面的检测装置可以向服务器发送检测信息,以将检测信息发送至多个待检测用户界面的客户端。可选地,用户界面的检测装置可以通过网络传输或者物理拷贝的方式向服务器发送检测信息。In this embodiment, after generating the detection information for detecting the user interface, the detection device of the user interface may send the detection information to the server to send the detection information to multiple clients of the user interface to be detected. Optionally, the detection device of the user interface can send the detection information to the server through network transmission or physical copy.
本发明实施例提供的用户界面的检测方法,在生成用于对用户界面进行检测的检测信息之后,向服务器发送检测信息,以将检测信息发送至多个待检测用户界面的客户端。由此,可实现将生成的用于对用户界面进行检测的检测信息通过服务器发送给对待检测用户界面进行检测的客户端,以进行用户界面检测。The user interface detection method provided by the embodiment of the present invention, after generating detection information for detecting the user interface, sends the detection information to the server to send the detection information to multiple clients of the user interface to be detected. Thus, the generated detection information for detecting the user interface can be sent through the server to the client that detects the user interface to be detected, so as to perform user interface detection.
与上述图1至图4实施例提供的用户界面的检测方法相对应,本发明还提供一种用户界面的检测装置,由于本发明实施例提供的用户界面的检测装置与上述图1至图4实施例提供的用户界面的检测方法相对应,因此在用户界面的检测方法的实施方式也适用于本发明实施例提供的用户界面的检测装置,在本发明实施例中不再详细描述。Corresponding to the user interface detection method provided by the above embodiments of Figures 1 to 4, the present invention also provides a user interface detection device, because the user interface detection device provided by the embodiment of the present invention is consistent with the above Figures 1 to 4 The detection method of the user interface provided by the embodiment corresponds to the detection method of the user interface. Therefore, the implementation of the detection method of the user interface is also applicable to the detection device of the user interface provided by the embodiment of the present invention, and will not be described in detail in the embodiment of the present invention.
图7为本发明实施例提供的一种用户界面的检测装置的结构示意图。FIG. 7 is a schematic structural diagram of a user interface detection device provided by an embodiment of the present invention.
如图7所示,该用户界面的检测装置包括:第一展示模块71、获取模块72、确定模块73和第一处理模块74。As shown in FIG. 7 , the user interface detection device includes: a first display module 71 , an acquisition module 72 , a determination module 73 and a first processing module 74 .
第一展示模块71,用于展示待检测的用户界面;The first display module 71 is used to display the user interface to be detected;
获取模块72,用于获取用户界面的检测信息,其中,检测信息用于指示多次操作行为的执行顺序,以及各次操作行为对应的界面截图;The acquisition module 72 is used to obtain detection information of the user interface, where the detection information is used to indicate the execution sequence of multiple operation behaviors, and the interface screenshots corresponding to each operation behavior;
确定模块73,用于根据各界面截图,确定对应操作行为所触发的目标控件;The determination module 73 is used to determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface;
第一处理模块74,用于依据执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件。The first processing module 74 is used to sequentially trigger the target controls corresponding to each operation behavior in the user interface according to the execution sequence.
进一步地,在本发明实施例的一种可能的实现方式中,获取模块72,还用于:Further, in a possible implementation of the embodiment of the present invention, the acquisition module 72 is also used to:
从服务器获取客户端生成的检测信息;Obtain the detection information generated by the client from the server;
其中,检测信息包括测试脚本和图片库;Among them, the detection information includes test scripts and image libraries;
图片库中包括多个界面截图;The image gallery includes multiple interface screenshots;
测试脚本,用于指示图片库中各界面截图所对应操作行为的执行顺序。The test script is used to instruct the execution sequence of the operation behaviors corresponding to the screenshots of each interface in the picture library.
进一步地,在本发明实施例的一种可能的实现方式中,确定模块73,还用于:Further, in a possible implementation of the embodiment of the present invention, the determination module 73 is also used to:
对初始运行的用户界面进行截图,以得到目标截图;Take a screenshot of the user interface of the initial run to obtain the target screenshot;
将目标截图,与首次操作行为对应的界面截图进行比较,以将存在显示差异的控件作为首次操作行为所触发的目标控件;Compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so as to use the control with display differences as the target control triggered by the first operation behavior;
针对除操作行为之后的任意一次后续操作行为,每当执行一次对目标控件的触发,对用户界面重新 截图,以更新目标截图;For any subsequent operation behavior except the operation behavior, every time the target control is triggered, the user interface will be re-screenshoted to update the target screenshot;
将更新的目标截图,与后续操作行为对应的界面截图进行比较,以将存在显示差异的控件作为后续操作行为所触发的目标控件。Compare the updated target screenshot with the interface screenshot corresponding to the subsequent operation behavior, so that the control with display differences is used as the target control triggered by the subsequent operation behavior.
进一步地,在本发明实施例的一种可能的实现方式中,确定模块73,还用于:Further, in a possible implementation of the embodiment of the present invention, the determination module 73 is also used to:
采用经过训练的对象识别模型,对任意的界面截图进行对象识别,以确定对应界面截图中光标聚焦的控件;Use a trained object recognition model to perform object recognition on any interface screenshot to determine the control on which the cursor is focused in the corresponding interface screenshot;
将光标聚焦的控件,确定为目标控件。Determine the control on which the cursor is focused as the target control.
进一步地,在本发明实施例的一种可能的实现方式中,对象识别模型是采用训练界面图像进行训练得到,其中,训练界面图像中标注有光标聚焦的控件。Further, in a possible implementation manner of the embodiment of the present invention, the object recognition model is trained using a training interface image, where a control with a cursor focus is marked in the training interface image.
基于上述实施例,本发明实施例还提供了一种用户界面的检测装置可能的实现方式,图8为本发明实施例提供的另一种用户界面的检测装置的结构示意图,在上一实施例的基础上,该用户界面的检测装置还包括:第二处理模块75。Based on the above embodiments, the embodiment of the present invention also provides a possible implementation of a user interface detection device. Figure 8 is a schematic structural diagram of another user interface detection device provided by the embodiment of the present invention. In the previous embodiment On the basis of , the user interface detection device also includes: a second processing module 75 .
第二处理模块75,用于至少一次操作行为对应的目标控件触发存在响应异常,向服务器发送异常提示信息,或者,在用户界面展示异常提示信息。The second processing module 75 is configured to trigger a response exception when the target control corresponding to at least one operation behavior occurs, and send exception prompt information to the server, or display the exception prompt information on the user interface.
本发明实施例提供的用户界面的检测装置,通过展示待检测的用户界面,实现获取用户界面的检测信息,其中,检测信息用于指示多次操作行为的执行顺序,以及各次操作行为对应的界面截图,从而根据各界面截图,确定对应操作行为所触发的目标控件,进而依据执行顺序,在用户界面中顺序触发各次操作行为对应的目标控件。由此,可实现基于用户界面截图对用户界面进行自动化检测,在保证用户界面检测的准确度的同时,节省了人力成本,提高了用户界面检测的效率,可适用于多种操作系统。The user interface detection device provided by the embodiment of the present invention achieves acquisition of detection information of the user interface by displaying the user interface to be detected, wherein the detection information is used to indicate the execution order of multiple operation behaviors, and the corresponding operation behaviors of each operation. Take screenshots of the interface to determine the target controls triggered by the corresponding operation behaviors based on the screenshots of each interface, and then sequentially trigger the target controls corresponding to each operation behavior in the user interface based on the execution sequence. As a result, the user interface can be automatically detected based on user interface screenshots, which not only ensures the accuracy of user interface detection, but also saves labor costs and improves the efficiency of user interface detection. It can be applied to a variety of operating systems.
与上述图5至图6实施例提供的用户界面的检测方法相对应,本发明还提供一种用户界面的检测装置,由于本发明实施例提供的用户界面的检测装置与上述图5至图6实施例提供的用户界面的检测方法相对应,因此在用户界面的检测方法的实施方式也适用于本发明实施例提供的用户界面的检测装置,在本发明实施例中不再详细描述。Corresponding to the user interface detection method provided by the above embodiments of Figures 5 to 6, the present invention also provides a user interface detection device, because the user interface detection device provided by the embodiment of the present invention is consistent with the above Figures 5 to 6 The detection method of the user interface provided by the embodiment corresponds to the detection method of the user interface. Therefore, the implementation of the detection method of the user interface is also applicable to the detection device of the user interface provided by the embodiment of the present invention, and will not be described in detail in the embodiment of the present invention.
图9为本发明实施例提供的另一种用户界面的检测装置的结构示意图。FIG. 9 is a schematic structural diagram of another user interface detection device provided by an embodiment of the present invention.
如图9所示,该用户界面的检测装置包括:第二展示模块91、第三处理模块92和生成模块93。As shown in FIG. 9 , the user interface detection device includes: a second display module 91 , a third processing module 92 and a generation module 93 .
第二展示模块91,用于展示用户界面;The second display module 91 is used to display the user interface;
第三处理模块92,用于每当响应遥控指令执行对应的操作行为完毕,对用户界面进行截图,以得到各次操作行为对应的界面截图;The third processing module 92 is configured to take a screenshot of the user interface every time the corresponding operation behavior is executed in response to the remote control command, so as to obtain the interface screenshot corresponding to each operation behavior;
生成模块93,用于根据各次操作行为的执行顺序,和各次操作行为对应的界面截图,生成用于对用户界面进行检测的检测信息。The generation module 93 is configured to generate detection information for detecting the user interface based on the execution sequence of each operation behavior and the screenshots of the interface corresponding to each operation behavior.
基于上一实施例,本发明实施例还提供了一种用户界面的检测装置可能的实现方式,图10为本发明实施例提供的另一种用户界面的检测装置的结构示意图,在上一实施例的基础上,该用户界面的检测装置还包括:发送模块94。Based on the previous embodiment, the embodiment of the present invention also provides a possible implementation of a user interface detection device. Figure 10 is a schematic structural diagram of another user interface detection device provided by the embodiment of the present invention. In the previous embodiment Based on the example, the user interface detection device also includes: a sending module 94 .
发送模块94,用于向服务器发送检测信息,以将检测信息发送至多个待检测用户界面的客户端。The sending module 94 is used to send detection information to the server, so as to send the detection information to multiple clients of the user interfaces to be detected.
本发明实施例提供的用户界面的检测装置,通过展示用户界面,实现每当响应遥控指令执行对应的操作行为完毕,对用户界面进行截图,以得到各次操作行为对应的界面截图,从而根据各次操作行为的执行顺序,和各次操作行为对应的界面截图,生成用于对用户界面进行检测的检测信息。由此,可实现每当响应遥控指令执行对应的操作行为完毕,对用户界面进行截图,以得到各次操作行为对应的界面截图,从而基于各次操作行为的执行顺序和各次操作行为对应的界面截图,生成用于对用户界面进行检测的检测信息。The user interface detection device provided by the embodiment of the present invention, by displaying the user interface, realizes that whenever the corresponding operation behavior is executed in response to the remote control instruction, a screenshot of the user interface is taken to obtain the interface screenshot corresponding to each operation behavior, so as to obtain the interface screenshot corresponding to each operation behavior. The execution sequence of each operation behavior and the interface screenshot corresponding to each operation behavior are used to generate detection information for detecting the user interface. Thus, whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface can be obtained to obtain the interface screenshot corresponding to each operation behavior, so that based on the execution sequence of each operation behavior and the corresponding operation behavior Take screenshots of the interface to generate detection information for testing the user interface.
为了实现上述实施例,本发明还提出一种电子设备,包括:至少一个处理器;以及与所述至少一个处理器通信连接的存储器;其中,所述存储器存储有可被所述至少一个处理器执行的指令,所述指令被所述至少一个处理器执行,以使所述至少一个处理器能够执行前述任一实施例提供的用户界面的检测方法。In order to implement the above embodiments, the present invention also proposes an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be used by the at least one processor. Instructions are executed by the at least one processor to enable the at least one processor to execute the user interface detection method provided by any of the foregoing embodiments.
为了实现上述实施例,本发明还提出一种存储有计算机指令的非瞬时计算机可读存储介质,其中,所述计算机指令用于使所述计算机执行前述任一实施例提供的用户界面的检测方法。In order to implement the above embodiments, the present invention also proposes a non-transient computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the user interface detection method provided in any of the foregoing embodiments. .
为了实现上述实施例,本发明还提出一种计算机程序产品,包括计算机程序,所述计算机程序在被处理器执行前述任一实施例提供的用户界面的检测方法。In order to implement the above embodiments, the present invention also proposes a computer program product, which includes a computer program. The computer program is executed by a processor for detecting the user interface provided by any of the foregoing embodiments.
图11为本发明实施例提供的一种电子设备的结构框图。图11示出的电子设备仅仅是一个示例,不应对本发明实施例的功能和使用范围带来任何限制。Figure 11 is a structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device shown in FIG. 11 is only an example and should not impose any restrictions on the functions and scope of use of the embodiments of the present invention.
如图11所示,电子设备10包括处理器11,其可以根据存储在只读存储器(ROM,Read Only Memory)12中的程序或者从存储器16加载到随机访问存储器(RAM,Random Access Memory)13中的程序而执 行各种适当的动作和处理。在RAM 13中,还存储有电子设备10操作所需的各种程序和数据。处理器11、ROM 12以及RAM 13通过总线14彼此相连。输入/输出(I/O,Input/Output)接口15也连接至总线14。As shown in Figure 11, the electronic device 10 includes a processor 11, which can be loaded into a random access memory (RAM, Random Access Memory) 13 according to a program stored in a read-only memory (ROM, Read Only Memory) 12 or from a memory 16 perform various appropriate actions and processing according to the program in it. In the RAM 13, various programs and data required for the operation of the electronic device 10 are also stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via the bus 14. An input/output (I/O, Input/Output) interface 15 is also connected to the bus 14 .
以下部件连接至I/O接口15:包括硬盘等的存储器16;以及包括诸如LAN(局域网,Local Area Network)卡、调制解调器等的网络接口卡的通信部分17,通信部分17经由诸如因特网的网络执行通信处理;驱动器18也根据需要连接至I/O接口15。The following components are connected to the I/O interface 15: a memory 16 including a hard disk, etc.; and a communication section 17 including a network interface card such as a LAN (Local Area Network) card, a modem, etc., which is executed via a network such as the Internet Communication processing; driver 18 is also connected to I/O interface 15 as needed.
特别地,根据本发明的实施例,上文参考流程图描述的过程可以被实现为计算机软件程序。例如,本发明的实施例包括承载在计算机可读介质上的计算机程序,该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中,该计算机程序可以通过通信部分17从网络上被下载和安装。在该计算机程序被处理器11执行时,执行本发明的方法中限定的上述功能。In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication part 17. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the present invention are performed.
在示例性实施例中,还提供了一种包括指令的存储介质,例如包括指令的存储器16,上述指令可由电子设备10的处理器11执行以完成上述用户界面的检测方法。可选地,存储介质可以是非临时性计算机可读存储介质,例如,所述非临时性计算机可读存储介质可以是ROM、随机存取存储器(RAM)、CD-ROM、磁带、软盘和光数据存储设备等。In an exemplary embodiment, a storage medium including instructions, such as a memory 16 including instructions, is also provided. The instructions can be executed by the processor 11 of the electronic device 10 to complete the above user interface detection method. Optionally, the storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage Equipment etc.
在本说明书的描述中,参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施例或示例中。在本说明书中,对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且,描述的具体特征、结构、材料或者特点可以在任一个或多个实施例或示例中以合适的方式结合。此外,在不相互矛盾的情况下,本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。In the description of this specification, reference to the terms "one embodiment," "some embodiments," "an example," "specific examples," or "some examples" or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the invention. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine different embodiments or examples and features of different embodiments or examples described in this specification unless they are inconsistent with each other.
此外,术语“第一”、“第二”仅用于描述目的,而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此,限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本发明的描述中,“多个”的含义是至少两个,例如两个,三个等,除非另有明确具体的限定。In addition, the terms “first” and “second” are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically limited.
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为,表示包括一个或更多个用于实现定制逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分,并且本发明的优选实施方式的范围包括另外的实现,其中可以不按所示出或讨论的顺序,包括根据所涉及的功能按基本同时的方式或按相反的顺序,来执行功能,这应被本发明的实施例所属技术领域的技术人员所理解。Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments, or portions of code that include one or more executable instructions for implementing customized logical functions or steps of the process. , and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed out of the order shown or discussed, including in a substantially simultaneous manner or in the reverse order, depending on the functionality involved, which shall It should be understood by those skilled in the art to which embodiments of the present invention belong.
在流程图中表示或在此以其他方式描述的逻辑和/或步骤,例如,可以被认为是用于实现逻辑功能的可执行指令的定序列表,可以具体实现在任何计算机可读介质中,以供指令执行系统、装置或设备(如基于计算机的系统、包括处理器的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用,或结合这些指令执行系统、装置或设备而使用。就本说明书而言,"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下:具有一个或多个布线的电连接部(电子装置),便携式计算机盘盒(磁装置),随机存取存储器(RAM),只读存储器(ROM),可擦除可编辑只读存储器(EPROM或闪速存储器),光纤装置,以及便携式光盘只读存储器(CDROM)。另外,计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质,因为可以例如通过对纸或其他介质进行光学扫描,接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序,然后将其存储在计算机存储器中。The logic and/or steps represented in the flowcharts or otherwise described herein, for example, may be considered a sequenced list of executable instructions for implementing the logical functions, and may be embodied in any computer-readable medium, For use by, or in combination with, instruction execution systems, devices or devices (such as computer-based systems, systems including processors or other systems that can fetch instructions from and execute instructions from the instruction execution system, device or device) or equipment. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wires (electronic device), portable computer disk cartridges (magnetic device), random access memory (RAM), Read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, and subsequently edited, interpreted, or otherwise suitable as necessary. process to obtain the program electronically and then store it in computer memory.
应当理解,本发明的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中,多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。如,如果用硬件来实现和在另一实施方式中一样,可用本领域公知的下列技术中的任一项或他们的组合来实现:具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路,具有合适的组合逻辑门电路的专用集成电路,可编程门阵列(PGA),现场可编程门阵列(FPGA)等。It should be understood that various parts of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if it is implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: discrete logic gate circuits with logic functions for implementing data signals; Logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成,所述的程序可以存储于一种计算机可读存储介质中,该程序在执行时,包括方法实施例的步骤之一或其组合。Those of ordinary skill in the art can understand that all or part of the steps involved in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The program can be stored in a computer-readable storage medium. When executed, one of the steps of the method embodiment or a combination thereof is included.
此外,在本发明各个实施例中的各功能单元可以集成在一个处理模块中,也可以是各个单元单独物理存在,也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现,也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时,也可以存储在一个计算机可读取存储介质中。In addition, each functional unit in various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software function modules. If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
上述提到的存储介质可以是只读存储器,磁盘或光盘等。尽管上面已经示出和描述了本发明的实施 例,可以理解的是,上述实施例是示例性的,不能理解为对本发明的限制,本领域的普通技术人员在本发明的范围内可以对上述实施例进行变化、修改、替换和变型。The storage media mentioned above can be read-only memory, magnetic disks or optical disks, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above-mentioned embodiments are illustrative and should not be construed as limitations of the present invention. Those of ordinary skill in the art can make modifications to the above-mentioned embodiments within the scope of the present invention. The embodiments are subject to changes, modifications, substitutions and variations.

Claims (19)

  1. 一种用户界面的检测方法,其特征在于,包括以下步骤:A user interface detection method, characterized by including the following steps:
    展示待检测的用户界面;Display the user interface to be tested;
    获取所述用户界面的检测信息,其中,所述检测信息用于指示多次操作行为的执行顺序,以及各次所述操作行为对应的界面截图;Obtain detection information of the user interface, wherein the detection information is used to indicate the execution order of multiple operation behaviors, and interface screenshots corresponding to each operation behavior;
    根据各所述界面截图,确定对应操作行为所触发的目标控件;According to the screenshots of each interface, determine the target control triggered by the corresponding operation behavior;
    依据所述执行顺序,在所述用户界面中顺序触发各次所述操作行为对应的目标控件According to the execution sequence, the target controls corresponding to each operation behavior are sequentially triggered in the user interface.
  2. 根据权利要求1所述的方法,其特征在于,所述获取所述用户界面的检测信息,包括:The method according to claim 1, characterized in that said obtaining the detection information of the user interface includes:
    从服务器获取客户端生成的所述检测信息;Obtain the detection information generated by the client from the server;
    其中,所述检测信息包括测试脚本和图片库;Wherein, the detection information includes test scripts and picture libraries;
    所述图片库中包括多个界面截图;The picture library includes multiple interface screenshots;
    所述测试脚本,用于指示所述图片库中各所述界面截图所对应操作行为的执行顺序。The test script is used to instruct the execution sequence of the operation behaviors corresponding to each of the interface screenshots in the picture library.
  3. 根据权利要求1或2所述的方法,其特征在于,所述根据各所述界面截图,确定对应操作行为所触发的目标控件,包括:The method according to claim 1 or 2, characterized in that determining the target control triggered by the corresponding operation behavior based on each interface screenshot includes:
    对初始运行的所述用户界面进行截图,以得到目标截图;Take a screenshot of the user interface of the initial run to obtain the target screenshot;
    将所述目标截图,与首次操作行为对应的界面截图进行比较,以将存在显示差异的控件作为所述首次操作行为所触发的目标控件;Compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so that the control with a display difference is used as the target control triggered by the first operation behavior;
    针对除首次操作行为之后的任意一次后续操作行为,每当执行一次对目标控件的触发,对所述用户界面重新截图,以更新所述目标截图;For any subsequent operation behavior after the first operation behavior, each time the target control is triggered, the user interface is re-screenshoted to update the target screenshot;
    将更新的所述目标截图,与所述后续操作行为对应的界面截图进行比较,以将存在显示差异的控件作为所述后续操作行为所触发的目标控件。The updated target screenshot is compared with the interface screenshot corresponding to the subsequent operation behavior, so that the control with a display difference is used as the target control triggered by the subsequent operation behavior.
  4. 根据权利要求1或2所述的方法,其特征在于,所述根据各所述界面截图,确定对应操作行为所触发的目标控件,包括:The method according to claim 1 or 2, characterized in that determining the target control triggered by the corresponding operation behavior based on each interface screenshot includes:
    采用经过训练的对象识别模型,对任意的所述界面截图进行对象识别,以确定对应界面截图中光标聚焦的控件;Using a trained object recognition model, perform object recognition on any of the interface screenshots to determine the control on which the cursor is focused in the corresponding interface screenshot;
    将所述光标聚焦的控件,确定为所述目标控件。The control on which the cursor is focused is determined as the target control.
  5. 根据权利要求4所述的方法,其特征在于,所述对象识别模型是采用训练界面图像进行训练得到,其中,所述训练界面图像中标注有光标聚焦的控件。The method of claim 4, wherein the object recognition model is trained using a training interface image, wherein the training interface image is marked with a cursor focus control.
  6. 根据权利要求1-5任一项所述的方法,其特征在于,所述方法还包括:The method according to any one of claims 1-5, characterized in that the method further includes:
    至少一次所述操作行为对应的目标控件触发存在响应异常,向服务器发送异常提示信息,或者,在所述用户界面展示异常提示信息。At least once, a target control corresponding to the operation behavior triggers a response exception, and the exception prompt information is sent to the server, or the exception prompt information is displayed on the user interface.
  7. 一种用户界面的检测方法,其特征在于,包括以下步骤:A user interface detection method, characterized by including the following steps:
    展示用户界面;Display user interface;
    每当响应遥控指令执行对应的操作行为完毕,对所述用户界面进行截图,以得到各次操作行为对应的界面截图;Whenever the corresponding operation behavior is executed in response to the remote control command, a screenshot of the user interface is taken to obtain an interface screenshot corresponding to each operation behavior;
    根据各次操作行为的执行顺序,和各次所述操作行为对应的界面截图,生成用于对所述用户界面进行检测的检测信息。According to the execution order of each operation behavior and the interface screenshot corresponding to each operation behavior, detection information for detecting the user interface is generated.
  8. 根据权利要求7所述的方法,其特征在于,所述方法还包括:The method of claim 7, further comprising:
    向服务器发送所述检测信息,以将所述检测信息发送至多个待检测所述用户界面的客户端。The detection information is sent to a server to send the detection information to multiple clients that are to detect the user interface.
  9. 一种用户界面的检测装置,其特征在于,包括:A user interface detection device, characterized by including:
    第一展示模块,用于展示待检测的用户界面;The first display module is used to display the user interface to be tested;
    获取模块,用于获取所述用户界面的检测信息,其中,所述检测信息用于指示多次操作行为的执行顺序,以及各次所述操作行为对应的界面截图;An acquisition module, used to obtain detection information of the user interface, wherein the detection information is used to indicate the execution sequence of multiple operation behaviors, and interface screenshots corresponding to each operation behavior;
    确定模块,用于根据各所述界面截图,确定对应操作行为所触发的目标控件;The determination module is used to determine the target control triggered by the corresponding operation behavior based on the screenshots of each interface;
    第一处理模块,用于依据所述执行顺序,在所述用户界面中顺序触发各次所述操作行为对应的目标控件。The first processing module is configured to sequentially trigger target controls corresponding to each operation behavior in the user interface according to the execution sequence.
  10. 根据权利要求9所述的装置,其特征在于,所述获取模块,还用于:The device according to claim 9, characterized in that the acquisition module is also used to:
    从服务器获取客户端生成的所述检测信息;Obtain the detection information generated by the client from the server;
    其中,所述检测信息包括测试脚本和图片库;Wherein, the detection information includes test scripts and image libraries;
    所述图片库中包括多个界面截图;The picture library includes multiple interface screenshots;
    所述测试脚本,用于指示所述图片库中各所述界面截图所对应操作行为的执行顺序。The test script is used to instruct the execution sequence of the operation behaviors corresponding to each of the interface screenshots in the picture library.
  11. 根据权利要求8或9所述的装置,其特征在于,所述确定模块,还用于:The device according to claim 8 or 9, characterized in that the determining module is also used to:
    对初始运行的所述用户界面进行截图,以得到目标截图;Take a screenshot of the user interface of the initial run to obtain the target screenshot;
    将所述目标截图,与首次操作行为对应的界面截图进行比较,以将存在显示差异的控件作为所述首次操作行为所触发的目标控件;Compare the target screenshot with the interface screenshot corresponding to the first operation behavior, so that the control with a display difference is used as the target control triggered by the first operation behavior;
    针对除首次操作行为之后的任意一次后续操作行为,每当执行一次对目标控件的触发,对所述用户界面重新截图,以更新所述目标截图;For any subsequent operation behavior after the first operation behavior, each time the target control is triggered, the user interface is re-screenshoted to update the target screenshot;
    将更新的所述目标截图,与所述后续操作行为对应的界面截图进行比较,以将存在显示差异的控件作为所述后续操作行为所触发的目标控件。The updated target screenshot is compared with the interface screenshot corresponding to the subsequent operation behavior, so that the control with a display difference is used as the target control triggered by the subsequent operation behavior.
  12. 根据权利要求8或9所述的装置,其特征在于,所述确定模块,还用于:The device according to claim 8 or 9, characterized in that the determining module is also used to:
    采用经过训练的对象识别模型,对任意的所述界面截图进行对象识别,以确定对应界面截图中光标聚焦的控件;Using a trained object recognition model, perform object recognition on any of the interface screenshots to determine the control on which the cursor is focused in the corresponding interface screenshot;
    将所述光标聚焦的控件,确定为所述目标控件。The control on which the cursor is focused is determined as the target control.
  13. 根据权利要求12所述的装置,其特征在于,所述对象识别模型是采用训练界面图像进行训练得到,其中,所述训练界面图像中标注有光标聚焦的控件。The device according to claim 12, wherein the object recognition model is trained using a training interface image, wherein a cursor focus control is marked in the training interface image.
  14. 根据权利要求8-13任一项所述的装置,其特征在于,所述装置还包括:The device according to any one of claims 8-13, characterized in that the device further includes:
    第二处理模块,用于至少一次所述操作行为对应的目标控件触发存在响应异常,向服务器发送异常提示信息,或者,在所述用户界面展示异常提示信息。The second processing module is configured to trigger a response exception at least once on the target control corresponding to the operation behavior, send exception prompt information to the server, or display the exception prompt information on the user interface.
  15. 一种用户界面的检测装置,其特征在于,包括:A user interface detection device, characterized by including:
    第二展示模块,用于展示用户界面;The second display module is used to display the user interface;
    第三处理模块,用于每当响应遥控指令执行对应的操作行为完毕,对所述用户界面进行截图,以得到各次操作行为对应的界面截图;The third processing module is used to take screenshots of the user interface every time the corresponding operation behavior is completed in response to the remote control command, so as to obtain the interface screenshot corresponding to each operation behavior;
    生成模块,用于根据各次操作行为的执行顺序,和各次所述操作行为对应的界面截图,生成用于对所述用户界面进行检测的检测信息。The generation module is configured to generate detection information for detecting the user interface based on the execution sequence of each operation behavior and the screenshots of the interface corresponding to each operation behavior.
  16. 根据权利要求15所述的装置,其特征在于,所述装置还包括:The device according to claim 15, characterized in that the device further includes:
    发送模块,用于向服务器发送所述检测信息,以将所述检测信息发送至多个待检测所述用户界面的客户端。A sending module, configured to send the detection information to a server, so as to send the detection information to multiple clients that need to detect the user interface.
  17. 一种电子设备,其特征在于,包括:An electronic device, characterized by including:
    至少一个处理器;以及at least one processor; and
    与所述至少一个处理器通信连接的存储器;其中,a memory communicatively connected to the at least one processor; wherein,
    所述存储器存储有可被所述至少一个处理器执行的指令,所述指令被所述至少一个处理器执行,以使所述至少一个处理器执行权利要求1-6中任一项所述的方法,或者,执行权利要求7-8中任一项所述的方法。The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor performs the method described in any one of claims 1-6. Method, or perform the method according to any one of claims 7-8.
  18. 一种存储有计算机指令的非瞬时计算机可读存储介质,其特征在于,所述计算机指令用于使所述计算机执行权利要求1-6中任一项所述的方法,或者,执行权利要求7-8中任一项所述的方法。A non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to cause the computer to execute the method described in any one of claims 1-6, or to execute claim 7 The method described in any one of -8.
  19. 一种计算机程序产品,其特征在于,包括计算机程序,所述计算机程序在被处理器执行时实现权利要求1-6中任一项所述的方法,或者,实现权利要求7-8中任一项所述的方法。A computer program product, characterized in that it includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6, or implements any one of claims 7-8 method described in the item.
PCT/CN2022/093164 2022-05-16 2022-05-16 User interface testing methods and apparatus WO2023220902A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2022/093164 WO2023220902A1 (en) 2022-05-16 2022-05-16 User interface testing methods and apparatus
CN202280004099.5A CN117413256A (en) 2022-05-16 2022-05-16 User interface detection method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/093164 WO2023220902A1 (en) 2022-05-16 2022-05-16 User interface testing methods and apparatus

Publications (1)

Publication Number Publication Date
WO2023220902A1 true WO2023220902A1 (en) 2023-11-23

Family

ID=88834341

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/093164 WO2023220902A1 (en) 2022-05-16 2022-05-16 User interface testing methods and apparatus

Country Status (2)

Country Link
CN (1) CN117413256A (en)
WO (1) WO2023220902A1 (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105260301A (en) * 2015-09-24 2016-01-20 努比亚技术有限公司 Visual effect comparison test device of user interfaces and realization method of visual effect comparison test device of user interfaces
CN109857652A (en) * 2019-01-16 2019-06-07 深圳壹账通智能科技有限公司 A kind of automated testing method of user interface, terminal device and medium
CN110287094A (en) * 2019-05-20 2019-09-27 平安普惠企业管理有限公司 UI automated testing method, device and computer readable storage medium
CN112241361A (en) * 2019-07-19 2021-01-19 北京搜狗科技发展有限公司 Test case generation method and device and problem scene automatic reproduction method and device
US20210081294A1 (en) * 2019-09-13 2021-03-18 TestRigor, Inc. Processing screenshots of an application user interface to detect errors
US20210081308A1 (en) * 2019-09-13 2021-03-18 TestRigor, Inc. Generating automated tests based on user interaction with an application

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105260301A (en) * 2015-09-24 2016-01-20 努比亚技术有限公司 Visual effect comparison test device of user interfaces and realization method of visual effect comparison test device of user interfaces
CN109857652A (en) * 2019-01-16 2019-06-07 深圳壹账通智能科技有限公司 A kind of automated testing method of user interface, terminal device and medium
CN110287094A (en) * 2019-05-20 2019-09-27 平安普惠企业管理有限公司 UI automated testing method, device and computer readable storage medium
CN112241361A (en) * 2019-07-19 2021-01-19 北京搜狗科技发展有限公司 Test case generation method and device and problem scene automatic reproduction method and device
US20210081294A1 (en) * 2019-09-13 2021-03-18 TestRigor, Inc. Processing screenshots of an application user interface to detect errors
US20210081308A1 (en) * 2019-09-13 2021-03-18 TestRigor, Inc. Generating automated tests based on user interaction with an application

Also Published As

Publication number Publication date
CN117413256A (en) 2024-01-16

Similar Documents

Publication Publication Date Title
CN108959068B (en) Software interface testing method, device and storage medium
US10127032B2 (en) System and method for unified firmware management
US20210081294A1 (en) Processing screenshots of an application user interface to detect errors
US20140331209A1 (en) Program Testing Service
US9679090B1 (en) Systematically exploring programs during testing
US20210081308A1 (en) Generating automated tests based on user interaction with an application
US20130019171A1 (en) Automating execution of arbitrary graphical interface applications
US11019153B2 (en) Dynamically managing data sharing
AU2016286308A1 (en) Robotic process automation
US9292423B1 (en) Monitoring applications for compatibility issues
US11709591B2 (en) Flowchart-style diagramming tool to build automated workflows
WO2018184361A1 (en) Application test method, server, terminal, and storage media
KR20140088551A (en) Server upgrades with safety checking and preview
US20120084684A1 (en) Rapid Automation Front-end Framework Library and Executable Graphic User Interface Testing System and Method
JP6283096B2 (en) Program test service
CN111522749A (en) Page testing method and device, readable storage medium and electronic equipment
US20140331205A1 (en) Program Testing Service
US20240311277A1 (en) Automated testing of mobile devices using visual analysis
US9830184B2 (en) Systems and methods for determining desktop readiness using interactive measures
CN113495843B (en) Method and apparatus for testing play-up performance of video player
CN113448730B (en) Service processing method, device, computer equipment and storage medium
CN112911283B (en) Smart television testing method and device
US11431557B1 (en) System for enterprise event analysis
WO2023220902A1 (en) User interface testing methods and apparatus
US20180219938A1 (en) Dynamically managing data sharing

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 202280004099.5

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22941946

Country of ref document: EP

Kind code of ref document: A1