CN111209215A

CN111209215A - Application program testing method and device, computer equipment and storage medium

Info

Publication number: CN111209215A
Application number: CN202010113947.9A
Authority: CN
Inventors: 李旭冬; 黄超; 王洁梅; 周大军; 张力柯
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-02-24
Filing date: 2020-02-24
Publication date: 2020-05-29
Anticipated expiration: 2040-02-24
Also published as: CN111209215B

Abstract

The application provides a method and a device for testing an application program, computer equipment and a storage medium, and belongs to the technical field of software testing. The method comprises the steps of obtaining the ith display interface of an application program, wherein i is a positive integer; identifying at least one control in the ith display interface; determining a target control in at least one control of the ith display interface; acquiring an i +1 th display interface triggered by a target control; and outputting a jump relation network for determining the test result of the application program according to the interface jump relation among the ith display interface, the target control and the (i + 1) th display interface. By the method and the device, the probability of clicking the control on the display interface can be improved, the time for searching the control is shortened, the testing equipment can traverse the controls in the application program as much as possible, and the effect of automatically testing the application program by the testing equipment can be improved.

Description

Application program testing method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of software testing technologies, and in particular, to a method and an apparatus for testing an application program, a computer device, and a storage medium.

Background

In the field of application development, in order to enable an application to achieve a desired effect, a developer develops an application and then tests the application.

Because the number of the controls in the application program is huge, in order to improve the testing efficiency, the testing equipment can automatically test the application program. For example, after an application program to be tested is opened, the test equipment may obtain a current display interface of the application program, the test equipment performs random click operation on the current display interface, and if a control on the current display interface is clicked, the application program may be switched to a next display interface. And the tester supervising the test equipment judges whether the interface switching corresponding to the control is correct or not according to the function of the clicked control, the current display interface and the next display interface switched by the control.

When the test equipment automatically tests the application program, the test equipment randomly clicks on the display interface, the probability of clicking the control on the display interface is low, the time consumption is long, and particularly under the condition that the display interface has few controls, the probability of clicking the control on the display interface is low, so that the effect of automatically testing the application program by the test equipment is poor.

Disclosure of Invention

The embodiment of the application provides a method and a device for testing an application program, computer equipment and a storage medium, and can solve the problem that in the related art, the effect of automatic testing of the application program by testing equipment is poor due to the fact that the probability of clicking a control on a display interface is low and time consumption is long. The technical scheme is as follows:

in one aspect, a method for testing an application program is provided, where the method includes:

acquiring an ith display interface of the application program, wherein i is a positive integer;

identifying at least one control in the ith display interface;

determining a target control in at least one control of the ith display interface;

acquiring an (i + 1) th display interface triggered by the target control;

and outputting a jump relation network for determining the test result of the application program according to the interface jump relation among the ith display interface, the target control and the (i + 1) th display interface.

Optionally, the inputting the ith display interface into a control identification model, and identifying at least one control in the ith display interface includes:

inputting the ith display interface into the control identification model, and identifying the control type of at least one control in the ith display interface;

the determining a target control in at least one control of the ith display interface includes:

determining the target control according to the control type of the control in at least one control of the ith display interface;

or the like, or, alternatively,

and in at least one control of the ith display interface, determining the target control according to at least one of the historical click times, the control type and the random selection mode of the control.

Optionally, the determining, in the at least one control of the ith display interface, the target control according to the control type of the control includes:

selecting a control with the control type being closed from at least one control of the ith display interface as the target control;

or the like, or, alternatively,

and selecting the control with the returned control type from at least one control of the ith display interface as the target control.

Optionally, the determining, in the at least one control of the ith display interface, the target control according to at least one of the historical click times of the control, the control type, and the random selection manner includes:

in response to that the historical click times of at least one control are equal to a target value, selecting the control for triggering display of an (i + 1) th display interface as the target control, wherein the control with the minimum historical click times and the historical click times smaller than the target value exists in the (i + 1) th display interface, and the target value is the number of testing rounds of the control of the application program;

in at least one control of the ith display interface, responding that the historical click times of at least one control are equal to the target numerical value, and selecting the control with the control type being closed as the target control if no control for triggering the display of the (i + 1) th display interface exists;

in at least one control of the ith display interface, responding that the historical click times of at least one control are equal to the target numerical value, no control for triggering display of the (i + 1) th display interface exists, no control with the control type being closed exists, and the control with the control type being returned is selected as the target control;

in at least one control of the ith display interface, responding that the historical click times of at least one control are equal to the target numerical value, no control for triggering display of the (i + 1) th display interface exists, no control with the closed control type exists, no control with the returned control type exists, and one control is randomly selected as the target control.

Optionally, the obtaining an ith display interface of the application program includes:

acquiring accumulated historical click times of the application program, wherein the accumulated historical click times are the sum of the historical click times of clicked controls of the application program;

and responding to the condition that the accumulated historical click times are not larger than the time threshold value, and acquiring the ith display interface of the application program.

In another aspect, a method for testing an application program is provided, the method including:

the first equipment sends an ith display interface of the application program to the second equipment, wherein i is a positive integer;

the second equipment acquires the ith display interface and sends the control position of a target control to the first equipment, wherein the target control is determined in at least one control of the ith display interface;

the first equipment responds to the received control position of the target control and executes click operation at the control position corresponding to the target control in the ith display interface;

the first equipment responds to the click operation triggered on the target control and sends the (i + 1) th display interface triggered by the target control to the second equipment;

the second device obtains the (i + 1) th display interface of the application program and outputs a jump relation network used for determining the test result of the application program, wherein the jump relation network is determined according to interface jump relations among the (i) th display interface, the target control and the (i + 1) th display interface.

executing click operation at a control position corresponding to a target control in the ith display interface, wherein the target control is determined in at least one control of the ith display interface;

acquiring an (i + 1) th display interface triggered by the target control;

and outputting a jump relation network used for determining the test result of the application program, wherein the jump relation network is determined according to interface jump relations among the ith display interface, the target control and the (i + 1) th display interface.

In another aspect, an apparatus for testing an application is provided, the apparatus including:

the first acquisition module is used for acquiring the ith display interface of the application program, wherein i is a positive integer;

the identification module is used for identifying at least one control in the ith display interface;

the control determining module is used for determining a target control in at least one control of the ith display interface;

the second acquisition module is used for acquiring the (i + 1) th display interface triggered by the target control;

and the output module is used for outputting a jump relation network used for determining the test result of the application program according to the interface jump relation among the ith display interface, the target control and the (i + 1) th display interface.

In another aspect, a system for testing an application is provided, the system comprising a first device and a second device, wherein:

the first device is used for sending an ith display interface of the application program to the second device, wherein i is a positive integer;

the second device is used for acquiring the ith display interface and sending a control position of a target control to the first device, wherein the target control is determined in at least one control of the ith display interface;

the first device is used for responding to the received control position of the target control and executing click operation at the control position corresponding to the target control in the ith display interface;

the first device is used for responding to the click operation triggered on the target control and sending the (i + 1) th display interface triggered by the target control to the second device;

the second device is configured to acquire an (i + 1) th display interface of the application program and output a jump relation network used for determining a test result of the application program, where the jump relation network is determined according to an interface jump relation among the (i) th display interface, the target control, and the (i + 1) th display interface.

the click execution module is used for executing click operation at a control position corresponding to a target control in the ith display interface, wherein the target control is determined in at least one control of the ith display interface;

and the output module is used for outputting a jump relation network used for determining the test result of the application program, and the jump relation network is determined according to the interface jump relation among the ith display interface, the target control and the (i + 1) th display interface.

In another aspect, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method for testing the application program.

In another aspect, a computer-readable storage medium is provided, in which at least one instruction is stored, and the instruction is loaded and executed by a processor to implement the method for testing an application program described above.

In the embodiment of the application, each time the test equipment acquires a display interface, the display interface can be input into the control identification model to obtain the control positions of each control in the display interface, and then a target control is selected from the controls to be used as the next clicked control. And then, the test equipment can output a jump relation network, wherein the jump relation network is a relation network determined according to interface jump relations among the target control, the display interface where the target control is located and the display interface triggered by the target control. Therefore, a tester of the test equipment or the supervision detection equipment can judge whether the interface jump relation corresponding to each control in the jump relation network is correct or not according to the jump relation network. Therefore, the test equipment identifies the controls in the display interface through the control identification model, and after the positions of the controls in the display interface are identified, the target control to be clicked can be determined in the controls. And then the probability of clicking the control on the display interface can be improved, the time for searching the control is shortened, the testing equipment can traverse the controls in the application program as much as possible, and the effect of automatically testing the application program by the testing equipment can be improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a scene schematic diagram of a method for testing an application according to an embodiment of the present application;

fig. 2 is a schematic flowchart of a method for testing an application according to an embodiment of the present application;

fig. 3 is a schematic flowchart of a method for testing an application according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a display interface provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of a jump relation network according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a training process of a control recognition model according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a sample display interface provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of a sample display interface provided by an embodiment of the present application;

fig. 9 is a flowchart illustrating a process of calculating a control similarity according to an embodiment of the present application;

fig. 10 is a schematic diagram of a correspondence relationship between a control image and a region image according to an embodiment of the present application;

FIG. 11 is a schematic diagram of a jump relation network according to an embodiment of the present application;

FIG. 12 is a schematic diagram illustrating updating a jump relation network according to an embodiment of the present application;

fig. 13 is a schematic flowchart of a method for testing an application according to an embodiment of the present application;

fig. 14 is a schematic flowchart of a method for testing an application according to an embodiment of the present application;

fig. 15 is a schematic structural diagram of an apparatus for testing an application according to an embodiment of the present application;

fig. 16 is a schematic structural diagram of an apparatus for testing an application according to an embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

With the research and progress of artificial intelligence technology, the artificial intelligence technology is developed and applied in a plurality of fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, unmanned aerial vehicles, robots, smart medical care, smart customer service, and the like.

The control identification model used in the test scheme of the application program provided by the embodiment of the present application relates to technologies such as artificial intelligence machine learning, and the like, and is specifically described by the following embodiments.

Machine Learning (ML) is a multi-domain cross subject, and relates to multiple subjects such as probability theory, statistics, approximation theory, convex analysis and algorithm complexity theory. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formal education learning.

The application program testing method provided by the embodiment of the application program can test whether normal interface skipping is performed when the control in the application program is clicked.

The method can be executed by the test equipment, and can also be finished by the interaction of the terminal and the test equipment. For example, it may be performed by a testing device, which may be a computer device, on which an application to be tested and a program for testing are installed. The process of completing the test by the test equipment may be as follows:

first, the testing device may obtain an ith display interface of an application program to be tested, where i is a positive integer. And then, inputting the ith display interface into the control identification model, and identifying at least one control in the ith display interface. Then, the test device may determine a target control as a next control to be clicked in the at least one control. And then, the test equipment can acquire the (i + 1) th display interface triggered by the target control by clicking the target control. Finally, the test equipment can output a jump relation network for determining the test result of the application program according to the interface jump relation among the ith display interface, the target control and the (i + 1) th display interface.

The method can also be completed by the interaction of the terminal and the testing device, as shown in fig. 1, which is a scene diagram of the interaction of the terminal and the testing device to complete the testing. The terminal may be a mobile terminal, such as a mobile phone and a tablet computer, and the testing device may be a computer device. The terminal is provided with an application program to be tested, the testing equipment is provided with a program for testing, and the terminal is in communication connection with the testing equipment. As shown in fig. 2, the process of interactively completing the test by the terminal 110 and the test device 120 may be as follows:

in step 201, the terminal may send an ith display interface of the application to be tested to the testing device, where i is a positive integer. In step 202, after the test device obtains the ith display interface, the ith display interface may be input into the control identification model, at least one control in the ith display interface is identified, and the target control is determined from the at least one control to be clicked as the next control to be clicked. In step 203, the test equipment sends the control position of the target control to the terminal. In step 204, after receiving the control position of the target control, the terminal may execute a click operation at the control position corresponding to the target control in the ith display interface. In step 205, after the terminal performs the click operation on the target control, an i +1 th display interface triggered by the target control may be sent to the test device. In step 206, after the test device receives the (i + 1) th display interface, a jump relationship network for determining a test result of the application program may be generated according to the interface jump relationship among the (i) th display interface, the target control, and the (i + 1) th display interface. Thereafter, in step 207, the test equipment may output a jump relation net.

Therefore, no matter the test is completed by the test equipment or the test is completed by the interaction of the terminal and the test equipment, the test equipment can output the jump relation network outwards. A tester supervising the test equipment can judge whether the interface jump relation among each control in the jump relation network, the display interface where the control is located and the display interface triggered by the control is correct or not according to the jump relation network.

The above is a description of a scenario of the embodiment, and a process of testing an application to be tested is described in detail below.

Fig. 3 is a schematic flow diagram of a testing device side in a method for testing an application according to an embodiment of the present disclosure, as shown in fig. 3, a processing flow of the method may include the following steps:

in step 301, the test device obtains the ith display interface of the application program.

Wherein i is a positive integer, and the application is an application to be tested, for example, the application may be a multimedia application, a game application, a social application, or the like, and the specific type of the application is not specifically limited in this embodiment.

The display Interface may also be referred to as a UI (User Interface) Interface, and may be a login Interface of the application program, a home page display Interface of the application program, a setting Interface of the application program, or the like. Fig. 4 is a schematic diagram of a display interface of a home page of a game application.

In one example, the test device may obtain the ith display interface through the terminal, for example, the terminal installed with the application to be tested may send the ith display interface to the test device. The test equipment can also acquire the ith display interface by clicking a control of the application program. For example, the test device clicks a control on the (i-1) th display interface, and the (i) th display interface triggered by the control can be acquired.

In the case that the test device acquires the ith display interface through the terminal, since the resolution of the terminal may be different from the resolution of the test device, correspondingly, the ith display interface acquired by the test device is the display interface with the adjusted resolution, for example, the test device may adjust the resolution of the acquired display interface to be adapted to the target resolution of the test device.

In step 302, the test equipment identifies at least one control in the ith display interface.

The control is a component capable of triggering interface jump in the display interface, and may be a button, an operation gesture, an action expression, and the like.

In one example, the test device may identify a control in the display interface through the trained control identification model, for example, the test device may input the ith display interface into the control identification model, and identify at least one control in the ith display interface. The control identification model can identify at least one control in the ith display interface, that is, the control identification model can mark the control in the ith display interface, and correspondingly, the control identification model can identify the control position of the control in the ith display interface.

The control position can be a coordinate position of a center point of the control in the ith display interface.

In step 303, the test equipment determines a target control in at least one control of the ith display interface.

In one example, the test device inputs the ith display interface into the control identification model, and may identify at least one control in the ith display interface, that is, may identify a control position of the at least one control. After the test equipment obtains the control position of at least one control in the ith display interface, one control can be selected from the at least one control to serve as a target control, and the target control is the next control to be clicked.

The method for selecting the target control in the at least one control in the ith display interface has various modes, for example, the target control can be selected according to the historical click times of each control in the at least one control, and also can be selected according to the space type of each control in the at least one control. The selection of the target control will be described in detail below.

In step 304, the test equipment acquires the (i + 1) th display interface triggered by the target control.

In one example, similar to obtaining the ith display interface, the test device may obtain the (i + 1) th display interface through the terminal. For example, after the test device determines the target control, the control position of the target control on the ith display interface may be sent to the terminal, and after the terminal receives the control position of the target control, a click operation may be executed at the control position corresponding to the target control on the ith display interface, and the terminal is triggered to display the (i + 1) th display interface triggered by the target control. And then, the terminal can send the (i + 1) th display interface to the test equipment, and the test equipment further obtains the (i + 1) th display interface triggered by the target control through the terminal.

The test equipment sends the control position of the target control on the ith display interface to the terminal, the resolution of the terminal is different from the resolution of the test equipment as described above, the control position can be the control position after the test equipment is converted into the resolution suitable for the terminal through the resolution of the test equipment and the resolution of the terminal, and after the terminal receives the control position of the target control, the terminal does not need to convert and can execute the click operation based on the control position of the target control. Or, the control position may be a position adapted to the resolution of the test device, and after the test device sends the control position of the target control to the terminal, the terminal performs conversion based on the conversion relationship between the resolution of the test device and the resolution of the terminal, and performs a click operation based on the converted control position of the target control after the conversion is converted into a resolution adapted to the terminal.

For another example, the test device may also trigger the (i + 1) th display interface through the target control, the test device may execute a click operation at a control position on the ith display interface corresponding to the target control, the display interface on the trigger test device may jump from the ith display interface to the (i + 1) th display interface, and then the (i + 1) th display interface triggered by the target control is acquired.

In step 305, the test device outputs a jump relation network for determining the test result of the application program according to the interface jump relation among the ith display interface, the target control and the (i + 1) th display interface.

The jump relation network is an interface network consisting of a plurality of interface jump relations, each interface jump relation consists of two display interfaces and control information triggering the two display interfaces to jump, and the control information can be control positions.

In an example, as shown in fig. 5, a vertex of the jump relation network is a display interface, and an arrow located between two display interfaces represents a jump relation between the two display interfaces, where each arrow further carries control information of a control that generates the arrow. As shown in fig. 5, an arrow between the display interface 1 and the display interface 2, which points to the display interface 2 from the display interface 1, may indicate that when a certain control on the display interface 1 is clicked, the display interface may jump from the display interface 1 to the display interface 2, an arrow between the display interface 1 and the display interface 2, which points to the display interface 1 from the display interface 2, may indicate that when a certain control on the display interface 2 is clicked, the display interface may jump from the display interface 2 to the display interface 1.

After the test equipment outputs the jump relation network, whether the interface jump corresponding to each control in the jump relation network is correct or not can be judged in various modes.

For example, one way of determining whether the interface jump relationship corresponding to the control is correct may be that the test device may output a test result, for example, the test device may further obtain a reference jump relationship network, where the reference jump relationship network is a jump relationship network generated when the application program is developed. And comparing the reference jump relation network with the output jump relation network, and outputting the test result of the application program. The test result may include control information of the control and an interface jump result, and the interface jump result may include a correct jump and an incorrect jump. If the interface jump relationship corresponding to a certain control is the same in the reference jump relationship network and the output jump relationship network, the interface jump result corresponding to the control is correct jump, and if the interface jump relationship corresponding to the control is different in the reference jump relationship network and the output jump relationship network, the interface jump result corresponding to the control is error jump.

For another example, another way for determining whether the interface jump relationship corresponding to the control is correct may be that a tester supervising the test equipment has an application description of the application program, and the application description has the interface jump relationship corresponding to each control. After the test equipment outputs the jump relation network, a tester can judge whether the control carries out correct interface jump in the test by comparing the interface jump relation of the control in the use description with the interface jump relation of the control in the jump relation network.

For another example, another way for determining whether the interface jump relationship corresponding to the control is correct may be that, during the process of testing the application program by the testing device, a tester supervising the testing device may determine whether the interface jump relationship corresponding to the control in the jump relationship network is correct according to the function and experience of the control.

In addition, in the process of testing the application program by the testing equipment, a tester supervising the testing equipment can also find out which controls, and after the clicking operation is executed, the testing equipment is broken down, so that problem troubleshooting can be performed on the basis of the controls, and the controls with problems can be found out in time.

Based on the above, each time the test equipment acquires a display interface, the display interface can be input into the control identification model, the control position of at least one control in the display interface is identified, a target control is selected from the controls, then the display interface triggered by the target control is acquired, and then the jump relation network is determined according to the target control, the display interface where the target control is located and the display interface triggered by the target control. After the jump relation network is obtained, a tester of the testing device or the supervision and detection device can determine whether the interface jump relation corresponding to the control in the jump relation network is correct.

Therefore, the test equipment identifies the controls in the display interface through the control identification model, and after the positions of the controls in the display interface are identified, the target controls to be clicked can be determined in the controls. The position of the target control in the display interface is determined, so that the target control does not need to be controlled in a random click mode, and the click operation is directly executed at the position corresponding to the target control in the display interface, so that the probability of clicking the control on the display interface can be improved, the time for searching the control is shortened, the test equipment can traverse the controls in the application program as much as possible, even all the controls of the application program can be traversed by the test equipment, and the effect of automatically testing the application program by the test equipment can be improved.

Moreover, the test equipment can output the jump relation network outwards, so that in the process of testing the application program, the test equipment does not need continuous field supervision by a tester beside the test equipment, the tester only needs to call the stored jump relation network from the test equipment, and whether the control in the application program has the correct interface jump relation can be judged by checking the jump relation network. And then can improve tester's work efficiency, reduce the human cost in the test process.

The testing device may determine whether to stop testing the application program in multiple ways, for example, the testing device may end testing the application program according to the number of the acquired display interfaces, and correspondingly, when the testing device detects that i is not greater than a preset value, the ith display interface of the application program is acquired, where the preset value is an upper limit number of the acquired display interfaces. For example, the preset value may be 500, and when the test device detects that i is less than or equal to 500, the ith display interface of the application program is acquired.

The upper limit number of the display interfaces can be determined according to specific application programs, the upper limit number can be larger for application programs with more display interfaces, the upper limit number can be smaller for application programs with less display interfaces, and testers can flexibly set the upper limit number according to the actual conditions of the application programs to be tested.

For another example, the test device may end the test on the application program according to the historical click times of the control in the application program, and correspondingly, the test device may obtain the accumulated historical click times of the application program before obtaining the ith display interface; and when the test equipment detects that the accumulated historical click times are not greater than the time threshold value, acquiring the ith display interface of the application program.

The accumulated historical click times are the sum of the historical click times of clicked controls of the application program, and the historical click times of the controls are the times of the controls being clicked in the process of testing the application program.

The frequency threshold is the upper limit frequency of executing the clicking operation on the application program, the frequency threshold is determined according to the specific application program, the frequency threshold can be larger for the application program with more display interfaces, the frequency threshold can be smaller for the application program with less display interfaces, and a tester can flexibly set the frequency threshold according to the actual situation of the application program to be tested.

For example, in the process of testing the application program, when the testing device clicks one control, the position of the control corresponding to the clicked control and the historical click times corresponding to the control can be recorded, and the accumulated historical click times can be accumulated by 1 on the original basis. And if the accumulated historical click times are not larger than the time threshold value, continuing to execute the test process on the control of the application program according to the mode.

The control identification model is obtained according to a machine learning technology in artificial intelligence, and a training process of the control identification model can be executed according to a flow shown in fig. 6:

the training of the control identification model may be performed by a test device, or may be performed by other devices, which is not limited in this embodiment, and may be performed by a test device for example.

The control identification model can select a target detection model with high detection speed and high accuracy, such as Yolov3(You Only Look one: unified, real-time target detection), Fast R-CNN (Fast regional convolutional Neural Network), and RefineNet (refinement Network).

In step 601, training samples are obtained.

The training sample can include a sample display interface and sample control information of the sample display interface.

In one example, the number of training samples may be multiple, each training sample including a sample display interface and sample control information for the sample display interface. The sample control information may be a sample control position, or a sample control position and a sample control type, where the sample control type is a category into which the control is divided according to the executed function, and may include, for example, cancel, close, determine, return, start, menu, and the like, where the other may be a control for adjusting the volume, or a control for commenting and replying, and the like.

The process of obtaining the training sample may be that, each time a technician obtains a sample display interface of a training sample, the control information may be marked on the sample display interface as the sample control information, for example, the control position and the space type in the sample display interface may be marked as the sample control information of the sample display interface.

For example, as shown in fig. 7, the sample display interface is used for setting, where a close button, a cancel button, and a confirm button in the sample display interface are all controls, a position framed by a dashed box is a control position corresponding to the control, and closing, canceling, and confirming are corresponding control types. As shown in fig. 8, the display interface is a sample display interface, where the position enclosed by the dashed box in the display interface is the position of the control corresponding to the control, and each dashed box also corresponds to the control type of the control.

In one example, in order to reduce the influence on the accuracy of the recognition result caused by the different resolutions of the sample display interfaces, a plurality of sample display interfaces with the same display content and different resolutions may be included in the training sample.

For example, the display contents in the two sample display interfaces are the same, but the resolutions have a proportional relationship, and the aspect ratio conversion of the resolutions is realized. If the sample display interface 1 and the sample display interface 2 have the same display content, but the resolution of the sample display interface 1 is a × b, and the resolution of the sample display interface 2 is m × (a × b), where a, b, and m are all positive numbers of 0, a may be the number of pixels in the length direction, b may be the number of pixels in the width direction, and m may be a scaling ratio not equal to 1.

For example, two sample display interfaces with the same display content but with resolutions of 1280 × 720 and 640 × 360 may be included in the training sample.

For another example, the display contents in the two sample display interfaces are the same, but the number of pixels in the length direction is not equal, or the number of pixels in the width direction is not equal. If the resolution of the sample display interface 3 is c × d, the resolution of the sample display interface 4 is c × e, and the resolution of the sample display interface 5 is f × d, where c, d, e, and f are positive numbers other than 0, c and f may be the number of pixels in the length direction, and d and e may be the number of pixels in the width direction.

For example, three sample display interfaces with the same display content but with resolutions of 1280 × 720, 1280 × 640, and 1280 × 606 may be included in the training sample.

Therefore, the control identification model to be trained is trained by using the sample display interfaces with different resolutions, so that the trained control identification model can identify the display interfaces with different resolutions, the inaccuracy of identification results caused by the difference of resolutions is avoided, and the identification rate of the control identification model can be improved.

In step 602, a sample display interface of the training sample is input to the control identification model to be trained, so as to obtain test control information of the training sample.

In one example, after the testing device obtains a plurality of training samples, the sample display interface of each training sample may be input to the control identification model to be trained, and the control identification model to be trained may output test control information of the training sample, where the test control information is information corresponding to the sample control information.

In step 603, a loss value corresponding to the training sample is determined according to the test control information and the sample control information.

In one example, after the test equipment obtains the test control information of each training sample, a loss function may be used to obtain a loss value between the test control information and the sample control information, and determine the loss value as the loss value of the training sample.

In step 604, the lost value is used to perform error back propagation training on the control identification model to be trained, so as to obtain the control identification model.

In one example, error back propagation is a process opposite to forward propagation, the sample display interface is input through an input layer, propagated to an output layer through various layers such as a convolutional layer and a pooling layer, and the process of outputting the test control information is forward propagation. And the process of back propagating the loss value between the test control information and the sample control information from the output layer to the input layer may be referred to as error back propagation. In the error reverse retransmission process, the loss value can be distributed to each layer between the output layer and the input layer, so that loss value signals of each layer can be obtained, and the loss value signals can be used as a basis for modifying and adjusting parameters of each layer. In this way, after multiple times of forward propagation and error backward propagation are performed on multiple training samples, when the loss value meets the condition, or when the training times reach the threshold value, the control identification model can be obtained.

In at least one control of the ith display interface, the process of determining the target control by the test equipment may be as follows:

the test equipment can determine a target control in at least one control of the ith display interface according to the historical click times of the control. Or, the test equipment may determine the target control according to the control type of the control in at least one control of the ith display interface. Or, the test equipment may determine the target control in at least one control of the ith display interface according to a random selection mode.

Or, the test device may determine the target control in at least one control of the ith display interface according to at least one of the historical click times, the control type, and the random selection mode of the control. For example, as described above, the target control is determined according to one of the historical click times, the control type and the random selection mode of the control. For example, the target control may be determined based on the control's historical number of clicks and control type. As another example, the target control may be determined based on historical click times and random selection. For another example, the target control can be determined according to the control type and the random selection mode of the control. For another example, the target control can be determined according to the historical click times of the control, the control type and the random selection mode.

The control type is identified through the control identification model, for example, the control identification model can identify not only the control position of the control but also the space type of the control, and correspondingly, the ith display interface is input into the control identification model, and the control position and the control type of at least one control in the ith display interface are identified.

As described above, control types may include cancel, close, ok, return, start, menu, and others.

The target control is determined by the test equipment in at least one control of the ith display interface according to the historical click times of the control, wherein the control with the smallest historical click times and the historical click times smaller than the target value is selected as the target control in the at least one control of the ith display interface by the test equipment, and the target value is the number of rounds of testing the control of the application program.

For example, for an application program with a large number of display interfaces, the test device may perform a round of test, where the target value may be 1, and correspondingly, the control with the smallest historical click number and the historical click number smaller than the target value, that is, the control that is not clicked in at least one control of the ith display interface, takes the control as the target control.

If a plurality of control elements which are not clicked exist in the ith display interface, one control element can be selected as a target control element from the plurality of control elements which are not clicked through a random selection mode.

For another example, for an application program with not too many display interfaces, the test device may perform multiple rounds of tests, and accordingly, the target value may be greater than 1, for example, the target value is 2, that is, in the process of performing the 2 nd round of test on the application program, the test device is a control with the smallest historical click frequency and the historical click frequency being less than the target value, that is, a control with the historical click frequency being 1.

The test equipment determines a target control according to the control type of the control in at least one control of the ith display interface, that is, the test equipment selects the control with the closed control type as the target control in at least one control of the ith display interface. Or selecting the control with the returned control type from at least one control of the ith display interface as the target control.

The process of determining the target control by the test equipment in at least one control of the ith display interface according to the historical click times, the control type and the random selection mode of the control can be as follows:

firstly, the test equipment determines whether a control with the minimum historical click frequency and the historical click frequency smaller than a target numerical value exists in at least one control of the ith display interface, and if so, the control with the minimum historical click frequency and the historical click frequency smaller than the target numerical value is used as the target control. And if the historical click times of all the controls of the ith display interface are not the target numerical values, namely the historical click times of all the controls of the ith display interface are the target numerical values, performing a second step.

For example, when the target value is 1, the test equipment first determines whether an un-clicked control exists in at least one control of the ith display interface, and if the un-clicked control exists, the un-clicked control is used as the target control. And if all the controls in at least one control of the ith display interface are clicked, namely the historical click times of all the controls of the ith display interface are all 1, performing the second step.

If a plurality of controls with the smallest historical click times and the historical click times smaller than the target numerical value exist in the ith display interface, one of the controls can be randomly selected as the target control.

And secondly, in at least one control of the ith display interface, responding to the fact that the historical click times of the at least one control are equal to the target value, selecting a control for triggering and displaying the (i + 1) th display interface as the target control, wherein the control with the minimum historical click times and the historical click times smaller than the target value exists in the (i + 1) th display interface.

The historical click times of at least one control are equal to the target value, that is, all controls in the ith display interface are clicked in the first target value round test. Correspondingly, because each control in the ith display interface is clicked, the test equipment can acquire the display interface triggered by each control, and the test equipment judges whether the control with the smallest historical clicking frequency and the historical clicking frequency smaller than the target value exists in the display interfaces. And if the display interface exists, taking a control for triggering and displaying the (i + 1) th display interface as a target control, and if the display interface does not exist, performing the third step, wherein the control with the minimum historical click frequency and the historical click frequency smaller than the target numerical value exists in the (i + 1) th display interface.

For example, the ith display interface has two controls, namely a first control and a second control, and if a control with the smallest historical click number and the historical click number smaller than the target numerical value exists in the display interface triggered by the first control, the first control in the ith display interface can be used as the target control. And if the display interface triggered and displayed by the first control and the display interface triggered and displayed by the second control do not have the control with the minimum historical clicking frequency and the historical clicking frequency smaller than the target numerical value, performing the third step.

If the display interface triggered and displayed by the first control and the display interface triggered and displayed by the second control both have a control with the minimum historical click frequency and the historical click frequency being smaller than the target numerical value, one of the first control and the second control can be randomly selected as the target control.

And thirdly, in at least one control of the ith display interface, responding that the historical click times of the at least one control are equal to a target numerical value, and selecting the control with the control type of being closed as the target control when the control for triggering the display of the (i + 1) th display interface does not exist, wherein the control with the minimum historical click times and the historical click times smaller than the target numerical value exists in the (i + 1) th display interface. And if the control types in the ith display interface are not closed, performing the fourth step.

For example, taking the target numerical value as 1, the ith display interface includes a first control and a second control for example, the first control and the second control in the ith display interface are both clicked, and the controls in the display interface triggered and displayed by the first control and the display interface triggered and displayed by the second control are both clicked, then the control of which the control type is closed can be used as the target control. And if the control types of the first control and the second control are not closed, the fourth step is carried out.

And fourthly, in at least one control of the ith display interface, responding to the fact that the historical click times of the at least one control are equal to the target numerical value, selecting the control with the control type of returning as the target control, wherein the control used for triggering the display of the (i + 1) th display interface does not exist, the control with the control type of closing does not exist. And if the control types in the ith display interface are not closed or returned, performing the fifth step.

For example, if the target numerical value is 1, the ith display interface includes a first control and a second control for example, the first control and the second control in the ith display interface are both clicked, the controls in the display interface triggered and displayed by the first control and the display interface triggered and displayed by the second control are both clicked, and the control types of the first control and the second control are not closed, the control type of the control can be a returned control, and the returned control can be used as the target control. And if the control types of the first control and the second control are not closed or returned, performing the fifth step.

And fifthly, in at least one control of the ith display interface, responding to the fact that the historical click times of the at least one control are equal to a target numerical value, randomly selecting one control as the target control, wherein the historical click times of the at least one control are not equal to the target numerical value, the control for triggering the display of the (i + 1) th display interface does not exist, the control with the closed type does not exist, and the control with the returned type does not exist.

For example, the target numerical value is still 1, the ith display interface includes a first control and a second control for example, the first control and the second control in the ith display interface are both clicked, the controls in the display interface triggered and displayed by the first control and the display interface triggered and displayed by the second control are both clicked, the control types of the first control and the second control are not closed and are not returned, and then one control can be randomly selected from the first control and the second control to serve as the target control.

The above is the process of determining the target control in at least one control of the ith display interface by the test equipment.

From the above, the test device may output a jump relation network, where the generation process of the jump relation network may be as follows:

the generation of the jump relation network may also be referred to as an update process of the jump relation network.

And when i is equal to 1, namely the ith display interface acquired by the test equipment is the 1 st display interface, under the condition, the jump relation network is generated according to the interface jump relation among the 1 st display interface, the target control and the 2 nd display interface triggered and displayed by the target control, and is used as an initial jump relation network.

And when i is larger than 1, in response to the fact that the interface jump relationship among the ith display interface, the target control and the (i + 1) th display interface does not exist in the jump relationship network, updating the jump relationship network according to the interface jump relationship among the ith display interface, the target control and the (i + 1) th display interface.

That is, after the test device obtains the (i + 1) th display interface, it may determine whether the interface jump relationship is stored in the current jump relationship network according to the interface jump relationship among the (i) th display interface, the target control, and the (i + 1) th display interface, and if not, may update the current jump relationship network to obtain the updated jump relationship network. And if the interface jump relation exists in the current jump relation network, the current jump relation network is not updated.

And judging whether the interface jump relationship among the ith display interface, the target control and the (i + 1) th display interface exists in the current jump relationship network, namely judging whether the (i + 1) th display interface exists in the current jump relationship network, and judging whether interface jump exists between the ith display interface and the (i + 1) th display interface.

In one example, the test device may determine whether a display interface identical to the (i + 1) th display interface exists in the jump relationship network, if not, the (i + 1) th display interface does not exist in the jump relationship network, and if so, the (i + 1) th display interface exists in the jump relationship network.

In another example, in order to increase the determination speed, the test device may first determine whether a candidate display interface similar to the (i + 1) th display interface exists in the jump relation network, and then determine whether a display interface identical to the (i + 1) th display interface exists in the candidate display interface.

For example, if a display interface similar to the (i + 1) th display interface does not exist in the jump relationship network, the (i + 1) th display interface does not exist in the jump relationship network.

And if a display interface similar to the (i + 1) th display interface exists in the jump relation network, the (i + 1) th display interface is likely to exist in the jump relation network, and the display interface similar to the (i + 1) th display interface can be called a candidate display interface. And then, the test equipment judges whether a display interface which is the same as the (i + 1) th display interface exists in the candidate display interfaces or not, and if the display interface which is the same as the (i + 1) th display interface does not exist in the candidate display interfaces, the (i + 1) th display interface does not exist in the jump relation network. And if the display interface identical to the (i + 1) th display interface exists in the candidate display interfaces, the (i + 1) th display interface exists in the jump relation network.

Based on the principle of judging similarity and then judging identity, the test equipment can judge whether a candidate display interface similar to the (i + 1) th display interface exists in the jump relation network according to the overall similarity between the (i + 1) th display interface and each display interface in the jump relation network.

The overall similarity may also be referred to as a global similarity, that is, whether two display interfaces are similar or close to each other is determined from the overall or global aspects of the two display interfaces.

For example, in response to that the overall similarity is smaller than the first threshold, it is determined that the (i + 1) th display interface does not exist in the jump relation network, and then the interface jump relation among the (i) th display interface, the target control, and the (i + 1) th display interface does not exist in the jump relation network.

That is, if the overall similarity between each display interface in the jump relationship network and the (i + 1) th display interface is smaller than the first threshold, the (i + 1) th display interface does not exist in the jump relationship network, and accordingly, the interface jump relationship among the (i) th display interface, the target control, and the (i + 1) th display interface does not exist in the jump relationship network.

For example, one way of calculating the overall similarity may be that the test device may calculate the overall similarity according to the display content of each display interface in the (i + 1) th display interface and the jump relationship network, and if all the overall similarities are smaller than the first threshold, the (i + 1) th display interface does not exist in the jump relationship network.

For another example, another way to calculate the overall similarity may also be that the test device may calculate the overall similarity according to the i +1 th display interface and the control layout of each display interface in the jump relation network, and if all the overall similarities are smaller than the first threshold, the i +1 th display interface does not exist in the jump relation network.

The control layout is also the distribution condition of the controls in the display interface, and the mode of calculating the overall similarity according to the control layout may be as follows:

firstly, the testing equipment can adjust the pixel values of the (i + 1) th display interface and each display interface in the jump relation network, the pixel values of the pixel points at the control position in the (i + 1) th display interface and each display interface are adjusted to be 255, the pixel values of the pixel points at positions except the control position are adjusted to be 0, and the black and white images corresponding to the (i + 1) th display interface and each display interface are obtained.

And then, the test equipment calculates the similarity of the (i + 1) th display interface and each display interface after the pixel value adjustment to obtain the overall similarity according to the control layout.

For another example, another way to calculate the overall similarity may be to calculate the overall similarity according to the display content and the control layout of each display interface in the (i + 1) th display interface and the jump relationship network.

As an example, first, the test device may calculate a first overall similarity between the (i + 1) th display interface and a first display interface according to display contents of the (i + 1) th display interface and the first display interface, where the first display interface is any one display interface in the jump relation network. Then, the test equipment can calculate a second overall similarity between the (i + 1) th display interface and the first display interface according to the control layout of the (i + 1) th display interface and the first display interface. Finally, the test device may use an average value of the first overall similarity and the second overall similarity as the overall similarity between the (i + 1) th display interface and the first display interface. Or the test equipment displays the control layout according to the weight coefficient a of the display content, the weight coefficient b of the control layout and the first overall similarity S₁And a second global similarity S₂By the formula a × S₁+b×S₂And determining the overall similarity between the (i + 1) th display interface and the first display interface.

The method for calculating the overall similarity between the (i + 1) th display interface and each display interface in the jump relation network is not limited in this embodiment, and the above is only an example and is not specifically limited.

Wherein, the overall similarity can be calculated by using an image similarity calculation method of correlation coefficient matching; the overall similarity can also be calculated by using a cosine matching image similarity calculation method; the overall similarity can be calculated by using an image similarity calculation method of standard deviation matching; an image similarity calculation method using gradient matching, calculation of overall similarity, and the like may also be used.

After the test equipment calculates the overall similarity between the (i + 1) th display interface and each display interface in the jump relation network, if the overall similarity which is not smaller than the first threshold exists, at least one display interface of which the overall similarity is not smaller than the first threshold can be determined as a candidate display interface which is similar to the (i + 1) th display interface, and then whether the display interface which is the same as the (i + 1) th display interface exists in the candidate display interfaces is judged.

In one example, the test device may determine whether the display interface identical to the (i + 1) th display interface exists in the candidate display interfaces through the control similarity. For example, the test equipment calculates the control similarity between the (i + 1) th display interface and the candidate display interface according to the control image of each control in the (i + 1) th display interface and the control image of each control in the candidate display interface; and responding to the control similarity smaller than a second threshold value, determining that the (i + 1) th display interface does not exist in the jump relation network, and then not allowing the interface jump relation among the (i) th display interface, the target control and the (i + 1) th display interface to exist in the jump relation network.

The control image is also an image corresponding to the control position where the control is located in the display interface.

In one example, the control similarity may be a similarity calculated from the control image, and may include a first similarity calculated from the control image on the (i + 1) th display interface and the region image on the candidate display interface, and a second similarity calculated from the control image on the candidate display interface and the region image on the (i + 1) th display interface. Accordingly, the control similarity is smaller than the second threshold, that is, the first similarity is smaller than the second threshold, and the second similarity is smaller than the second threshold. The calculation process of the first similarity and the second similarity may be performed according to the flow shown in fig. 9:

any one of the candidate display interfaces may be marked as a target candidate display interface for convenience of description.

In step 901, the test device determines, according to the control position of each control in the (i + 1) th display interface, a region image at a position corresponding to each control position in the (i + 1) th display interface in the target candidate display interface.

In an example, as shown in fig. 10, a first control and a second control may be provided in the (i + 1) th display interface for example, and after the testing device determines a first control image corresponding to the first control and a second control image corresponding to the second control, a first region image corresponding to the first control image may be determined in the target candidate display interface according to the size of the first control image and the position of the center point of the first control image on the (i + 1) th display interface. The test device may also determine a second region image in the target candidate display interface that corresponds to the second control image.

For example, if the size of the (i + 1) th display interface is the same as that of the target candidate display interface, the control position of the center point of the first control image in the (i + 1) th display interface is p × q, where p is the distance value from the center point of the first control image to the long side of the (i + 1) th display interface, and q is the distance value from the center point of the first control image to the short side of the (i + 1) th display interface. Then, the test device may determine a first region image having the same size as the first control image as a region image corresponding to the first control, with the position determined by p × q as a center point in the target candidate display interface. And the test equipment determines the area image corresponding to the second control in the target candidate display interface in the same way.

In step 902, the test device calculates a first similarity according to each control image of the (i + 1) th display interface and the corresponding region image of the target candidate display interface.

In one example, after obtaining a control image corresponding to a first control and a control image corresponding to a second control in the (i + 1) th display interface and a region image corresponding to the first control in the target candidate display interface, the test device may calculate a first similarity according to the control image and the region image corresponding to the first control and calculate a first similarity according to the control image and the region image corresponding to the second control.

The number of the first similarities is related to the number of the controls in the (i + 1) th display interface, for example, if one control is provided in the (i + 1) th display interface, the test device may determine one first similarity, and if two controls are provided in the (i + 1) th display interface, the test device may determine two first similarities.

And the position corresponding to the position of the control of the (i + 1) th display interface in the target candidate display interface may or may not have the control. And if the first similarity is not less than the second threshold, the control is provided, and the control is the same as or close to the control of the (i + 1) th display interface. And if the first similarity is smaller than a second threshold, the position, corresponding to the position of the control of the (i + 1) th display interface, in the target candidate display interface does not have the control, or the position has a control different from the control of the (i + 1) th display interface.

In step 903, the test equipment determines, according to the control position of each control in the target candidate display interface, a region image at a position corresponding to each control position in the target candidate display interface in the (i + 1) th display interface.

In the (i + 1) th display interface, the process of determining the region image at the position corresponding to each control position in the target candidate display interface is similar to the process of determining the region image at the position corresponding to each control position in the (i + 1) th display interface in the target candidate display interface in the step 901 described above, which may be referred to above, and thus is not described any more.

In step 904, the test device calculates a second similarity according to each control image in the target candidate display interface and the corresponding region image of the (i + 1) th display interface.

The process of calculating the second similarity is similar to that of calculating the first similarity in step 902, and reference may be made to the above description, so that details are not repeated.

The number of the second similarities is related to the number of the controls in the target candidate display interface, for example, if one control is provided in the target candidate display interface, the test device may determine one second similarity, and if two controls are provided in the target candidate display interface, the test device may determine two second similarities.

After the test equipment calculates the first similarity and the second similarity in the above manner, it may be determined whether a display interface identical to the (i + 1) th display interface exists in the candidate display interfaces according to the first similarity and the second similarity.

For example, if all the first similarities between the (i + 1) th display interface and the target candidate display interface are not less than the second threshold and all the second similarities are not less than the second threshold, the (i + 1) th display interface and the target candidate display interface belong to the same display interface, and the display interface which is the same as the (i + 1) th display interface exists in the candidate display interfaces.

And if at least one first similarity between the (i + 1) th display interface and the target candidate display interface is smaller than a second threshold value, or at least one second similarity is smaller than a second threshold value, the (i + 1) th display interface and the target candidate display interface do not belong to the same display interface, and the display interface which is the same as the (i + 1) th display interface does not exist in the candidate display interfaces.

The calculation method of the first similarity and the second similarity is similar to the calculation method of the overall similarity. For example, the first similarity and the second similarity may be calculated using an image similarity calculation method of correlation coefficient matching; the first similarity and the second similarity can also be calculated by using an image similarity calculation method of cosine matching; the first similarity and the second similarity can be calculated by using an image similarity calculation method of standard deviation matching; the first similarity and the like and the second similarity may also be calculated using an image similarity calculation method of gradient matching.

The above is a process of judging whether an i +1 th display interface exists in a current jump relation network, if the i +1 th display interface does not exist in the jump relation network, the interface jump relation among the target control, the i th display interface and the i +1 th display interface is inevitably also not existed in the jump relation network, the test equipment updates the current jump relation network according to the interface jump relation among the target control, the i th display interface and the i +1 th display interface, that is, the i th display interface is added to the current jump relation network, and an arrow pointing to the i +1 th display interface is added at a position of the control corresponding to the target control on the i th display interface.

For another example, another way of determining whether the (i + 1) th display interface exists in the jump relation network through the overall similarity and the control similarity, and then determining whether the jump relation network is updated may also be, where the way is to determine the similarity according to the overall similarity and then determine the identity according to the control similarity, and the specific process may be as follows:

because a large number of display interfaces are arranged in the jump relation network, the test equipment can group the display interfaces according to the overall similarity and number each group.

First, the test device may calculate a first overall similarity between the (i + 1) th display interface and the display interface in each group according to the display content, and then calculate a first overall average similarity of the group based on each first overall similarity in each group.

Then, the test equipment calculates a second overall similarity between the (i + 1) th display interface and the display interfaces in each group according to the control layout, and calculates a second overall average similarity of the group based on each second overall similarity in each group.

And then, the test equipment calculates the overall similarity of each group according to the sum of the first overall average similarity and the second overall average similarity of each group.

In this way, after the test equipment obtains the overall similarity of each group, the overall similarity of each group is compared with the relationship between the overall similarity of each group and the first threshold value, if the overall similarity of all groups is smaller than the first threshold value, the (i + 1) th display interface does not exist in each group, the (i + 1) th display interface does not exist in the jump relationship network, and when the test equipment adds the (i + 1) th display interface to the jump relationship network, a group number is allocated to the (i + 1) th display interface.

And if the overall similarity of one or more groups is not less than the first threshold, selecting the group corresponding to the maximum overall similarity from the groups, for example, the group may be recorded as a target display interface group, and the target display interface group may also be regarded as a group composed of the at least one candidate display interface.

After the test equipment selects a group with the maximum overall similarity which is not less than the first threshold, that is, after the target display interface group is selected, the control similarity between the (i + 1) th display interface and each display interface in the target display interface group can be calculated according to the control image.

The calculation process of the control similarity is the same as that described above, and reference may be made to the above contents, which is not repeated here.

After the display interfaces in the jump relation network are grouped, as shown in fig. 11, the name of each display interface in the jump relation network may be display interface (m, n), where m is the number of the group, the value of m is an integer, and sequential values may be performed from 0, n is the number of the display interface in the group, and the value of n is also an integer, and sequential values may be performed from 0. In the jump relation network, each display interface uniquely corresponds to one name, and one name uniquely corresponds to one display interface.

For example, the number of the target display interface group in the jump relation network is 1, and if the target display interface group has a display interface that is the same as the (i + 1) th display interface, for example, the display interface (1, 3) is the same as the (i + 1) th display interface, the (i + 1) th display interface may be denoted as the display interface (1, 3).

If the display interface group does not have the same display interface as the (i + 1) th display interface, the test device may name the (i + 1) th display interface as the display interface (1, n) in the jump relation network when adding the (i + 1) th display interface to the jump relation network, and if the target display interface group already has three display interfaces, the (i + 1) th display interface may be named as the display interface (1, 4) in the jump relation network.

Based on the above, the test device may add each acquired display interface to the jump relation network in the above manner, and add the interface jump relation among the display interface, the control information triggering the control of the display interface, and the display interface where the control is located to the jump relation network.

For example, the test device may determine whether the (i + 1) th display interface exists in the jump relation network and whether the jump relation network is updated according to the flow shown in fig. 12.

In step 1201, the overall similarity between the (i + 1) th display interface and the display interfaces in the respective groups is calculated, and in step 1202, it is determined whether the group number of the (i + 1) th display interface in the jump relation network is empty, that is, it is determined whether a group with the maximum overall similarity and larger than a first threshold exists, if the group does not exist, the group number of the (i + 1) th display interface in the jump relation network is empty, step 1203 is executed, and a new group number is allocated to the (i + 1) th display interface. If the group number exists, step 1204 is executed to obtain the group number of the (i + 1) th display interface, where the group number is the group number of the group with the largest overall similarity and greater than the first threshold.

Then, step 1205 is executed, in the group where the (i + 1) th display interface is located, it is determined whether the number of the (i + 1) th display interface in the group is empty, that is, it is determined whether a display interface identical to the (i + 1) th display interface exists in the group where the (i + 1) th display interface is located, if not, the number of the (i + 1) th display interface in the group is empty, step 1206 is executed, a new number is assigned to the (i + 1) th display interface, and if a display interface identical to the (i + 1) th display interface exists in the group where the (i + 1) th display interface is located, step 1207 is executed, the number of the (i + 1) th display interface in the group where the (i + 1) th display interface is located is obtained, then step 1208 is executed, an interface jump relationship is added between the (i + 1) th display interface and the interface jump relationship is true, and then, executing step 1209 to output the updated jump relation network.

To sum up, the process of testing the application program by the testing device may be executed according to the flow shown in fig. 13:

in step 1301, a display interface is obtained. In step 1302, the acquired display interface is input into a control recognition model. In step 1303, it is determined whether a control exists in the display interface according to the result output by the control recognition model, and if no control exists, the process goes to step 1301. If the control exists, then step 1304 is performed, and the jump relation network is updated according to the display interface, the control triggering the display interface, and the interface jump relation between the display interfaces where the control is located. The testing device may further perform step 1305, selecting a next target control to be clicked from the controls in the display interface. The testing device may first perform step 1304 and then perform step 1305, may first perform step 1305 and then perform step 1304, or may perform step 1304 and step 1305 in parallel. After the step 1304 and the step 1305, the process returns to the step 1301, the above process is executed in a circulating mode until all the controls in the application program are clicked, or until the accumulated historical click times of the controls of the application program reach a time threshold value, and a jump relation network is output.

After the jump relation network is obtained in the above manner, a tester of the testing device or the supervision and detection device can determine whether the interface jump relation corresponding to the control in the jump relation network is correct.

Based on the above, referring to fig. 14, the test device obtains the ith display interface, inputs the ith display interface into the control identification model, and obtains control information of each control of the ith display interface through control identification, that is, can obtain the control position and the control type of each control. Then, according to the above-mentioned manner of selecting the target control, one control may be selected from the at least one control as the target control. Then, based on the control position of the target control, a click operation is executed on the ith display interface at the position corresponding to the target control, so that the display interface can jump to the (i + 1) th display interface triggered by the target control from the current ith display interface. And then, the test equipment can obtain the (i + 1) th display interface, and further can update the current jump relation network according to the interface jump relation among the target control, the (i) th display interface and the (i + 1) th display interface to obtain the updated jump relation network.

In this way, the test device executes the flow shown in fig. 14 for each acquired display interface, and finally, the test device may output the jump relation network.

In the embodiment of the application, each time the test equipment acquires a display interface, the display interface can be input into the control identification model to obtain the control positions of each control in the display interface, and then a target control is selected from the controls to be used as the next clicked control. And then, the test equipment can output a jump relation network, wherein the jump relation network is a relation network determined according to interface jump relations among the target control, the display interface where the target control is located and the display interface triggered by the target control. Therefore, a tester of the test equipment or the supervision detection equipment can judge whether the interface jump relation corresponding to each control in the jump relation network is correct or not according to the jump relation network.

Therefore, the test equipment identifies the controls in the display interface through the control identification model, and after the positions of the controls in the display interface are identified, the target controls to be clicked can be determined in the controls. The clicking operation is directly executed at the position corresponding to the target control in the display interface, the target control does not need to be controlled in a random clicking mode, the probability of clicking the control on the display interface can be improved, the time for searching the control is shortened, the testing equipment can traverse the controls in the application program as much as possible, even the testing equipment can traverse all the controls of the application program, and the effect of the testing equipment on automatically testing the application program can be improved.

Based on the same technical concept, the embodiment of the present application further provides a method for testing an application, where the method is executed by a first device and a second device, where the first device may be the terminal described above, and the second device may be the testing device described above.

the second equipment acquires the ith display interface and sends a control position of a target control to the first equipment, wherein the target control is determined in at least one control of the ith display interface, and the control position is identified by a control identification model;

The method for testing the application program provided in the above embodiment belongs to the same concept as the above embodiment, and specific implementation processes thereof are described in the method embodiment and are not described herein again.

Based on the same technical concept, the embodiment of the application also provides a method for testing the application program, and the method is executed by the testing device or the terminal.

Acquiring an ith display interface of the application program, wherein i is a positive integer; executing click operation at a control position corresponding to a target control in the ith display interface, wherein the target control is determined in at least one control in the ith display interface, and the control position is identified by a control identification model; acquiring an (i + 1) th display interface triggered by the target control; and outputting a jump relation network used for determining the test result of the application program, wherein the jump relation network is determined according to interface jump relations among the ith display interface, the target control and the (i + 1) th display interface.

Based on the same technical concept, an embodiment of the present application further provides an apparatus for testing an application, where the apparatus may be the above-mentioned testing device, as shown in fig. 15, and the apparatus includes:

a first obtaining module 1501, configured to obtain an ith display interface of the application program, where i is a positive integer;

an identifying module 1502, configured to identify at least one control in the ith display interface;

a control determining module 1503, configured to determine a target control in at least one control of the ith display interface;

a second obtaining module 1504, configured to obtain an i +1 th display interface triggered by the target control;

an output module 1505, configured to output a jump relationship network for determining a test result of the application program according to an interface jump relationship among the ith display interface, the target control, and the (i + 1) th display interface.

Optionally, the identifying module 1502 is specifically configured to input the ith display interface into a control identifying model, and identify at least one control in the ith display interface.

Optionally, the control identification model is obtained by training according to the following steps:

obtaining a training sample, wherein the training sample comprises a sample display interface and sample control information of the sample display interface;

inputting a sample display interface of the training sample into a control identification model to be trained to obtain test control information of the training sample;

determining a loss value corresponding to the training sample according to the test control information and the sample control information;

and carrying out error back propagation training on the control identification model to be trained by using the loss value to obtain the control identification model.

Optionally, the training sample includes a plurality of sample display interfaces with the same display content and different resolutions.

Optionally, the control determining module 1503 is specifically configured to:

determining the target control in at least one control of the ith display interface according to the historical click times of the control;

or the like, or, alternatively,

and determining the target control in at least one control of the ith display interface according to a random selection mode.

Optionally, the control determining module 1503 is specifically configured to:

selecting a control with the smallest historical click frequency and the historical click frequency smaller than a target numerical value from at least one control of the ith display interface as the target control;

and the target numerical value is the number of rounds of testing the control of the application program.

Optionally, the control determining module 1503 is specifically configured to:

selecting a control for triggering and displaying the (i + 1) th display interface from at least one control of the ith display interface as the target control;

and the (i + 1) th display interface has a control with the minimum historical click frequency and the historical click frequency smaller than a target numerical value, and the target numerical value is the number of rounds of testing the control of the application program.

Optionally, the identifying module 1502 is specifically configured to:

the control determining module 1503 is specifically configured to:

or the like, or, alternatively,

Optionally, the control determining module 1503 is specifically configured to:

or the like, or, alternatively,

Optionally, the control determining module 1503 is specifically configured to:

Optionally, the first obtaining module 1501 is specifically configured to:

Optionally, the output module 1505 is specifically configured to:

when i is equal to 1, generating an initial jump relation network according to interface jump relations among the 1 st display interface, the target control and the 2 nd display interface;

and when i is larger than 1, in response to that the interface jump relationship among the ith display interface, the target control and the (i + 1) th display interface does not exist in the jump relationship network, updating the jump relationship network according to the interface jump relationship among the ith display interface, the target control and the (i + 1) th display interface.

Optionally, the output module 1505 is specifically configured to:

calculating the overall similarity of the (i + 1) th display interface and each display interface in the jump relation network;

in response to the overall similarity being smaller than a first threshold, determining that the (i + 1) th display interface does not exist in the jump relation network, and interface jump relations among the (i) th display interface, the target control and the (i + 1) th display interface do not exist in the jump relation network;

in response to at least one display interface with the overall similarity not smaller than the first threshold, determining at least one display interface with the overall similarity not smaller than the first threshold as a candidate display interface similar to the (i + 1) th display interface; calculating the control similarity of the (i + 1) th display interface and the candidate display interface according to the control image of each control in the (i + 1) th display interface and the control image of each control in the candidate display interface; and in response to the control similarity being smaller than a second threshold value, determining that the (i + 1) th display interface does not exist in the jump relation network, and the interface jump relation among the (i) th display interface, the target control and the (i + 1) th display interface does not exist in the jump relation network.

Optionally, the output module 1505 is specifically configured to:

calculating the overall similarity according to the display contents of the (i + 1) th display interface and each display interface in the jump relation network;

or, calculating the overall similarity according to the control layout of the (i + 1) th display interface and each display interface in the jump relation network;

or, calculating the overall similarity according to the display content and control layout of the (i + 1) th display interface and each display interface in the jump relation network.

In the embodiment of the application, each time a display interface is obtained, the device can input the display interface into the control identification model to obtain the control positions of each control in the display interface, and then selects a target control from the controls to be used as a next clicked control. And then, the device can output a jump relation network, wherein the jump relation network is a relation network determined according to interface jump relations among the target control, the display interface where the target control is located and the display interface triggered by the target control. Therefore, the device or a tester supervising the device can judge whether the interface jump relation corresponding to each control in the jump relation network is correct or not according to the jump relation network.

Therefore, the device identifies the controls in the display interface through the control identification model, and can determine the target control to be clicked in the controls after the positions of the controls in the display interface are identified. The clicking operation is directly executed at the position corresponding to the target control in the display interface, the target control does not need to be controlled in a random clicking mode, the probability of clicking the control on the display interface can be improved, the time for searching the control is shortened, the device can traverse the controls in the application program as much as possible, even all the controls of the application program can be traversed by the device, and the effect of automatically testing the application program by the device can be improved.

It should be noted that: in the test device for an application program provided in the above embodiments, only the division of the functional modules is illustrated in the test of the application program, and in practical applications, the function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the test apparatus for an application program and the test method for an application program provided in the above embodiments belong to the same concept, and specific implementation processes thereof are detailed in the method embodiments and will not be described herein again.

Based on the same technical concept, an embodiment of the present application further provides a system for testing an application, where the system may include a first device and a second device, the first device may be the terminal described above, and the second device may be the testing device described above, where:

the second device is used for acquiring the ith display interface and sending a control position of a target control to the first device, wherein the target control is determined in at least one control of the ith display interface, and the control position is identified by a control identification model;

The test coefficient of the application program provided in the above embodiments and the test method embodiment of the application program belong to the same concept, and the specific implementation process thereof is described in detail in the method embodiment and will not be described herein again.

Based on the same technical concept, an embodiment of the present application further provides an apparatus for testing an application, where the apparatus may be the above-mentioned test device or terminal, and the apparatus includes:

the click execution module is used for executing click operation at a control position corresponding to a target control in the ith display interface, the target control is determined in at least one control in the ith display interface, and the control position is identified by a control identification model;

Fig. 16 is a schematic structural diagram of a computer device, which may be the test device described above, where the computer device 1600 may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 1601 and one or more memories 1602, where the memory 1602 stores at least one instruction, and the at least one instruction is loaded and executed by the processors 1601 to implement the steps of the test method of the application program described above.

The embodiment of the present application also provides a computer-readable storage medium, where at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the method for testing an application program as described above.

The above description is only exemplary of the present application and should not be taken as limiting, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for testing an application, the method comprising:

identifying at least one control in the ith display interface;

acquiring an (i + 1) th display interface triggered by the target control;

2. The method of claim 1, wherein the identifying at least one control in the ith display interface comprises:

and inputting the ith display interface into a control identification model, and identifying at least one control in the ith display interface.

3. The method of claim 2, wherein the control recognition model is trained according to the following steps:

4. The method of claim 3, wherein the training sample comprises a plurality of sample display interfaces with the same display content and different resolutions.

5. The method according to any one of claims 1 to 4, wherein the determining a target control in the at least one control of the ith display interface comprises:

or the like, or, alternatively,

6. The method according to claim 5, wherein the determining a target control according to the historical click times of the control in the at least one control of the ith display interface comprises:

7. The method according to claim 5, wherein the determining a target control according to the historical click times of the control in the at least one control of the ith display interface comprises:

8. The method according to any one of claims 1 to 7, wherein the outputting a jump relation network for determining a test result of the application program according to the interface jump relation among the ith display interface, the target control and the (i + 1) th display interface comprises:

9. The method of claim 8, further comprising:

10. The method according to claim 9, wherein the calculating the overall similarity between the (i + 1) th display interface and each display interface in the jump relation network comprises:

11. A method for testing an application, the method comprising:

12. An apparatus for testing an application, the apparatus comprising:

13. A system for testing an application, the system comprising a first device and a second device, wherein:

14. A computer device comprising a processor and a memory, the memory having stored therein at least one instruction that is loaded and executed by the processor to implement a method of testing an application program according to any one of claims 1 to 10.

15. A computer-readable storage medium having stored thereon at least one instruction which is loaded and executed by a processor to implement a method of testing an application program according to any one of claims 1 to 10.