WO2019136961A1

WO2019136961A1 - User interface display identification method, terminal device, storage medium and apparatus

Info

Publication number: WO2019136961A1
Application number: PCT/CN2018/097518
Authority: WO
Inventors: 彭远杰; 刘慧众
Original assignee: 深圳壹账通智能科技有限公司
Priority date: 2018-01-12
Filing date: 2018-07-27
Publication date: 2019-07-18
Also published as: CN108363599B; CN108363599A

Abstract

The application relates to the field of mobile terminal technology, providing a user interface display identification method, a terminal device, a computer readable storage medium and an apparatus. The method comprises: acquiring a target interface screenshot of a user interface to undergo detection and extracting a feature vector of the target interface screenshot; acquiring a normal interface screenshot of a pre-saved normal user interface and extracting a feature vector of the normal interface screenshot; determining, on the basis of the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot, a cosine distance between the target interface screenshot and the normal interface screenshot; determining whether the cosine distance exceeds a preset threshold; if the cosine distance exceeds the preset threshold, then acquiring the level and position of each basic control component of the user interface; detecting whether the level and position of each basic control component are normal; and if the detected level and position of each basic control component are normal, then determining that the user interface is displaying normally, thereby resolving the issue of a conventional user interface display detection method in which script maintenance efficiency is low.

Description

User interface display identification method, terminal device, storage medium and device

The present application claims priority to Chinese Patent Application No. 201810029553.8, filed on Jan. 12, 2018, entitled "User Interface Display Identification Method and Terminal Device", the entire contents of which are incorporated herein by reference. in.

Technical field

The present application belongs to the field of mobile terminal technologies, and in particular, to a user interface display and identification method, a terminal device, a computer readable storage medium, and a device.

Background technique

At present, mobile phones and other mobile terminals can be applied to a variety of operating systems, such as the Android operating system. Since different models of mobile phones have different resolutions, multiple operating systems are applied to mobile phones of different resolutions, and must be compatible with mobile phones of different resolutions. In order to make the user interface display properly. The existing detection user interface display is detected by the test script, that is, the entire test script is executed, and the test result is compared with the set fixed value after the test is completed. If the two are consistent, the test user interface is displayed normally, otherwise, the test user interface is tested. The display is not normal. However, the existing detection method has the problem of low script maintenance efficiency. If the user interface is adjusted, the entire test case needs to be rewritten, and the fixed value needs to be reset again, and the test cost is high, which is not suitable for practical application.

technical problem

The embodiment of the present invention provides a user interface display and identification method, a terminal device, a computer readable storage medium, and a device, so as to solve the problem that the script maintenance efficiency of the existing user interface display detection method is low.

Technical solution

A first aspect of the embodiment of the present application provides a user interface display and identification method, including:

Obtaining a screenshot of the target interface of the user interface to be detected, and extracting a feature vector of the screenshot of the target interface;

Obtaining a normal interface screenshot of the pre-stored normal user interface, and extracting a feature vector of the normal interface screenshot;

Determining a cosine distance between the target interface screenshot and the normal interface screenshot according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot;

Determining whether the cosine distance exceeds a preset threshold;

If it is determined that the cosine distance exceeds the preset threshold, acquiring a level and a position of each element control of the user interface to be detected;

Detecting whether the level and position of each of the element controls are abnormal;

If it is detected that the level and position of each of the element controls are normal, it is determined that the user interface to be detected is displayed normally.

A second aspect of the embodiments of the present application provides a user interface display identifying terminal device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, the processor executing the The steps of the above user interface display identification method are implemented when the computer program is described.

A third aspect of the embodiments of the present application provides a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the user interface display identification method.

A fourth aspect of the embodiments of the present application provides a user interface display identification apparatus, which may include a module for implementing the steps of the user interface display identification method.

Beneficial effect

Compared with the prior art, the embodiment of the present application has the following beneficial effects: the embodiment of the present application determines the cosine distance between the target interface screenshot and the normal interface screenshot by acquiring a target interface screenshot of the user interface to be detected, and determines that the cosine distance exceeds the pre-predetermined When the threshold is set, it is detected whether the level and position of each element control of the user interface to be detected are abnormal. When the detection is normal, it is determined that the user interface to be detected is displayed normally, and the problem that the existing user interface display detection method has low script maintenance efficiency is solved. At the same time, combining the screenshot of the user interface to be detected and the level and position of each element control of the user interface to be detected, detecting whether the user interface is abnormally displayed, improving the accuracy of the user interface display recognition, reducing the test cost, and being suitable for practical applications.

DRAWINGS

1 is a schematic flowchart of a user interface display and identification method provided by an embodiment of the present application;

2 is a schematic flowchart of a user interface display and identification method according to another embodiment of the present application;

FIG. 3 is a schematic flowchart of a user interface display and identification method according to still another embodiment of the present application; FIG.

4 is a schematic flowchart of a user interface display and identification method according to another embodiment of the present application;

FIG. 5 is a schematic flowchart of a user interface display and identification method according to another embodiment of the present application; FIG.

FIG. 6 is a schematic block diagram of a user interface display recognition program according to an embodiment of the present application; FIG.

FIG. 7 is a schematic block diagram of a user interface display and identification terminal device provided by an embodiment of the present application.

Embodiments of the invention

Referring to FIG. 1 , FIG. 1 is a schematic flowchart of a user interface display and identification method according to an embodiment of the present disclosure. In this embodiment, an angle trigger of a terminal is used as an example. Here, the terminal may be a smart phone or a tablet. Mobile terminals such as computers. As shown in FIG. 1, in this embodiment, the processing of the terminal may include the following steps:

S101: Obtain a target interface screenshot of the user interface to be detected, and extract a feature vector of the target interface screenshot.

Here, the user interface to be detected is any user interface that needs to be detected, and the user interface may also be referred to as a user interface, which is a medium for interaction and information exchange between the system and the user, and realizes an internal form of information and a human acceptable form. The conversion between.

The method for obtaining the screenshot of the target interface of the user interface to be detected may be obtained in real time or acquired in a preset time. The specific acquisition mode may be set according to actual needs. After obtaining the screenshot of the target interface of the user interface to be detected, the obtained target interface screenshot may be displayed, and a prompt for re-acquiring the screenshot may be generated. If the re-acquisition of the screenshot request is received, the target interface of the user interface to be detected is re-acquired according to the request. The screenshot can be obtained by taking a screenshot of the target interface of the user interface to be detected multiple times, and selecting a target interface screenshot that meets the requirements from the obtained plurality of target interface screenshots, for example, selecting the clearest target interface screenshot.

After extracting the feature vector of the target interface screenshot, the extracted feature vector may be saved to facilitate subsequent data query.

S102: Obtain a normal interface screenshot of the pre-stored normal user interface, and extract a feature vector of the normal interface screenshot.

Specifically, the manner of obtaining the pre-stored normal user interface includes: pre-storing the correspondence between the user interface type and the normal user interface, obtaining the type of the user interface to be detected, and determining the normal user interface corresponding to the acquired type according to the foregoing relationship.

The pre-stored normal user interface may also be determined by an interface when the user interface to be detected is normally displayed.

The manner of obtaining the screenshot of the normal interface of the pre-stored normal user interface is the same as the way of obtaining the screenshot of the target interface of the user interface to be detected, which may be acquired in real time or acquired at a preset time. Here, it is also possible to obtain a normal interface screenshot that meets the requirements from the obtained multiple normal interface screenshots by acquiring the normal interface screenshot of the pre-stored normal user interface multiple times.

S103: Determine a cosine distance between the target interface screenshot and the normal interface screenshot according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot.

Here, the cosine distance, also called cosine similarity, is a measure of the magnitude of the difference between two individuals using the cosine of the angles of the two vectors in the vector space.

A vector is a directional line segment in a multidimensional space. If the directions of the two vectors are the same, that is, the angle is close to zero, then the two vectors are similar.

S104: Determine whether the cosine distance exceeds a preset threshold.

Specifically, the preset threshold is set according to an actual requirement. If the cosine distance exceeds the preset threshold, step S105 is performed. If the cosine distance does not exceed the preset threshold, the user interface to be detected may be directly determined to be abnormal. The alarm is reminded.

The above-mentioned abnormality of the user interface to be detected is determined by the cosine distance, and the abnormality of the user interface to be detected is relatively large, because the above-mentioned screenshot of the target interface of the user interface to be detected is extracted, and the feature of the target interface screenshot is extracted. Vector, that is, the subsequent image display abnormality judgment is performed according to the feature vector of the image, and the feature vector is an overall feature of the image. This judgment mode determines that the cosine distance exceeds a preset threshold when the abnormality of the display of the user interface to be detected is relatively small. When a misjudgment is apt to occur, a subsequent detection step is performed when it is determined that the cosine distance exceeds a preset threshold.

S105: If it is determined that the cosine distance exceeds the preset threshold, acquiring a level and a position of each element control of the user interface to be detected.

Here, the level of each element control of the above-mentioned user interface to be detected may include an application in a Mac, a browser, an iOS APP, and the like.

Obtaining the hierarchy and location of each element control of the user interface to be detected may be obtained during an automated testing process or during a multi-machine compatibility testing process.

S106: Detect whether the level and position of each of the element controls are abnormal.

Specifically, the corresponding relationship between the user interface type and the abnormal control detection rule may be preset, the type of the user interface to be detected is obtained, and the abnormal control detection rule corresponding to the type of the user interface to be detected is determined according to the foregoing relationship, and the abnormal control is detected according to the abnormality control. The rule detects whether the level and position of each of the above element controls are abnormal.

S107: If it is detected that the level and position of each of the element controls are normal, it is determined that the user interface to be detected is displayed normally.

Here, if it is detected that there is a layer abnormality of one or more element controls in the hierarchy of each of the element controls, it is determined that the user interface to be detected displays an abnormality, and the first abnormality information may be generated, and may further be based on the first abnormality information. The first alarm information is generated, and the alarm is performed according to the first alarm information. If it is detected that the position of the one or more element controls in the position of each of the element controls is abnormal, determining that the user interface to be detected displays an abnormality, may generate a second abnormality information, and further may generate a second abnormal information according to the second abnormality information. Second, the alarm information is alarmed according to the second alarm information.

It can be seen from the above description that the user interface display identification method in the embodiment of the present application can solve the problem that the script maintenance efficiency of the existing user interface display detection method is low, and at the same time, the user interface to be detected can be detected through the screenshot of the user interface to be detected. The large anomaly display, combined with the level and position of each element control of the user interface to be detected, can further detect the abnormal display of the user interface, improve the accuracy of the user interface display recognition, reduce the test cost, and is suitable for practical applications.

Referring to FIG. 2, FIG. 2 is a schematic flowchart of a user interface display identification method according to another embodiment of the present application. The difference between the embodiment corresponding to FIG. 1 is that the acquiring a target interface screenshot of the user interface to be detected, and extracting the feature vector of the target interface screenshot may include S201. S202 to S207 are the same as S102 to S107 in the previous embodiment. For details, refer to the related descriptions of S102 to S107 in the foregoing embodiment, and details are not described herein. Specifically, S201 may include S2011 to S2014:

S2011: Obtain a target interface screenshot of the user interface to be detected, select a first preset number of samples from the target interface screenshot, and extract a second preset number of features for each sample.

Here, the first preset number and the second preset number may be set according to actual needs, for example, m samples may be selected from the acquired screenshots, and n features are extracted for each sample.

S2012: Form a first matrix according to the first preset number of samples and a second preset number of features extracted by each sample.

Specifically, taking m samples from the acquired screenshots, and extracting n features for each sample as an example, each sample may be taken as one row to form a first matrix A of m*n.

S2013: Obtain a second matrix according to the first matrix and the transposed matrix of the first matrix.

Here, the matrix of the first matrix can be transposed and multiplied by the original matrix to obtain a second matrix C=A(t)*A.

S2014: Calculate a feature vector of the second matrix, and determine a feature vector of the target interface screenshot according to the feature vector of the second matrix.

Specifically, the feature vector of the matrix C is obtained, and the feature vector is the feature vector of the screenshot.

Here, by selecting samples from the target interface screenshots, extracting features from each sample, and obtaining feature vectors of the target interface screenshots, the method is simple, the obtained results are accurate, and are suitable for application.

Please refer to FIG. 3. FIG. 3 is a schematic flowchart of a user interface display identification method according to another embodiment of the present application. The difference between the embodiment corresponding to FIG. 1 is that: according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot, determining that the cosine distance of the target interface screenshot and the normal interface screenshot may include S303 . S301 to S302 are the same as S101 to S102 in the previous embodiment, and S304 to S307 are the same as S104 to S107 in the previous embodiment. For details, refer to the descriptions of S101 to S102 and S104 to S107 in the above embodiment. I will not go into details. Specifically, S303 may include S3031 to S3033:

S3031: Calculate a feature vector of the target interface screenshot and a length of a feature vector of the normal interface screenshot, respectively.

S3032: Obtain an inner product of a feature vector of the target interface screenshot and a feature vector of the normal interface screenshot.

S3033: Determine a cosine distance between the target interface screenshot and the normal interface screenshot according to the length of the feature vector of the target interface screenshot, the length of the feature vector of the normal interface screenshot, and the inner product.

Specifically, the cosine theorem describes the relationship between any one of the triangles and the three sides. Given the three sides of a triangle, you can use the cosine theorem to find the angle of each corner of the triangle. Assuming that the three sides of the triangle are a, b and c, and the corresponding three angles are A, B and C, then the cosine of the angle A is:

If the two sides b and c of a triangle are treated as two vectors, the above formula is equivalent to:

The denominator represents the length of two vectors, and the numerator represents the inner product of two vectors.

As a concrete example, if the feature vector of the target interface screenshot X and the feature vector of the normal interface screenshot Y are: x1, x2, ..., x10 and y1, y2, ..., y10, respectively.

Then, the cosine distance between them can be expressed by the cosine of the angle between them:

When the vector angle cosine of the two screenshots is equal to 1, the two screenshots are completely repeated. When the cosine of the angle is close to 1, the two screenshots are similar, and the smaller the cosine of the angle is, the less relevant the two screenshots are.

Please refer to FIG. 4. FIG. 4 is a schematic flowchart of a user interface display and identification method according to another embodiment of the present disclosure. The difference between the embodiment corresponding to FIG. 1 is that if the cosine distance is determined to exceed the preset threshold, acquiring the level and position of each element control of the user interface to be detected may include S405. S401 to S404 are the same as S101 to S104 in the previous embodiment, and S406 to S407 are the same as S106 to S107 in the previous embodiment. For details, refer to the descriptions of S101 to S104 and S106 to S107 in the above embodiment. I will not go into details. Specifically, S405 may include S4051 to S4052:

S4051: If it is determined that the cosine distance exceeds the preset threshold, an operation state acquisition command is generated by using an SDK.

S4052: Acquire a level and a position of each element control of the user interface to be detected in real time according to the running state obtaining command.

Here, the running state obtaining command is generated by the SDK, and the level and position information of each element control of the user interface to be detected is obtained in real time according to the running state obtaining command (returning the rectangular coordinates of the upper left and lower right coordinate points), which is simple and convenient. Speed up subsequent processing.

Please refer to FIG. 5. FIG. 5 is a schematic flowchart of a user interface display and identification method according to another embodiment of the present application. The difference from the embodiment corresponding to FIG. 1 is that the detecting whether the level and position of each of the element controls are abnormal may include S506. S501 to S505 are the same as S101 to S105 in the previous embodiment, and S507 is the same as S107 in the previous embodiment. For details, refer to the related descriptions of S101 to S105 and S107 in the foregoing embodiment, and details are not described herein. Specifically, S506 may include S5061~S5062:

S5061: Determine, according to a level of each of the element controls, a child control in each of the element controls, and a parent control corresponding to the child control.

S5062: detecting whether a location of the target sub-control is set at a front end of the target parent control corresponding to the target sub-control, and detecting whether a range occupied by the target sub-control exceeds a range occupied by the target parent control, the target sub-control To determine any of the child controls in the child controls in each of the described element controls.

Specifically, the control one and the second control are taken as an example, the second control is a child control of the control one, the control one is the parent control of the second control, the second control is the front end of the control one, and the control one is a rectangular frame, and the position is (x1, y1) ) (x2, y2), representing the lower left and lower right corners of the rectangle, and the second control is also a rectangle with a position of (x3, y3) (x4, y4). The detection rule corresponding to the user interface to be detected may be: the child control is set at the front end of the parent control, and the range of the child control cannot exceed the range occupied by the parent control. That is, the above control 2 is at the front end of the control, and x3>x1, x4<x2, y1>y3, y2<y4 meet the rule, otherwise the display of the user interface to be detected has a problem.

Here, it is determined whether there is a problem in the display of the user interface to be detected by detecting whether the level and position of each element control are abnormal. Because the judgment is based on the position of the control, a relatively small interface display abnormality can be found, which satisfies the actual application needs.

Corresponding to the user interface display identification method described in the above embodiment, FIG. 6 is a schematic diagram showing the operating environment of the user interface display recognition program provided by the embodiment of the present application. The user interface display recognition program 600 is installed and runs in the terminal device 60. The terminal device 60 can be a mobile terminal, a palmtop computer, a server, or the like. The terminal device 60 can include, but is not limited to, a memory 601, a processor 602, and a display 603.

Please refer to FIG. 7 , which is a functional block diagram of the user interface display recognition program 600 provided by the embodiment of the present application. The user interface display identification device provided by the embodiment of the present application may also include these functional modules. In this embodiment, the user interface display recognition program 600 may be divided into one or more modules, the one or more modules being stored in the memory 601 and being processed by one or more processors ( This embodiment is performed by the processor 602) to complete the application. For example, in FIG. 8, the user interface display recognition program 600 can be divided into a first screenshot processing unit 701, a second screenshot processing unit 702, a cosine distance determining unit 703, a cosine distance determining unit 704, and an element control acquiring unit. 705. The element control detecting unit 706 and the user interface display determining unit 707.

The first screenshot processing unit 701 is configured to acquire a target interface screenshot of the user interface to be detected, and extract a feature vector of the target interface screenshot. The second screenshot processing unit 702 is configured to acquire a normal interface screenshot of the pre-stored normal user interface, and extract a feature vector of the normal interface screenshot. The cosine distance determining unit 703 is configured to determine a cosine distance between the target interface screenshot and the normal interface screenshot according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot. The cosine distance determining unit 704 is configured to determine whether the cosine distance exceeds a preset threshold. The element control obtaining unit 705 is configured to acquire a level and a position of each element control of the user interface to be detected if it is determined that the cosine distance exceeds the preset threshold. The element control detecting unit 706 is configured to detect whether the level and position of each of the element controls are abnormal. The user interface display determining unit 707 is configured to determine that the user interface to be detected is displayed normally if it is detected that the level and position of each of the element controls are normal.

Optionally, the first screenshot processing unit 701 may be divided into a sample selecting unit 7011, a first matrix forming unit 7012, a second matrix obtaining unit 7013, and a feature vector determining unit 7014.

The sample selection unit 7011 is configured to acquire a screenshot of the target interface of the user interface to be detected, select a first preset number of samples from the target interface screenshot, and extract a second preset number of features for each sample. The first matrix constituting unit 7012 is configured to form a first matrix according to the first preset number of samples and a second preset number of features extracted by each sample. The second matrix obtaining unit 7013 is configured to obtain a second matrix according to the first matrix and the transposed matrix of the first matrix. The feature vector determining unit 7014 is configured to calculate a feature vector of the second matrix, and determine a feature vector of the target interface screenshot according to the feature vector of the second matrix.

Alternatively, the cosine distance determining unit 703 may be divided into a vector length calculating unit 7031, a vector inner product obtaining unit 7032, and a cosine distance processing unit 7033.

The vector length calculation unit 7031 is configured to separately calculate a feature vector of the target interface screenshot and a length of the feature vector of the normal interface screenshot. The vector inner product obtaining unit 7032 is configured to obtain an inner product of the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot. a cosine distance processing unit 7033, configured to determine a cosine of the target interface screenshot and the normal interface screenshot according to a length of a feature vector of the target interface screenshot, a length of a feature vector of the normal interface screenshot, and the inner product distance.

Optionally, the element control obtaining unit 705 may be divided into a command generating unit 7051 and an element control processing unit 7052.

The command generating unit 7051 is configured to generate an operation state acquisition command through the SDK if it is determined that the cosine distance exceeds the preset threshold. The element control processing unit 7052 is configured to acquire, in real time, the level and position of each element control of the user interface to be detected according to the running state acquisition command.

Optionally, the element control detecting unit 706 may also be divided into a child-parent control determining unit 7061 and a child-parent control detecting unit 7062.

The child control determining unit 7061 is configured to determine, according to the level of each of the element controls, a child control in each of the element controls, and a parent control corresponding to the child control. The child control detection unit 7062 is configured to detect whether the position of the target child control is set at the front end of the target parent control corresponding to the target child control, and detect whether the target child control occupies a range beyond the target parent control Range, the target child control is any one of the child controls in each of the determined element controls.

The present application implements all or part of the processes in the above embodiments, and may also be completed by a computer program to instruct related hardware. The computer program may be stored in a computer readable storage medium, and the computer program is in the processor. When executed, the steps of the various method embodiments described above can be implemented.

Claims

A user interface display identification method, comprising:

Obtaining a screenshot of the target interface of the user interface to be detected, and extracting a feature vector of the screenshot of the target interface;

Obtaining a normal interface screenshot of the pre-stored normal user interface, and extracting a feature vector of the normal interface screenshot;

Determining a cosine distance between the target interface screenshot and the normal interface screenshot according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot;

Determining whether the cosine distance exceeds a preset threshold;

If it is determined that the cosine distance exceeds the preset threshold, acquiring a level and a position of each element control of the user interface to be detected;

Detecting whether the level and position of each of the element controls are abnormal;

If it is detected that the level and position of each of the element controls are normal, it is determined that the user interface to be detected is displayed normally.
The user interface display identification method according to claim 1, wherein the extracting the feature vector of the target interface screenshot comprises:

Selecting a first preset number of samples from the target interface screenshot, and extracting a second preset number of features for each sample;

Forming a first matrix according to the first preset number of samples and a second preset number of features extracted by each sample;

Obtaining a second matrix according to the first matrix and the transposed matrix of the first matrix;

Calculating a feature vector of the second matrix, and determining a feature vector of the target interface screenshot according to the feature vector of the second matrix.
The user interface display identification method according to claim 1, wherein the determining the target interface screenshot and the normal interface screenshot according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot The cosine distance includes:

Calculating a feature vector of the target interface screenshot and a length of a feature vector of the normal interface screenshot respectively;

Obtaining an inner product of a feature vector of the target interface screenshot and a feature vector of the normal interface screenshot;

Determining a cosine distance between the target interface screenshot and the normal interface screenshot according to a length of a feature vector of the target interface screenshot, a length of a feature vector of the normal interface screenshot, and the inner product.
The user interface display identification method according to claim 1, wherein the acquiring the level and location of each element control of the user interface to be detected comprises:

Generate a running status acquisition command through the software development kit SDK;

Acquiring the level and location of each element control of the user interface to be detected in real time according to the running state acquisition command.
The user interface display identification method according to claim 1, wherein the detecting whether the level and position of each of the element controls are abnormal comprises:

Determining a child control in each of the element controls according to a level of each of the element controls, and a parent control corresponding to the child control;

Detecting whether the position of the target child control is set at the front end of the target parent control corresponding to the target child control, and detecting whether the target child control occupies a range beyond the target parent control, and the target child control is determined Any of the child controls in each of the described element controls.
A user interface display identification terminal device, comprising: a memory, a processor, and a computer program stored in the memory and operable on the processor, wherein when the processor executes the computer program, the implementation is as follows step:

Obtaining a screenshot of the target interface of the user interface to be detected, and extracting a feature vector of the screenshot of the target interface;

Obtaining a normal interface screenshot of the pre-stored normal user interface, and extracting a feature vector of the normal interface screenshot;

Determining a cosine distance between the target interface screenshot and the normal interface screenshot according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot;

Determining whether the cosine distance exceeds a preset threshold;

If it is determined that the cosine distance exceeds the preset threshold, acquiring a level and a position of each element control of the user interface to be detected;

Detecting whether the level and position of each of the element controls are abnormal;

If it is detected that the level and position of each of the element controls are normal, it is determined that the user interface to be detected is displayed normally.
The user interface display identification terminal device according to claim 6, wherein the extracting the feature vector of the target interface screenshot comprises:

Selecting a first preset number of samples from the target interface screenshot, and extracting a second preset number of features for each sample;

Forming a first matrix according to the first preset number of samples and a second preset number of features extracted by each sample;

Obtaining a second matrix according to the first matrix and the transposed matrix of the first matrix;

Calculating a feature vector of the second matrix, and determining a feature vector of the target interface screenshot according to the feature vector of the second matrix.
The user interface display identification terminal device according to claim 6, wherein the determining the target interface screenshot and the normal interface according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot The cosine distance of the screenshot includes:

Calculating a feature vector of the target interface screenshot and a length of a feature vector of the normal interface screenshot respectively;

Obtaining an inner product of a feature vector of the target interface screenshot and a feature vector of the normal interface screenshot;

Determining a cosine distance between the target interface screenshot and the normal interface screenshot according to a length of a feature vector of the target interface screenshot, a length of a feature vector of the normal interface screenshot, and the inner product.
The user interface display and identification terminal device of claim 6, wherein the obtaining the level and location of each element control of the user interface to be detected comprises:

Generate a running status acquisition command through the software development kit SDK;

Acquiring the level and location of each element control of the user interface to be detected in real time according to the running state acquisition command.
The user interface display and identification terminal device according to claim 6, wherein the detecting whether the level and position of each of the element controls are abnormal comprises:

Determining a child control in each of the element controls according to a level of each of the element controls, and a parent control corresponding to the child control;

Detecting whether the position of the target child control is set at the front end of the target parent control corresponding to the target child control, and detecting whether the target child control occupies a range beyond the target parent control, and the target child control is determined Any of the child controls in each of the described element controls.
A computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the following steps:

Obtaining a screenshot of the target interface of the user interface to be detected, and extracting a feature vector of the screenshot of the target interface;

Obtaining a normal interface screenshot of the pre-stored normal user interface, and extracting a feature vector of the normal interface screenshot;

Determining a cosine distance between the target interface screenshot and the normal interface screenshot according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot;

Determining whether the cosine distance exceeds a preset threshold;

If it is determined that the cosine distance exceeds the preset threshold, acquiring a level and a position of each element control of the user interface to be detected;

Detecting whether the level and position of each of the element controls are abnormal;

If it is detected that the level and position of each of the element controls are normal, it is determined that the user interface to be detected is displayed normally.
The computer readable storage medium of claim 11, wherein the extracting the feature vector of the target interface screenshot comprises:

Selecting a first preset number of samples from the target interface screenshot, and extracting a second preset number of features for each sample;

Forming a first matrix according to the first preset number of samples and a second preset number of features extracted by each sample;

Obtaining a second matrix according to the first matrix and the transposed matrix of the first matrix;

Calculating a feature vector of the second matrix, and determining a feature vector of the target interface screenshot according to the feature vector of the second matrix.
The computer readable storage medium according to claim 11, wherein the determining the target interface screenshot and the normal interface screenshot according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot The cosine distance includes:

Calculating a feature vector of the target interface screenshot and a length of a feature vector of the normal interface screenshot respectively;

Obtaining an inner product of a feature vector of the target interface screenshot and a feature vector of the normal interface screenshot;

Determining a cosine distance between the target interface screenshot and the normal interface screenshot according to a length of a feature vector of the target interface screenshot, a length of a feature vector of the normal interface screenshot, and the inner product.
The computer readable storage medium according to claim 11, wherein the obtaining the level and location of each element control of the user interface to be detected comprises:

Generate a running status acquisition command through the software development kit SDK;

Acquiring the level and location of each element control of the user interface to be detected in real time according to the running state acquisition command.
The computer readable storage medium according to claim 11, wherein the detecting whether the level and position of each of the element controls are abnormal comprises:

Determining a child control in each of the element controls according to a level of each of the element controls, and a parent control corresponding to the child control;

Detecting whether the position of the target child control is set at the front end of the target parent control corresponding to the target child control, and detecting whether the target child control occupies a range beyond the target parent control, and the target child control is determined Any of the child controls in each of the described element controls.
A user interface display identification device, comprising:

a first screenshot processing unit, configured to acquire a target interface screenshot of the user interface to be detected, and extract a feature vector of the target interface screenshot;

a second screenshot processing unit, configured to acquire a normal interface screenshot of the pre-stored normal user interface, and extract a feature vector of the normal interface screenshot;

a cosine distance determining unit, configured to determine a cosine distance of the target interface screenshot and the normal interface screenshot according to the feature vector of the target interface screenshot and the feature vector of the normal interface screenshot;

a cosine distance determining unit, configured to determine whether the cosine distance exceeds a preset threshold;

An element control acquiring unit, configured to acquire a level and a position of each element control of the user interface to be detected if it is determined that the cosine distance exceeds the preset threshold;

An element control detecting unit, configured to detect whether the level and position of each of the element controls are abnormal;

The user interface display determining unit is configured to determine that the user interface to be detected is displayed normally if it is detected that the level and position of each of the element controls are normal.
The user interface display and recognition device of claim 16, wherein the first screenshot processing unit comprises:

a sample selection unit, configured to acquire a target interface screenshot of the user interface to be detected, select a first preset number of samples from the target interface screenshot, and extract a second preset number of features for each sample;

a first matrix constituting unit, configured to form a first matrix according to the first preset number of samples and a second preset number of features extracted by each sample;

a second matrix obtaining unit, configured to obtain a second matrix according to the first matrix and the transposed matrix of the first matrix;

And a feature vector determining unit, configured to calculate a feature vector of the second matrix, and determine a feature vector of the target interface screenshot according to the feature vector of the second matrix.
The user interface display identification device according to claim 16, wherein the cosine distance determining unit comprises:

a vector length calculation unit, configured to separately calculate a feature vector of the target interface screenshot and a length of a feature vector of the normal interface screenshot;

a vector inner product obtaining unit, configured to obtain an inner product of a feature vector of the target interface screenshot and a feature vector of the normal interface screenshot;

a cosine distance processing unit, configured to determine a cosine distance between the target interface screenshot and the normal interface screenshot according to a length of a feature vector of the target interface screenshot, a length of a feature vector of the normal interface screenshot, and the inner product .
The user interface display and recognition device according to claim 16, wherein the element control acquisition unit comprises:

a command generating unit, configured to generate an operation state acquisition command through the SDK if it is determined that the cosine distance exceeds the preset threshold;

The element control processing unit is configured to acquire, in real time, the level and position of each element control of the user interface to be detected according to the running state acquisition command.
The user interface display and recognition device according to claim 19, wherein the element control detecting unit comprises:

a child control unit determining unit is configured to determine a child control in each of the element controls according to a level of each of the element controls, and a parent control corresponding to the child control;

a child control detection unit is configured to detect whether a position of the target child control is set at a front end of the target parent control corresponding to the target child control, and detect whether the target child control occupies a range beyond the target parent control The target child control is any one of the child controls in each of the determined element controls.