CN115357490A

CN115357490A - Interface scaling defect detection method for mobile application and electronic device

Info

Publication number: CN115357490A
Application number: CN202210991441.7A
Authority: CN
Inventors: 苏宇辉; 王俊杰; 刘哲; 王丹丹; 李守斌; 王青
Original assignee: Institute of Software of CAS
Current assignee: Institute of Software of CAS
Priority date: 2022-08-18
Filing date: 2022-08-18
Publication date: 2022-11-18

Abstract

The invention discloses an interface scaling defect detection method for mobile application and an electronic device. The method comprises the following steps: 1) Zooming a user interface of a target mobile application at different zooming ratios to obtain an interface under each zooming ratio; establishing corresponding relations of the same component in interfaces with different scaling ratios by taking the component in the interface as granularity; 2) Based on the corresponding relation, performing inter-component analysis, namely comparing the position relation of the same component under different scaling interfaces, and detecting scaling defects caused by the change of the position relation among the components; 3) And performing intra-component analysis based on the corresponding relation, comparing and analyzing the content displayed by the component, and detecting the scaling defect caused by the inconsistency of the component content. The invention can more comprehensively and accurately find problems, quickly and automatically finish the zooming barrier-free test of the application program and ensure the development quality of mobile application.

Description

Interface scaling defect detection method for mobile application and electronic device

Technical Field

The invention relates to the technical field of computers, in particular to an automatic detection method for interface scaling defects of mobile applications and an electronic device.

Background

Nowadays, the popularity of mobile intelligent terminal devices is very wide, an android operating system has become one of the most common systems, and at least 260 ten thousand mobile application programs are published in the application market of android to date, and cover various functions.

Users with visual impairments, such as elderly people, tend to increase the font size or display size of the screen when using mobile applications to manipulate the application more easily.

However, the graphical user interfaces of many applications are designed only for default set font or display sizes, since most developers are sighted, and most of them rarely or never consider visually impaired users. This results in inconsistencies in the user interface when transitioning from default settings to larger font or display sizes, which may result in scaling defects for components of the user interface, such as component overlap, component missing, and component occlusion problems, which are also the most common of several. The user interface comprises components, the components are basic units forming the user interface, and the components are in a wide variety, such as text components, image components, drop-down box components, custom components and the like. The wide variety of components also makes the performance of scaling problems diverse and also increases the difficulty of detecting scaling problems.

In the prior art, there are several methods for detecting the scaling problem as follows:

and (5) manual detection. The basic idea is to rely on the developer to manually explore the application, record the screen display at the default scale and the corresponding screen display at the larger scale, check where the display is inconsistent by comparing the display results of the components, and record the components with display defects in these components that represent the inconsistency. However, inspection by manpower is time-consuming and laborious.

And deep learning detection based on the screenshot. The basic idea is to detect the position where zooming is problematic directly on the interface with a large zooming ratio through screenshot, and the core detection means includes but is not limited to an image feature detection method such as deep learning, but detects zooming defects only on a single screenshot without considering the dynamic property of the layout, and the problems of misjudgment and missed detection may exist. At present, an automatic and accurate interface scaling defect detection technology for mobile applications is urgently needed.

Disclosure of Invention

In view of this, the present invention provides an automatic detection method for an interface scaling defect of a mobile application, which solves the problem that the current detection scheme cannot solve. The invention provides a detection method based on comparison, namely, the scaling problem is detected by comparing the position relation and the content of layout files and components in the layout under different scaling ratios, which comprises the following steps:

s1, zooming a user interface of a target mobile application at different zooming ratios to obtain an interface under each zooming ratio; establishing corresponding relation of the same component in interfaces with different scaling ratios by taking the component as granularity in the interfaces with different scaling ratios;

and S2, performing inter-component analysis based on the corresponding relation of the components, and detecting the scaling defects caused by the change of the position relation among the components by comparing the position relation among the components under different scaling ratios.

And S3, performing intra-component analysis based on the corresponding relation of the components, comparing and analyzing the content displayed by the components, and detecting the scaling defect caused by the inconsistency of the component content.

In one embodiment, step S1 specifically includes:

s1.1, acquiring all layout files in the installation package of the mobile application program, and injecting unique component identifiers into components contained in the layout files to distinguish different types of components.

And S1.2, packaging the application program injected with the unique identifier of the component into an installation package, and automatically installing the installation package on the mobile phone.

And S1.3, installing and operating the application program on the mobile phone with different scaling ratios, enabling the application program to reach the same interface p, obtaining a layout file of the interface p and image information of components in the interface p, and identifying component pairs corresponding to the same content by utilizing the component identifiers in the layout file and the contained character strings to generate component IDs of the components.

In one embodiment, step S2 specifically includes:

and S2.1, analyzing among the assemblies according to the assembly tree of the interface, namely analyzing whether the position relation of a certain assembly relative to other assemblies changes or not, so that the display result of the assembly is changed, and the scaling problem is generated. Specifically, whether each non-root component in the component tree is visible in the component directly containing the non-root component is analyzed in the sequence of first root traversal, and the visibility including the visible scale is analyzed through the image and the visible area of the non-root component and the image and the visible area of the component containing the component. And if the non-root component has the brother component, analyzing whether the brother component shields the component or not according to the drawing sequence of the non-root component and the brother component and the visible regions and images of the components, and recording the component ID of the brother component shielding the component to form a shielding ID set. The scaling problem (visibility problem, overlap problem) is analyzed accordingly.

And S2.2, after the interfaces with two different scaling ratios are subjected to inter-component analysis, comparing and analyzing whether the two parts of contents of the components change under different scaling ratios, namely the change of visibility and shielding relation, according to the visibility of the components in the component pairs and the IDs of brother components shielding the components, and judging that the scaling problem exists if the visibility and shielding relation change.

In one embodiment, step S3 specifically includes:

and S3.1, classifying the component pairs formed by the leaf nodes in the component trees under the two scaling ratios into character component pairs and non-character component pairs.

And S3.2, extracting the character part image of the character component pair from the image of the character component under different zooming ratios by using a binarization method, carrying out connected domain statistics on the character part image, comparing the connected domains, counting the area change regions, summing the area change regions to obtain the character area, and judging that the zooming problem exists if the ratio change of the two character areas under different zooming ratios is inconsistent with the zooming ratio.

And S3.3, extracting the foreground of the component by using a binarization mode for the non-character component pair, counting the number of connected domains, judging that the scaling problem does not exist if the components under two scaling ratios have the same connected domain, otherwise calling an SSIM algorithm to identify the similarity of the image, and judging that the scaling problem exists if the similarity is lower than 0.9.

The invention also encompasses a storage medium having a computer program stored thereon, wherein the computer program is configured to carry out the method described above when executed.

Also the invention comprises an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer to perform the method as described above.

One or more technical solutions in the embodiments of the present application have at least one or more of the following technical effects:

the invention aims to detect the scaling problem, firstly, component pairs are searched for applied components, and the position relation and the visibility among the components and the change of the internal content of the components are respectively detected according to the analysis among the components and the analysis in the components which are completed by the component pairs, so as to detect whether the scaling problem exists in the components. The invention finally forms a complete component scaling detection scheme.

The method and the detection scheme of the scaling problem solve the problems of high cost and low efficiency of manual detection, and can find the problem more comprehensively and accurately by means of comparison.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.

Fig. 1 is a flowchart of an interface scaling defect automatic detection technology for a mobile application according to an embodiment of the present invention.

Detailed Description

In order to achieve the technical effects, the invention mainly adopts the following concept:

the invention aims to provide a set of detection scheme for detecting the problems of interface defects of a mobile application program caused by adjusting the zoom setting of a mobile device, including component overlapping, component missing and component shading. The method specifically comprises three steps, namely, unique identifier injection is carried out on components in a layout file, so that the corresponding relation of the components is found in component trees under different scaling ratios, namely, a component pair, in order to analyze the scaling problem of the component pair, the visibility of the components under the corresponding scaling ratios, other components which shield the components and the content of the components need to be obtained in the subsequent steps, and whether the scaling problem exists in the components under a certain scaling ratio is judged by comparing the inconsistency of the information; specifically, firstly, the scaling problem between components is analyzed by using the position relationship between the component pair and the components in the component tree and the attributes (such as drawing sequence and the like) of the component image and the components, the visibility of the components is specifically obtained, identifiers of other components which shield the components are compared according to the component pair to determine whether the information has difference, and if the information has difference, the scaling problem (the visibility problem and the shielding problem) is reported; and then analyzing the scaling problem in the component by utilizing the component pair and the content of the component, specifically acquiring the content of the component, analyzing whether the content has a difference according to the attribute of the component, and reporting the scaling problem (the problem of content missing) if the content has the difference. Based on the three steps, a complete and automatic scaling defect detection scheme is finally formed.

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The embodiment provides a method for detecting an interface defect problem of a mobile application program caused by adjusting a zoom setting of a mobile device, please refer to fig. 1, and the method includes:

and S1, establishing a corresponding relation between components by taking the components as granularity in interfaces under different scaling ratios.

The method aims to accurately and finely detect the scaling problem, so the method takes the component as the minimum detection granularity and detects the scaling problem by comparing the display conditions of the component under different scaling ratios. However, it is often difficult to accurately and quickly find components corresponding to the same content in interfaces with different scales because it is not guaranteed that the components contain unique ID identifiers, and it is also difficult to establish the correspondence between the components. Meanwhile, the component tree may also contain list components capable of scrolling, list items in the list components may have repeatability in structure, and the list items may have the same ID identification, and the list items are difficult to distinguish simply by using the ID identification, which makes it difficult to determine the list items containing the same content at different scales and establish component pair relationships for the list items. In order to solve the problem, in step S1, the method establishes a correspondence between component pairs by injecting a component unique identifier and using a heuristic method in combination with a text inside a component. Specifically, the method comprises the following substeps:

s1.1, acquiring all layout files in the mobile application program, and injecting unique component identifiers into components contained in the layout files to distinguish different types of components. One interface may use multiple layout files, and one layout file may be used multiple times by one or more interfaces.

The layout File is a File that controls the display result of the mobile application program interface, and each component widget exists in the layout File in the form of a tree structure (i.e., a component tree widget tree). The component widget among the component tree widget trees may contain K (K)>0) Component { widget _j ,widget _j+1 ,…,widget _j+K Or zero components. In the component tree, one component can only be included by another component, so that a strict tree structure is formed. In the component tree, the display of a component is controlled by the component that directly contains the component. In the component tree, in most cases only leaf components (components that do not contain any components) carry content, e.g. text, images.

Due to the component widget in the layout file _i May lack a unique identifier and is difficult to identifyTo correctly distinguish between different types of components, this step assigns the attribute auxiliary ID _i I.e. unique string identifiers are injected into all widgets _i In each widget _i All have auxiliary ID _i Therefore, after the attributes are injected, the widget is relied upon _i Auxiliary ID of _i The method can distinguish the components according to the information.

And S1.2, packaging the application program injected with the unique identifier of the component into an installation package, and automatically installing the installation package on the equipment to be tested.

The specific implementation of the installation package packaging in the step can be finished from two angles, namely before the application program is packaged into the installation package, directly injecting the attributes of the components of the layout file from the source code, and compiling the source code into the installation package after injection; the other one is to perform decompiling on the packed installation package, perform attribute injection from the decompiled layout file, and pack the source code and various resources into the installation package after injection.

After the installation package is packaged, the installation package is subjected to necessary processing, such as operations of signature, byte alignment and the like, and then is installed on the device to be tested.

And S1.3, installing and operating the application program on the mobile phone with different scaling ratios, enabling the application program to reach the same interface, obtaining a layout file of the interface and image information of the components, generating a component ID of the corresponding component by using the component identifier of the component in the layout file and the character string contained in the component, and identifying the component pair corresponding to the same content.

After a corresponding program is installed on a device to be tested, the program is adjusted to a certain interface p to be tested and the scaling size s by means of operating the program by means of a test script and the like, and a component tree PageWidgetTree of the interface p is obtained ^p,s And all components therein

Image of (2)

Components in a component tree

Comprising a series of attributes

Such as height

Width of

Rendering order

Text

Rollability

Scaling

Whether to use an ellipsis to replace omitted characters after the characters overflow

Etc. in addition, the component itself contains an image in RGBA format (the common PNG format image is RGBA format, containing four channels)

I.e. a width of

Height of

A depth 4 multi-channel image, the last channel of which is a transparency channel image

The first three channels are red, green and blue channel images

By using

Image, can graying the image to obtain corresponding grayscale image

Collecting the component tree PageWidgetTree ^p,s For each component therein

Generating component ID attribute values

And find component pairs using the ID

Where s represents a normal scaling and s' represents a larger scaling, the component pair is generated as follows:

1. traversing all components in a component tree

Record the value of its attribute auxiliaryID

If the component is

Is unique, i.e. not identical to the auxiliaryID value of any other component, i.e. as the ID attribute value of the component

Then directly connect the components

Is

Value as a component

The component ID value of (1).

2. For other cases, the component is determined

Whether or not to have a component with the brother

(having the same parent component) has the same value of auxiliary ID, and if so, traverses this component in some determined order of traversal of the tree

All descendant components of

The traversal order may be one of a first root traversal order, a middle root traversal order, and a last root traversal order, and the traversal order must remain consistent in generating the component pairs. In the process of traversing, generating character strings according to the traversed components

If a certain descendant component is encountered

And this descendant component

auxiliaryID value of (a) and its sibling component

The auxiliary ID value of the child component is the same, then the descendant component

And

all descendant components of do not participate

Generating; for other traversed components

Connecting the value of the text attribute and the value of auxiliaryID into a character string according to the traversal order

After traversing all

Then, the component is assembled

Is

Value of (A) and

are connected together to form an assembly

ID value of (2). Since a certain layout file may be used by a page for multiple times, the auxiariaryID injected in the layout file may repeatedly appear in the page component tree, so that a new ID needs to be generated for distinguishing components in the page component tree; sibling components may originate from the same layout file and therefore may contain the same auxiliary id value, as they may be reused.

3. And establishing a component pair relation according to the generated ID value:

a) For injection of auxi by step 1Components of liaryID

Directly finding the component containing the same ID in another component tree with different scaling s

If finding the corresponding component, establishing the component pair relation for the pair of components

b) For components injected with auxiliary ID by step 2

If finding out the corresponding component, directly establishing the component-to-component relation

And is a descendant component of the two components

And

and establishing a corresponding relation. Specifically, first, the component is recorded as follows

ID value of

Traverse the descendant component of this component

In the process of traversing, if encounteringCertain descendant component

And this descendant component

Auxiliary ID value of and its sibling component

The auxiliary ID value of is the same, then this component is temporarily not used

And assembly

The method of establishing a relationship (according to step b) for the descendant component of (c), the component being selected from the group consisting of

Component pair relationships are established with its descendant components and the components in the component tree under s' because the components

auxiliaryID of (a) is also injected by step 2); for other descendant components

Then will be

And this descendant component

Is connected together as this descendant component

If it is possible to find a component having the same ID in another component tree

The pair of components are established into corresponding relation, namely, the component pair is generated

And S2, performing inter-component analysis based on the corresponding relation of the components, and detecting scaling defects caused by the change of the position relation and the change of the visibility between the components by comparing the position relation between the components under different scaling ratios.

In the component tree, the components are organized together according to the tree structure, the components are rendered on the interface according to the tree structure, and the problem of shielding exists among the components according to the difference of the rendering sequence drawingOrder of the components. Because of the structural relationship of the tree, the parent component controls the visibility of the direct sub-components, and if the visibility changes, the sub-components may be partially invisible or completely invisible. Therefore, based on the relationship between the tree structure and the parent-child components, the step carries out analysis among the components, and the scaling defects are analyzed according to the position relationship among the components and the change of visibility. Specifically, this step includes several substeps as follows.

And S2.1, analyzing among the assemblies according to the assembly tree of the interface, namely analyzing whether the position relation of a certain assembly relative to other assemblies changes or not, so that the display result of the assembly changes and the scaling problem is generated. Specifically, whether each non-root component in the component tree is visible in the component directly containing the non-root component is analyzed in the sequence of first root traversal, and the visibility including the visible scale is analyzed through the image and the visible area of the non-root component and the image and the visible area of the component containing the component. And if the non-root component has the brother component, analyzing whether the brother component shields the component or not according to the drawing sequence of the non-root component and the brother component and the visible regions and images of the components, and recording the component ID of the brother component shielding the component to form a shielding ID set. The scaling problem (visibility problem, overlap problem) is analyzed accordingly.

For a non-root component

In other words, its parent component

Controlling its visibility (visibility range) and its rendering order relative to its sibling components, this step analyzes each non-root component in the component tree in a root-first traversal manner

And its sibling component set

A condition in which it is occluded, i.e. an overlap condition, wherein

Corresponding to the component

We utilize the visibility matrix visible ^p,s,i To describe the assembly

Visibility of, to, the component

In other words, the visibility matrix is initialized to a width of

Has a height of

The value in this matrix is either 0 or 1,0 indicating invisible and 1 indicating visible. Thereafter, the image of the component is utilized

Alpha of transparency channel ^p,s,i To initialize the visibility matrix, i.e.:

transparency channel alpha is a width of

Has a height of

Alpha of the two-dimensional matrix ^p,s,i The size of the matrix is consistent with the size of the visible matrix, and for the element of the m-th row and the n-th column in the visible matrix, if the two-dimensional matrix is alpha ^p,s,i The value of the element in the mth row and the nth column in the matrix is greater than 0, the value of the element in the mth row and the nth column in the matrix is 1, otherwise, the value is 0.

This step traverses the component tree pagewidget tree in a root-first traversal manner ^p,s Specifically, after a node is traversed, if there is a subcomponent in the node, the subcomponent is traversed according to the drawing order from large to small. Traversing each non-root component, two parts of the content need to be analyzed, as follows:

a first part: for each non-root component

We want to determine if it is directly containing its parent component

Invisible in, whether or not there is a parent component

(ii) case of occlusion, calculating attribute

The corresponding judgment method is as follows:

1. first, determine the non-root component

Whether an area in the screen is in the parent component

Outside the area in the screen, if the area is outside and the parent component is not scrollable, then the non-root component is judged not to be visible, and the component is set

Is/are as follows

The attribute is false;

2. obtaining a cropped image of the non-root component inside the region of the parent component if the region of the non-root component is within the region of the parent component

And corresponding clipping coordinate region

3. Obtaining an image of a parent component, and utilizing a template matching algorithm in the image field to cut the image of the non-root component

Searching in the image of the parent component as a template, if the final searched area

And a clip coordinate region of the sub-assembly

If the intersection ratio of (1) is equal to 1, the non-root component is visible in the parent component, and the component is recorded

Is

Attribute false, otherwise invisible, and record component

Is

The attribute is true.

4. Obtaining a parent component

Visible matrix of ^p,s,j Obtaining a non-root component

Visible matrix visible ^p,s,i If the component

Is

With the attribute of false, the visible matrix visible ^p,s,i All values in (a) are assigned a value of 0. Otherwise, the step 5 is entered.

5. According to non-root components

Visible matrix visible ^p,s,i To obtain a corresponding visible area

According to the cutting coordinate area

Clipping the visible matrix of the non-root component and the visible matrix of the parent component (clipping the visible matrix, namely acquiring the visible matrix in clippingMatrix blocks inside the coordinate region) to get a crotedVisible ^p,s,i And crop Visible ^p,s,j And obtaining cropeddandibile ^p,s,i Namely:

in short, to the crop Visable ^p,s,i And cropedD ANDVisible ^p,s,j Performing a bitwise sum (AND) operation to obtain crop ANDVisible ^p,s,i . Thereafter, according to the clipping coordinate region

Will cropeddanducibile ^p,s,i Replace back the visable ^p,s,i In the clipping region of (2), obtain the updated visible ^p,s,i And calculating a new visible area

If it is

Is equal to

Recording assembly

Property of whether or not it is occluded by a parent component

False, otherwise true.

A second part: at the same time, for all

Non-root component with attribute of true

If its sibling component set

Not an empty set, taking advantage of its drawing order among parent components

Analyzing occlusion relation of non-root component and brother component set thereof, and recording occlusion non-root component

Of sibling components

The method comprises the following specific steps:

1. because components with larger drawingOrder may be drawn onto components with smaller drawingOrder, the non-root components and their sibling components are sorted in descending order of drawing order among the parent components, and their sibling components are filtered in order of drawing order from larger to smaller

Namely the assembly

If brother component

Namely the assembly

Is/are as follows

Is greater than

And is provided with

To true, this component is applied

Adding to an ordered set

Therein, wherein

Corresponding sibling component

Namely the assembly

2. Traversing the ordered set Peer according to the order from big to small of the drawing order ^p,s,i . According to the assembly

Position coordinates and currently traversed components

The position coordinates of the two components are calculated, and if the two components intersect in the screen, the components are updated

The visibility matrix of (a). Namely, calculating the intersection position area overlapping region of the two components in the screen ^p,s,i,k The width of the intersection region is pverlaparegionwidth ^p,s,i,k Height of overlap pRegion height ^p,s,i,k The coordinate of the upper left corner on the user interface is (overlap pRegionX) ^p,s,i,k ,overlapRegionY ^p,s,i,k ) According to the intersection area and its position and components on the screen

Visible matrix of ^p,s,i Visible matrix visible ^p,s,i Cut out, we get the assembly

The clipping visible matrix overlap visible ^p,s,i And an assembly

The clipping visible matrix overlap visible ^p,s,k . At this time, the visible matrix overlapVisible is calculated ^p,s,i Visible area of

According to the two clipping visible matrixes, an intersecting shielding matrix overlapBlockVisible is obtained through calculation ^p,s,i,k ：

Namely the quilt component

The blocked parts of the visible area are not visible, and the visibility of other parts which are not blocked is not influenced.

Overlap region according to intersection region ^p,s,i,k To overlay the logo ^p,s,i,k Matrix Replacing to overlapVisible ^p,s,i In the corresponding region, thereby updating the overlapVisible ^p,s,i . At the moment, an intersecting occlusion matrix overlapBlockedVisile is calculated ^p,s,i,k Visible area of

If it is

Explain the component

Sheltered from assembly

And assembly

ID attribute value of (2)

Joining to overlapping sets of sibling components

In (1). Post-analysis component

Whether to shield assembly

Until Peer is to be ordered set ^p,s,i And (5) finishing all analysis.

After performing inter-component analysis on the interface p of the two scales s and s', a PageWidgetTree is formed in a component tree ^p,s Each of the non-root components

Both have two attribute values

And collections

PageWidgetTree in component tree ^p,s′ Each of the non-root components

Both have two attribute values

And set

According to component pair

If it is

Or

Or

Then a scaling problem is determined to exist.

Since the leaf components are carried in the component tree, this step will analyze whether the contents of the leaf components have scaling defects, such as missing contents, at different scaling ratios. The components can be roughly divided into text components and other components, and since the text in the text components is wrapped due to the adjustment of the scaling, the recognition accuracy is reduced and the operation pressure is increased according to the common OCR method, so that the following steps are designed to complete the analysis in the components. Specifically, the following substeps are included.

For a component widget, the component widget is divided into a text component or a non-text component according to whether the component widget contains a text attribute, if the component widget contains the text attribute, the component is the text component, otherwise, the component is the non-text component.

And S3.2, extracting the character component pair from the images of the components under different scaling ratios by using a binarization method to obtain images of character parts of the components, carrying out statistics of connected domains for the images of the character parts, comparing the connected domains, counting regions with changed visible areas, summing the regions with changed visible areas to obtain the character visible areas, and judging that scaling problems exist if the ratio change of the character visible areas under two different scaling ratios is inconsistent with the scaling ratios. This step has several substeps as follows:

1. for text components

According to its image ^p,s,i Obtaining rgb ^p,s,i Image, alpha ^p,s,i Image and gray ^p,s,i And (4) an image.

2. Obtaining a binary image binary ^p,s,i And binary ^{^p,s,i} :

1) If gray ^p,s,i The image is an all 0 image, then alpha will be ^p,s,i The image is binarized by utilizing an Otsu algorithm to obtain a binarization threshold value

And obtaining binary image binary according to threshold value ^p,s,i And binary ^{^p,s,i} Namely:

2) If gray ^p,s,i The image is not all 0, and the gray image is processed as follows to obtain the gary ^{^p,s,i} Image:

then, in gary ^{^p,s,i} In the image, the calculation is carried out in the non-1 area by utilizing the Dajin algorithm

And using the threshold value to complete binarization on the gray image by the method in the step 2 to obtain binary ^p,s,i And binary ^{^p,s,i} And (4) an image.

3. Obtaining binary ^p,s,i And binary ^{^p,s,i} After the image, analyzing the binary by using a connected component analysis method ^p ^,s,i And binary ^{^p,s,i} The 8-connected domain with the value of 1 in the image obtains the area of each connected domain, namely the number of pixels with the value of 1. Obtaining a set of corresponding connected domains

And

wherein,

is binary ^p,s,i The area of the t-th connected component in the image,

is binary ^{^p,s,i} Area of the e-th connected component in the image.

4. After performing intra-component analysis on the interface p of the two scales S and S', a certain character component pair under the two scales is obtained by the method of step S1.3

Binary image of (1), binary ^p,s,i And binary ^p,s′,j And is combined withObtaining the corresponding connected domain set conn ^p,s,i And conn ^p,s′,j . According to the obtained areas of the connected domains, the connected domains with the same area in the two components are screened out, and the areas of the remaining connected domains are summed respectively to obtain connSum ^p,s,i And connSum ^p,s′,j If, if

The text scaling inside the text component is legal, otherwise, if so

True, then legal, otherwise, bind is used ^{^p,s,i} And binary ^{^p,s′,j} And calculating to obtain connSum ^{^p,s,i} And connSum ^{^p,s′,j} If, if

The text scaling inside the text component is legal, otherwise a scaling problem is reported.

And S3.3, extracting the foreground of the component by using a binarization mode for the non-character component pair, counting the number of connected domains, judging that the scaling problem does not exist if the components under two scaling ratios have the same connected domain, otherwise calling an SSIM algorithm, and judging that the scaling problem exists if the similarity is lower than 0.9.

1. For non-text components

2. Will gray ^p,s,i The image is calculated by the Dajin algorithm

And carrying out binarization to obtain a binarization graphImage bank ^p,s,i Namely:

3. in obtaining binary ^p,s,i After the image, the bind is analyzed by using the 8-connected domain analysis method ^p,s,i Obtaining the area of each connected domain, namely the number of pixels with the value of 1, and obtaining the number count of the connected domains ^p,s,i 。

4. Obtaining non-text component pairs at two scales by the method of step S1.3

Binary image of (1), binary ^p,s,i And binary ^p,s′,j And obtaining the corresponding connected domain number count ^p,s,i And count ^p,s,j . If count ^p,s,i ＝count ^p,s,j If yes, the content scaling inside the non-text component is legal, otherwise, step 5 is performed.

5. Will gray ^p,s,i And gray ^p,s′,j Cutting to obtain croppedGray ^p,s,i And croppedGray ^p,s′,j To cut gray ^p,s,i For example, croppedGray ^p,s,i The requirements are as follows:

croppedGray ^p,s,i has a width and a height of W ^p,s,i And H ^p,s,i

croppedGray ^p,s′,j Also following the same manner, by gray ^p,s′,j And (5) cutting.

6. Calculating croppedGray by using SSIM algorithm ^p,s,i And croppedGray ^p,s′,j If the similarity value is greater than 0.9, the content scaling is legal, otherwise a scaling problem is reported.

Additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

It will be apparent to those skilled in the art that various modifications and variations can be made in the embodiments of the present invention without departing from the spirit or scope of the embodiments of the invention. Thus, if such modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to encompass these modifications and variations.

Claims

1. An interface zooming defect detection method for mobile application comprises the following steps:

1) Zooming a user interface of a target mobile application at different zooming ratios to obtain an interface under each zooming ratio; establishing the corresponding relation of the same component in interfaces with different scaling ratios by taking the components in the interfaces as granularity;

2) Performing inter-component analysis based on the corresponding relationship, namely comparing the position relationship of the same component under different scaling interfaces, and detecting scaling defects caused by the change of the position relationship among the components;

3) And performing intra-component analysis based on the corresponding relation, comparing and analyzing the content displayed by the components, and detecting the scaling defects caused by the inconsistency of the component content.

2. The method according to claim 1, wherein the method for establishing the corresponding relationship is: acquiring all layout files of the target mobile application, and injecting a unique component identifier into the component contained in each layout file to distinguish different types of components; packaging the target mobile application injected with the component unique identifier into an installation package, installing the installation package on a plurality of smart phones which are displayed in different scaling ratios, and operating the target mobile application to reach the same interface p; the method comprises the steps of obtaining a layout file of an interface p and image information of components in the interface p, generating component IDs of corresponding components by utilizing component identifiers in the layout file and character strings contained in the components, taking the components with the same component IDs as a component pair, and establishing corresponding relations of the same component in interfaces with different scaling ratios.

3. The method of claim 2, wherein each component widget in the layout file forms a component tree widget in a tree structure form; ith component widget in component tree Widget _i Containing K components { widget _j ,widget _j+1 ,…,widget _j+K Either } or zero components; generating a unique string identifier for each component as the ID attribute auxiliaryID of the corresponding component; operating the target mobile application to reach the interface p, and acquiring the corresponding component tree PageWidget of the interface p when the scaling is s ^p,s And an image of each component therein, wherein

As a pagewidget tree ^p,s Middle ith component

The image of (a); according to

Component ID of (1) looking up the corresponding component in the interface p at the scaling s

Get the component pair

4. The method of claim 3, wherein generating a component

The method of component ID value of (1) is:

11 Traverse each component in the component tree Widget and record its ID attribute auxiliaryID; if it is

ID attribute value of

Is unique, it is taken as a component

ID attribute value of, component

As a component

Otherwise, step 12) is performed;

12 ) a judgment component

Whether it has the same auxiliary ID value as its sibling component, if any

Sibling components with the same auxiliary id value; traversing all the descendant components of the sibling component, stopping traversing the sibling component and the descendant components thereof if encountering the descendant component with the auxiaryID value not unique in the traversing process, connecting the text attribute values of the traversed components into a character string postfix, connecting the auxiaryID value and the postfix of the traversed components together to form the component

The component ID value of (1).

5. The method of claim 3, wherein the assembly is in a form of a kit

Including height

Width of

Rendering order drawingOrder _i ^p,s Text, text

Rollability

Scaling

The image

Is a width

Height

A depth 4 multi-channel image, the image

The last channel of (2) is a transparency channel image

The first three channels are red, green and blue channel images

6. The method of claim 5, wherein the inter-component analysis is performed by:

21 PageWidget Tree for the corresponding interface p at scale s ^p,s Traversing the component tree PageWidgetTree in a root-first traversal manner ^p,s If the current component is a non-root component

Then utilize the non-root component

Image of (2)

Transparency channel alpha of (1) ^p,s,i Is initialized to a width of

Has a height of

Two-dimensional visible matrix visible ^p,s,i ；

22 Determine non-root components

Whether a region in the screen is in its parent component

Outside the area in the screen, if outside the area, and parent component

If it is not rolling, then determine the non-root component

Is not visible, and is provided with a component

Is

The attribute is false; if within the region, then a non-root component is obtained

At its parent component

Within the region of (1)

And corresponding clipping coordinate region

23 Will crop the image

As a template, at the parent component

Performing a matching search in the image of (1), if the searched area

And cropping coordinate regions

If the cross-over ratio of (1) is equal to 1, then the non-root component is determined

At parent component

Visible recording assembly

Is/are as follows

Attribute false, otherwise invisible, and record component

Is/are as follows

Attribute is true;

24 Get parent component

Visible matrix of ^p,s,j Obtaining a non-root component

Visible matrix visible ^p,s,i If the component

Is/are as follows

With the attribute of false, the visible matrix visible ^p,s,i All values in (1) are assigned to 0; otherwise, go to step 25);

25 According to non-root components

Visible matrix of ^p,s,i Obtaining the corresponding visible area

According to the cutting coordinate region

For non-root component

Visual matrix and parent component

Clipping the visible matrix, and performing bitwise sum operation on the clipping result to obtain crop ANDVisible ^p,s,i (ii) a Then according to the cutting coordinate region

Will cropeddanducibile ^p,s,i Replace back the visable ^p,s,i In the cutting area of (2), obtaining an updated visible ^p,s,i And calculating a new visible area

If it is

Is equal to

Recording assembly

Properties of whether occluded by parent component

False, otherwise true;

26 For

Non-root component with attribute of true

If its sibling component set

If not, then draw order in parent component is utilized _i ^p,s Analyzing non-root components

Occlusion relationships with its set of sibling components and recording occluded non-root components

Of sibling components

27 According to the method of the steps 21 to 26) to scale the component tree PageWidget of the corresponding interface p when the scale s' is scaled ^p,s′ Processing to obtain the PageWidgetTree ^p,s′ Each inA non-root assembly

Property value of (2)

Overlapping sets with sibling components

28 For component pairs

If it is

Or alternatively

Or

It is determined that a scaling problem exists.

7. The method of claim 6, wherein an overlapping set of sibling components is obtained

The method comprises the following steps:

261 Will not be a root component

And its sibling component is based on the parent component

Drawing order of (1)Sorting in a descending order, and screening brother components of the sequence according to the sequence from big to small of the drawing order; wherein the non-root elements

Qth large sibling component of (1)

PageWidgetTree being a component tree ^p,s The kth component in

Brother component

Is/are as follows

Greater than drawingOrder _i ^p,s And is

To true, then the component is

Adding to an ordered set

In which

Corresponding sibling component

Namely the assembly

262 ) traverse the ordered set Peer in descending order of drawing order ^p,s,i According to the assembly

Position coordinates and currently traversed components

If the two components intersect in the screen, the intersection area of the two components in the screen is calculated, and the components are paired according to the intersection area and the positions of the components on the screen

Visible matrix of ^p,s,i Cutting to obtain assembly

The clipping visible matrix overlap visible ^p,s,i And assembly

The clipping visible matrix overlap visible ^p,s,k (ii) a Calculating the visible matrix overlapVisible ^p,s,i Visible area of

And intersecting occlusion matrix overlapBlockedVisible ^p,s,i,k ；

263 According to the intersection region, will intersect the occlusion matrix overlappblockdvisible ^p,s,i,k Replacement to overlapVisible ^p,s,i In the corresponding region, thereby updating the overlapVisible ^p,s,i (ii) a Calculating an intersecting occlusion matrix overlap pBlockedVisile ^p,s,i,k Area of (2)

If it is

Then the decision component

Shielding component

And the components

Add the ID attribute value of (c) to the sibling component overlapping set

In (1).

8. The method of claim 2, wherein the in-module analysis is performed by:

31 Classifying component pairs consisting of leaf nodes in the component trees under two different scales into literal component pairs and non-literal component pairs;

32 For a text component pair, extracting images of text parts in the component from images of the component at different scales, and judging that scaling problems exist if the ratio change of text areas at two different scales is inconsistent with the scales;

33 For non-character component pairs, extracting the foreground of the components and counting the number of connected domains, if the components under two scaling ratios have the same connected domain, judging that scaling problems do not exist, otherwise, calculating the image similarity of the two components, and if the similarity is lower than a set threshold, judging that scaling problems exist.

9. An electronic apparatus, comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program comprising instructions for carrying out the steps of the method of any of claims 1 to 8.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 8.