CN107315680B - Method and system for detecting Hybrid mobile application user interface - Google Patents
Method and system for detecting Hybrid mobile application user interface Download PDFInfo
- Publication number
- CN107315680B CN107315680B CN201710385933.0A CN201710385933A CN107315680B CN 107315680 B CN107315680 B CN 107315680B CN 201710385933 A CN201710385933 A CN 201710385933A CN 107315680 B CN107315680 B CN 107315680B
- Authority
- CN
- China
- Prior art keywords
- similarity
- user interface
- type
- calculating
- mobile application
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/38—Creation or generation of source code for implementing user interfaces
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Human Computer Interaction (AREA)
- User Interface Of Digital Computer (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a method and a system for detecting a Hybrid mobile application user interface, wherein the method comprises the following steps: acquiring HTML-format layout files of a user interface to be tested and a target user interface; extracting basic features representing the visual effect of the user interface from the layout file, wherein the basic features representing the visual effect of the user interface comprise: element type, element position, and element size; and calculating the similarity between the user interface to be detected and the target user interface according to the basic characteristics representing the visual effect of the user interface. The method can be used for analyzing the similarity of the user interface of the large-scale hybrid mobile application by statically extracting and analyzing the visual characteristics, has good expandability and has good efficiency gain compared with the traditional dynamic analysis method.
Description
Technical Field
The invention relates to the technical field of network technology and information security, in particular to a method and a system for detecting a Hybrid mobile application user interface.
Background
Hybrid mobile application, also called HTML 5-based mobile application, is a recently popular mobile application development model. The method takes a browser environment (WebView) built in an android platform as an operating environment, a JavaScript API for accessing android system services is provided for JavaScript codes in upper WebView through a system access plug-in, and the logic of an application program is expressed by JavaScript codes and HTML. The Hybrid mobile application solves the problems of customization, self-adaptation and the like of an application interface on a heterogeneous mobile terminal through a PhoneGap framework. The PhoneGap accesses the plug-in through the system and supports the conversion and deployment of the Web application to various different mobile platforms.
The Hybrid mobile application is an application program between a Web application and an Android Native (Native) application, and has the advantages of good user experience of the Native (Native) application and cross-platform development of the Web application. The Hybrid mobile uses HTML5 technology (HTML5+ JavaScript + CSS, where HTML5 is responsible for page content, JavaScript is responsible for function logic, and CSS is responsible for visual rendering effect) to write, and interprets running applications on WebView.
With the development of mobile internet in recent years, the interaction with users through GUI (Graphical user interface) is more and more complex, and many designs only consider the functional factors and ignore the security problem. With the popularization of Hybrid applications, attacks against security holes in the GUI design of Hybrid applications have emerged. Ranging from phishing attacks based on User Interface (UI) impersonation, to UI hijacking attacks that are specific to a particular UI Interface. The essence of these security issues is to exploit UI similarities to spoof attacks on the user's vision.
At present, most of detection methods for application UI similarity are directed at Web applications or one method is Native application to Android (Android), and the detection methods mainly include two types: the method comprises the steps of carrying out image processing on a rendered UI of an Android Native application, carrying out similarity analysis, carrying out screen capture on the UI of the application through dynamic testing, extracting characteristics of a screen captured picture by using an image processing algorithm, extracting information such as edges, colors and character contents, and finally carrying out similarity calculation by using the extracted information. Such as: luck Malisa et al propose a method for detecting similar UIs, which captures a screen of a UI during application running, extracts edge information of different elements in the UI by using an image processing algorithm, and forms the edge information into a feature set. And performing similarity calculation on the feature sets generated by the two UIs during detection so as to detect similar UIs.
And the other method is to perform static analysis on a source code file of the UI generated by the Web application, and perform similarity detection by analyzing the similarity of the source code or performing similarity detection on an intermediate result after the source code is subjected to some processing and analysis processing. Such as: the DOMAntiPhish analyzes a DOM Tree (Document Object Model Tree) of a website HTML file, calculates the similarity of the structure of the DOM Tree, designs a similarity calculation algorithm aiming at different tags, namely leaf nodes of the DOM Tree, and finally calculates and summarizes the similarity from the leaf nodes into the similarity of an overall webpage structure Tree, thereby detecting the structurally similar webpages.
In the related art, although the UI similarity detection method based on the image processing algorithm can directly detect the visual features presented to the user, the image processing algorithm has the defects of large resource occupation, long operation time, incapability of batch static analysis and the like, which are also very obvious. The method for analyzing the source code generating the UI can perform static analysis in a large scale, and meanwhile, the operation efficiency is higher than that of an image processing algorithm. However, the existing method for analyzing by using source code mainly aims at Web application and Native application, and cannot detect Hybrid application. Meanwhile, the detection method does not detect visual features representing the visual effect of the UI, so that the detection often deviates from the real problem, and the detection accuracy is influenced.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art described above.
Therefore, the invention aims to provide a detection method of the user interface of the Hybrid mobile application. The method for detecting the Hybrid mobile application user interface mainly analyzes static codes of visual features of the user interface, extracts the visual features from HTML layout files of the user interface to be detected and the target user interface, and calculates the similarity between the user interface to be detected and the target user interface according to the features.
Another objective of the present invention is to provide a detection system for a Hybrid mobile application user interface.
In order to achieve the above object, an aspect of the present invention discloses a method for detecting a user interface of a Hybrid mobile application, including: acquiring HTML-format layout files of a user interface to be tested and a target user interface; extracting basic features representing the visual effect of the user interface from the layout file, wherein the basic features representing the visual effect of the user interface comprise: element type, element position, and element size; and calculating the similarity between the user interface to be detected and the target user interface according to the basic characteristics representing the visual effect of the user interface.
According to the detection method of the Hybrid mobile application user interface, the visual characteristics are extracted from the acquired HTML layout files of the user interface to be detected and the target user interface, and the similarity between the user interface to be detected and the target user interface is calculated according to the characteristics.
In addition, the method for detecting the user interface of the Hybrid mobile application according to the above embodiment of the present invention may further have the following additional technical features:
further, the specific similarity calculation method is as follows: s10: combining all elements of the user interface to be tested and the target user interface pairwise to form a plurality of element combinations, and calculating the similarity of the element combinations one by one, wherein the calculating the similarity of the element combinations comprises the following steps: calculating element type similarity, calculating element position similarity and calculating element size similarity; s20: calculating the weighted similarity of the element combination one by utilizing the element area ratio; s30: and selecting the maximum value of the element combination weighting similarity as the similarity between the user interface to be tested and the target user interface.
Further, the extracting of the basic features representing the visual effect of the user interface from the layout file specifically includes: performing document-to-DOM Tree analysis calculation on the layout file in the HTML format to obtain the element type of each leaf node; and acquiring the element position and the element size of each leaf node according to the attribute tag of each leaf node of the DOM Tree.
Further, still include: storing the base features of leaf nodes representing the visual effect of the user interface as a data element, wherein a leaf node corresponds to a data element; a plurality of data elements are grouped into a data set.
Further, the obtaining of the layout file in the HTML format of the user interface to be tested is specifically to obtain the layout file in the HTML format of the relevant user interface from the installation package of the hybrid application to be tested and the installation package of the target hybrid application.
In another aspect of the present invention, a system for detecting a Hybrid mobile application user interface is disclosed, comprising: the layout acquisition module is used for acquiring layout files in HTML (hypertext markup language) formats of the user interface to be tested and the target user interface; a visual feature extraction module, configured to extract basic features representing a user interface visual effect from the layout file, where the basic features representing the user interface visual effect include: element type, element position, and element size; and the user interface similarity calculation module is used for calculating the similarity between the user interface to be detected and the target user interface according to the basic characteristics representing the visual effect of the user interface.
According to the detection system of the Hybrid mobile application user interface, the visual characteristics are extracted from the acquired HTML layout files of the user interface to be detected and the target user interface, and the similarity between the user interface to be detected and the target user interface is calculated according to the characteristics, so that the visual characteristics are extracted and analyzed statically, the method can be used for analyzing the similarity of the large-scale Hybrid mobile application user interface, has good expandability, and has good efficiency gain compared with the traditional dynamic analysis method.
In addition, the detection system of the Hybrid mobile application user interface according to the above embodiment of the present invention may further have the following additional technical features:
further, the specific similarity calculation mode of the user interface similarity calculation module is as follows: s10: combining all elements of the user interface to be tested and the target user interface pairwise to form a plurality of element combinations, and calculating the similarity of the element combinations one by one, wherein the calculating the similarity of the element combinations comprises the following steps: calculating element type similarity, calculating element position similarity and calculating element size similarity; s20: calculating the weighted similarity of the element combination one by utilizing the element area ratio; s30: and selecting the maximum value of the element combination weighting similarity as the similarity between the user interface to be tested and the target user interface.
Further, the visual feature extraction module specifically includes: performing DOMTree analysis calculation on the layout file in the HTML format to obtain the element type of each leaf node; and acquiring the element position and the element size of each leaf node according to the attribute tag of each leaf node of the DOM Tree.
Further, still include: storing the base features of leaf nodes representing the visual effect of the user interface as a data element, wherein a leaf node corresponds to a data element; a plurality of data elements are grouped into a data set.
Further, the layout obtaining module is specifically configured to obtain the layout files in the HTML format of the relevant user interface from the installation package of the hybrid application to be tested and the installation package of the target hybrid application.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow diagram of a method for detecting a Hybrid mobile application user interface according to one embodiment of the present invention;
FIG. 2 is a flowchart of a method for detecting a Hybrid mobile application user interface according to another embodiment of the present invention;
FIG. 3 is a flowchart of a method for detecting a Hybrid mobile application user interface according to yet another embodiment of the present invention; and
FIG. 4 is a block diagram of a detection system for a Hybrid mobile application user interface in accordance with one embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
The following describes a method and a system for detecting a Hybrid mobile application user interface according to an embodiment of the present invention with reference to the accompanying drawings.
FIG. 1 is a flowchart of a method for detecting a user interface of a Hybrid mobile application according to an embodiment of the present invention.
As shown in FIG. 1, a method for detecting a user interface of a Hybrid mobile application according to an embodiment of the present invention includes
S110: and acquiring layout files in HTML (hypertext markup language) formats of the user interface to be tested and the target user interface.
In some embodiments, the obtaining of the layout file in the HTML format of the user interface to be tested is specifically to obtain the layout file in the HTML format of the relevant user interface from the installation package of the hybrid application to be tested and the installation package of the target hybrid application.
As an example, the installation packages of two hybrid applications are first taken as input. And decoding the application installation package by using a tool, and extracting an HTML layout file from the decoded folder, wherein the decoded installation package comprises the contents of an assets directory, a smali decompiled code, a resource file directory and the like. The HTML layout file is located under the www directory under the decoded asset directory assets.
S120: extracting basic features representing the visual effect of the user interface from the layout file, wherein the basic features representing the visual effect of the user interface comprise: element type, element position, and element size.
In some embodiments, the method specifically comprises: performing DOM Tree analysis calculation on the layout file in the HTML format to obtain the element type of each leaf node; and acquiring the element position and the element size of each leaf node according to the attribute tag of each leaf node of the DOM Tree. Further, still include: storing the basic characteristics of the leaf nodes representing the visual effect of the user interface as a data element, wherein one leaf node corresponds to one data element; a plurality of data elements are grouped into a data set.
In some embodiments, a virtual browser test tool may be utilized to perform DOMTree parsing calculations on layout files in HTML format.
As an example, two HTML layout files are taken as input. The WebView rendering process is simulated by using a tool, the HTML layout file is analyzed, and basic features capable of representing the visual effect of the user interface are analyzed and extracted. For example: element type c.t, element position c.p and element size c.s for an element c in the user interface (c.t, c.p, c.s). After the basic features of all the elements are extracted, outputting the basic feature set F of the user interface to be tested and the basic feature set F of the target user interface { c ═ c1,c2,…,ci,…}。
S130: and calculating the similarity between the user interface to be detected and the target user interface according to the basic characteristics representing the visual effect of the user interface.
As an example, the input is a basic feature set file of a pair of Hybrid application user interfaces to be tested
Calculating the similarity S ═ S (F) of the pair of user interfaces to be tested1,F2) And outputs a similarity value.
As shown in fig. 2, the specific similarity calculation method is as follows:
s10: combining all elements of the user interface to be tested and the target user interface pairwise to form a plurality of element combinations, and calculating the similarity of the element combinations one by one, wherein the calculating the similarity of the element combinations comprises the following steps: calculating element type similarity, calculating element position similarity and calculating element size similarity.
S20: and calculating the weighted similarity of the element combination one by utilizing the element area ratio.
S30: and selecting the maximum value of the element combination weighting similarity as the similarity between the user interface to be tested and the target user interface.
Calculating similarity of element combinations specifically, element type similarity (s.t):
the visual effects of different tag renderings of the HTML layout file are different, so that the element type of the HTML layout file defining the user interface is defined according to different tag types. The HTML layout file defines different types by distinguishing elements of different functions and different expressions, elements of similar functions and similar expressions are defined as one type and different attributes are set to distinguish characteristics. So for elements that behave and function similarly, they can be considered similar, i.e., the same type is similar. For different types, they may be considered dissimilar, i.e. different types of elements are dissimilar, due to the large difference in both function and performance. Meanwhile, some different tags in the HTML layout file may have similar visual effects. Therefore, a plurality of large classes representing the types of the user interface elements are defined in a unified mode, similar different label classes are classified into different large classes respectively, comparison is carried out through the large classes, and the type similarity is calculated. Two elements c1,c2If the types are the same, s.t is 1; when the element type is different, s.t is 0.
Element position similarity (s.p):
two elements c1,c2The position proximity is defined as the similarity of the element positions, which is expressed as the distance of Euclidean distance in the user interface. The closer the positions of the elements in the two user interfaces are, the higher the similarity degree of the elements looks, and the farther the positions are, the lower the similarity degree. The similarity is highest when the positions coincide and lowest when the positions are farthest (diagonal positions). According to this rule, take element c1And c2The upper left corner of the area occupied in the user interface represents its position coordinates. The position similarity is calculated by using the Euclidean distance between two position points in the range of the diagonal line of the whole screen, and weighting is carried out so that the change trend of the position similarity is closer to 1 when the distance is shorter, the similarity is sharply reduced along with the distance is longer, and the position similarity is changed into 0 when the length of the diagonal line is longer.
Element size similarity (s.s):
two elements c1、c2Size similarity s.s of (1) is represented by width similarity s.swWith high degree of similarity s.shDegree of two partsAnd defining the difference between the width of the two elements and the ratio of the width of the respective interfaces as the width similarity, and the difference between the height of the two elements and the ratio of the height of the respective interfaces as the height similarity. The width and height reflect the size of the space occupied by the application. The relationship between visual similarity and width and height should be such that the closer the width the more similar the visual similarity, the closer the height the more similar the visual similarity. I.e., consistent width or height, with the highest similarity, increasing width or height differences and lower similarity.
Then, two elements c are calculated1、c2S (s.t, s.p, s.s).
Then, calculating the weighted similarity formula of each pair of elements in the two feature sets to be detected by using the area ratio of each pair of elements as follows:
wherein c is1.swIs an element c1Width of the element(s), likewise, c1.shIs an element c1Of (c) is measured.
Mapping f to elements in the first (target UI) element set
To elements in the second (UI under test) set
Pairwise matching/combining relationship.
The maximum value of the element combination weighted similarity can be selected as the similarity between the user interface to be tested and the target user interface by using the optimal matching algorithm
And finally, calculating the similarity between the user interface to be detected and the target user interface and outputting the similarity so as to be used by related safety analysis personnel.
In summary, in the following, an example of the present invention is that, firstly, an Apktool decompiling tool may be used to unpack an Android application installation package to be tested, and an HTML layout file corresponding to a specified UI is extracted from the package; and then, analyzing and calculating the HTML layout file through a virtual browser test tool Phantomjs, extracting the type of the basic element in the UI from the HTML layout file, and calculating the position, width and height of the element according to the attribute tag of each leaf node. Saving the element type, the element position and the element size into a visual characteristic data element, and saving the data elements of all the elements in the UI into a visual characteristic data set; and then calculating element type similarity, element position similarity and element size similarity between each data element by using a similarity algorithm. Finally, the best combination between the data sets is selected so that the similarity of the two data sets takes the maximum value, and the value is defined as the similarity of the two UIs.
With reference to fig. 3, the specific implementation flow is as follows:
step 0: inputting two applications to be tested including the UI to be tested and the layout file name of the UI to be tested.
Step 1: and extracting and analyzing the layout file.
Unpacking the APK file of the application installation package:
the layout files are all stored in the APK files of the application installation package, and the APK files are packaged files subjected to special encoding, so that the layout files can be obtained only by unpacking. The method uses an open source tool Apktool to decode the application installation package and extracts the layout file. Example code for this section is as follows:
| apktool d<file.apk><dir> |
step 2: analyzing the HTML layout file and extracting visual feature information:
this step, the HTML layout file is traversed by using the virtual simulator test tool Phantomjs. And (4) starting parsing from the root node of the layout file DOM Tree, respectively extracting the element types represented by the leaf nodes in the layout file, and taking the element positions and the element sizes as visual characteristic information. The specific definition is as follows:
the definition element is denoted as E.
The element type is e.type.
The element position is (x, y).
Element size is measured by width and height, labeled e.width and e.height, respectively. In order to ensure that the position value is as positive as possible, the lower left corner of the screen is defined as a coordinate origin, namely (0, 0), the x axis is a horizontal axis, and the right angle is positive; the y-axis is the vertical longitudinal axis and is positive upward.
According to the above definition, the single-element visual characteristic information element in the present invention is:
the following is a simple example of an element visual characteristic information element:
the above feature information elements are output in JSON format, the content corresponding to "type" indicates the element type, the content corresponding to "size" indicates the element size, the first number in parentheses indicates the width, and the second number indicates the height. The content corresponding to "position" represents element position information, the first value represents the x-axis position, and the second represents the y-axis position.
And extracting the characteristic information of all elements by completely traversing the DOM Tree of the layout file to form a visual characteristic information set of the layout file, wherein the visual characteristic information set of each layout file comprises visual characteristic information elements of a plurality of elements.
According to the above definition, the basic visual feature information set in the method is:
the following is a simple set of basic visual characteristic information for a layout file:
the feature information set is also defined in the JSON format and corresponds to visual feature elements of a plurality of elements in a user interface.
In the method, the JSON format is adopted to define and store the visual characteristic information, but the invention is not limited to this, and only an expression method capable of expressing the correlation is used.
Step 3: calculating the similarity of the visual basic feature information sets of the two user interfaces to be tested:
and calculating the similarity between each pair of elements in the visual basic feature information sets of the two user interfaces to be tested.
Each pair of elements c1、c2The similarity calculation is divided into a type similarity part, a position similarity part and a size similarity part.
Element type similarity calculation:
if Type (c)1)＝Type(c2) Then the element type similarity s.t is 1.
If Type (c)1)！＝Type(c2) Then the element type similarity s.t is 0.
Calculating the similarity of element positions:
the element position similarity calculation is shown in the following formula:
where s.p denotes element position similarity, c1、c2Two elements of position information are denoted c1.p,c2.p，||c1.p-c2P | | | denotes the Euclidean distance of two element positions, | | lmax|' represents the maximum (screen diagonal) euclidean distance in the UI.
Calculating the element size similarity:
the element size similarity is divided into a width similarity and a height similarity, and finally integrated into an element size similarity. The similarity calculation formula is as follows:
s.s thereinwElement Width similarity, s.shElement height similarity, s.s element size similarity. Wherein c is1.sw， c2.swAre respectively an element c1、c2Width of (c) same as1.sh，c2.shAre respectively an element c1、c2Of (c) is measured.
And (3) calculating the total similarity of the two elements:
the two-element similarity s is synthesized from the element type similarity s.t, the element position similarity s.p, and the element size similarity s.s.
And (3) carrying out weighted calculation on the similarity by using the area to obtain the weighted similarity:
in the above formula sWIs the similarity between the weighted elements.
After the weighted similarity is calculated for each pair of elements in the visual basic feature information sets of the two user interfaces to be tested, an inter-element similarity matrix is obtained:
wherein
Representing the ith element in the first set of visual characteristic information,
representing the jth element in the second set of visual characteristic information.
Representing the weighted similarity of the ith element in the first set to the jth element in the second set.
Defining a mapping fx(x＝1,2,3,…)For elements in the first set
To elements in the second set
Pairwise matching/combining relationship.
Finding the best matching mapping f using K-M algorithm (Hungarian algorithm)maxSo that
Defining a similarity formula of two visual basic feature sets of the user interface to be tested based on the mapping, wherein the similarity formula comprises the following steps:
according to the detection method of the Hybrid mobile application user interface, the visual characteristics are extracted from the acquired HTML layout files of the user interface to be detected and the target user interface, and the similarity between the user interface to be detected and the target user interface is calculated according to the characteristics, so that the visual characteristics are extracted and analyzed statically, the method can be used for analyzing the similarity of the large-scale Hybrid mobile application user interface, has good expandability, and has good efficiency gain compared with the traditional dynamic analysis method.
FIG. 4 is a block diagram of a detection system for a Hybrid mobile application user interface in accordance with one embodiment of the present invention.
As shown in fig. 4, a detection system 400 for a Hybrid mobile application user interface according to an embodiment of the present invention includes: a layout acquisition module 410, a visual feature extraction module 420, and a user interface similarity calculation module 430.
The layout obtaining module 410 is configured to obtain layout files in HTML format of the user interface to be tested and the target user interface. The visual feature extraction module 420 is configured to extract basic features representing a visual effect of a user interface from the layout file, where the basic features representing the visual effect of the user interface include: element type, element position, and element size. The user interface similarity calculation module 430 is configured to calculate a similarity between the user interface to be tested and the target user interface according to the basic features representing the visual effect of the user interface.
According to the detection system of the Hybrid mobile application user interface, the visual characteristics are extracted from the acquired HTML layout files of the user interface to be detected and the target user interface, and the similarity between the user interface to be detected and the target user interface is calculated according to the characteristics, so that the visual characteristics are extracted and analyzed statically, the method can be used for analyzing the similarity of the large-scale Hybrid mobile application user interface, has good expandability, and has good efficiency gain compared with the traditional dynamic analysis method.
Further, the specific similarity calculation manner of the user interface similarity calculation module 430 is as follows: s10: combining all elements of the user interface to be tested and the target user interface pairwise to form a plurality of element combinations, and calculating the similarity of the element combinations one by one, wherein the calculating the similarity of the element combinations comprises the following steps: calculating element type similarity, calculating element position similarity and calculating element size similarity. S20: and calculating the weighted similarity of the element combination one by utilizing the element area ratio. S30: and selecting the maximum value of the element combination weighting similarity as the similarity between the user interface to be tested and the target user interface.
Further, the visual feature extraction module 420 specifically includes: and performing DOM Tree analysis calculation on the layout file in the HTML format to obtain the element type of each leaf node. And acquiring the element position and the element size of each leaf node according to the attribute tag of each leaf node of the DOM Tree.
Further, still include: storing the basic characteristics of the leaf nodes representing the visual effect of the user interface as a data element, wherein one leaf node corresponds to one data element; a plurality of data elements are grouped into a data set.
Further, the layout obtaining module is specifically configured to obtain the layout file in the HTML format of the relevant user interface from the installation package of the hybrid application to be tested and the installation package of the target hybrid application.
It should be noted that a specific implementation manner of the detection system for the Hybrid mobile application user interface in the embodiment of the present invention is similar to a specific implementation manner of the detection method for the Hybrid mobile application user interface in the embodiment of the present invention, and please refer to the description of the detection method part of the Hybrid mobile application user interface specifically, which is not described herein again in order to reduce redundancy.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In the present invention, unless otherwise expressly stated or limited, the terms "mounted," "connected," "secured," and the like are to be construed broadly and can, for example, be fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or they may be connected internally or in any other suitable relationship, unless expressly stated otherwise. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (8)
1. A method for detecting a user interface of a Hybrid mobile application is characterized by comprising the following steps:
acquiring HTML-format layout files of a user interface to be tested and a target user interface;
extracting basic features representing the visual effect of the user interface from the layout file, wherein the basic features representing the visual effect of the user interface comprise: element type, element position, and element size;
calculating the similarity between the user interface to be detected and the target user interface according to the basic characteristics representing the visual effect of the user interface;
the specific similarity calculation mode is as follows:
s10: combining all elements of the user interface to be tested and the target user interface pairwise to form a plurality of element combinations, and calculating the similarity of the element combinations one by one, wherein the calculating the similarity of the element combinations comprises the following steps: calculating element type similarity, calculating element position similarity and calculating element size similarity;
wherein, the element type similarity calculation mode is as follows: if Type (c)1)＝Type(c2) If the element type similarity is 1; if Type (c)1)！＝Type(c2) Then the element Type similarity is 0, wherein Type (c)1) Is an element c1Type of (c)2) Is an element c2Type of (d);
calculating the element position similarity by the following formula:
wherein s.p is the element position similarity, c1P is said element c1Position information of c2P is said element c2Position information, | c1.p-c2P is the element c1And said element c2Of Euclidean distance, | lmaxII denotes the Euclidean distance of the largest screen diagonal in the UI;
calculating the element size similarity by the following formula:
wherein, s.swElement Width similarity, s.shIs element height similarity, s.s is the element size similarity, c1.sw，c2.swAre respectively the element c1、c2Width of c1.sh，c2.shAre respectively the element c1、c2The height of (d);
obtaining the similarity of the element combination through the following formula:
wherein s is the similarity of the element combination, s.t is the element type similarity, s.p is the element position similarity, s.s is the element size similarity;
s20: calculating the weighted similarity of the element combination one by utilizing the element area ratio;
calculating the weighted similarity of the combination of elements by the following formula:
wherein s isWF1 is the first visual characteristic information set of the user interface under test for the weighted similarity of the element combination,
the ith element in the first visual characteristic information set;
s30: and selecting the maximum value of the element combination weighting similarity as the similarity between the user interface to be tested and the target user interface.
2. The method for detecting a user interface of a Hybrid mobile application according to claim 1, wherein the extracting of the basic features representing the visual effect of the user interface from the layout file specifically comprises:
performing DOM Tree analysis calculation on the layout file in the HTML format to obtain the element type of each leaf node;
and acquiring the element position and the element size of each leaf node according to the attribute tag of each leaf node of the DOM Tree.
3. The method for detecting the user interface of the Hybrid mobile application as recited in claim 2, further comprising:
storing the base features of leaf nodes representing the visual effect of the user interface as a data element, wherein a leaf node corresponds to a data element;
a plurality of data elements are grouped into a data set.
4. The method for detecting the user interface of the Hybrid mobile application according to claim 1, wherein the obtaining of the layout file in the HTML format of the user interface to be detected is specifically obtaining the layout file in the HTML format of the relevant user interface from the installation package of the Hybrid application to be detected and the installation package of the target Hybrid application.
5. A Hybrid mobile application user interface detection system is characterized by comprising:
the layout acquisition module is used for acquiring layout files in HTML (hypertext markup language) formats of the user interface to be tested and the target user interface;
a visual feature extraction module, configured to extract basic features representing a user interface visual effect from the layout file, where the basic features representing the user interface visual effect include: element type, element position, and element size;
the user interface similarity calculation module is used for calculating the similarity between the user interface to be detected and the target user interface according to the basic characteristics representing the visual effect of the user interface;
the specific similarity calculation mode of the user interface similarity calculation module is as follows:
s10: combining all elements of the user interface to be tested and the target user interface pairwise to form a plurality of element combinations, and calculating the similarity of the element combinations one by one, wherein the calculating the similarity of the element combinations comprises the following steps: calculating element type similarity, calculating element position similarity and calculating element size similarity;
wherein, the element type similarity calculation mode is as follows: if Type (c)1)＝Type(c2) If the element type similarity is 1; if Type (c)1)！＝Type(c2) Then the element Type similarity is 0, wherein Type (c)1) Is an element c1Type of (c)2) Is an element c2Type of (d);
calculating the element position similarity by the following formula:
wherein s.p is the element position similarity, c1P is said element c1Position information of c2P is said element c2Position information, | c1.p-c2P is the element c1And said element c2Of Euclidean distance, | lmaxII denotes the Euclidean distance of the largest screen diagonal in the UI;
calculating the element size similarity by the following formula:
wherein, s.swElement Width similarity, s.shIs element height similarity, s.s is the element size similarity, c1.sw，c2.swAre respectively the element c1、c2Width of c1.sh，c2.shAre respectively the element c1、c2The height of (d);
obtaining the similarity of the element combination through the following formula:
wherein s is the similarity of the element combination, s.t is the element type similarity, s.p is the element position similarity, s.s is the element size similarity; s20: calculating the weighted similarity of the element combination one by utilizing the element area ratio;
calculating the weighted similarity of the combination of elements by the following formula:
wherein s isWF1 is the first visual characteristic information set of the user interface under test for the weighted similarity of the element combination,
the ith element in the first visual characteristic information set;
s30: and selecting the maximum value of the element combination weighting similarity as the similarity between the user interface to be tested and the target user interface.
6. The system for detecting a Hybrid mobile application user interface of claim 5, wherein the visual feature extraction module specifically comprises:
performing DOM Tree analysis calculation on the layout file in the HTML format to obtain the element type of each leaf node;
and acquiring the element position and the element size of each leaf node according to the attribute tag of each leaf node of the DOM Tree.
7. The system for detecting a Hybrid mobile application user interface of claim 6, further comprising:
storing the base features of leaf nodes representing the visual effect of the user interface as a data element, wherein a leaf node corresponds to a data element;
a plurality of data elements are grouped into a data set.
8. The system for detecting the user interface of the Hybrid mobile application according to claim 5, wherein the layout obtaining module is specifically configured to obtain the layout files in the HTML format of the relevant user interface from the installation package of the Hybrid application to be detected and the installation package of the target Hybrid application.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710385933.0A CN107315680B (en) | 2017-05-26 | 2017-05-26 | Method and system for detecting Hybrid mobile application user interface |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710385933.0A CN107315680B (en) | 2017-05-26 | 2017-05-26 | Method and system for detecting Hybrid mobile application user interface |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107315680A CN107315680A (en) | 2017-11-03 |
| CN107315680B true CN107315680B (en) | 2020-11-03 |
Family
ID=60183877
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710385933.0A Active CN107315680B (en) | 2017-05-26 | 2017-05-26 | Method and system for detecting Hybrid mobile application user interface |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107315680B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108898013B (en) * | 2018-06-14 | 2021-09-28 | 南京大学 | Android application interface similarity comparison method based on layout division feature vectors |
| CN109032722B (en) * | 2018-06-27 | 2021-09-28 | 广州视源电子科技股份有限公司 | Method, device, equipment and medium for updating display effect of UI component |
| CN109784369A (en) * | 2018-12-14 | 2019-05-21 | 深圳壹账通智能科技有限公司 | H5 page test method and device, electronic equipment, storage medium |
| CN109992518A (en) * | 2019-04-10 | 2019-07-09 | 禄鹏 | Detection method, device, electronic equipment and the storage medium at the interface UI |
| CN111273905B (en) * | 2020-01-17 | 2023-04-18 | 南京大学 | Application retrieval method and device based on interface sketch |
| CN112487437B (en) * | 2020-11-17 | 2023-04-18 | 北京航空航天大学 | Android counterfeit application detection method and device based on user interface characteristics |
| CN114185544A (en) * | 2021-12-06 | 2022-03-15 | 北京达佳互联信息技术有限公司 | Page data processing method and device, electronic equipment and storage medium |
| CN116795346B (en) * | 2023-06-26 | 2024-03-15 | 成都中科合迅科技有限公司 | Component interface drawing method and system based on visual contrast |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101894134A (en) * | 2010-06-21 | 2010-11-24 | 南京邮电大学 | Spatial layout-based fishing webpage detection and implementation method |
| CN104778409A (en) * | 2015-04-16 | 2015-07-15 | 电子科技大学 | Method and device for detecting similarities of Android application software |
| CN106445513A (en) * | 2016-09-12 | 2017-02-22 | 中山大学 | Similarity calculation method based on mobile application interface element |
-
2017
- 2017-05-26 CN CN201710385933.0A patent/CN107315680B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101894134A (en) * | 2010-06-21 | 2010-11-24 | 南京邮电大学 | Spatial layout-based fishing webpage detection and implementation method |
| CN104778409A (en) * | 2015-04-16 | 2015-07-15 | 电子科技大学 | Method and device for detecting similarities of Android application software |
| CN106445513A (en) * | 2016-09-12 | 2017-02-22 | 中山大学 | Similarity calculation method based on mobile application interface element |
Non-Patent Citations (1)
| Title |
|---|
| DroidEagle:seamless detection of visually similar android apps;Mingshen Sun;《WiSec "15：Proceedings of the 8th ACM Conference on Security & Privacy in Wireless and Mobile Networks》;20150630;第3页2段,第4页第1-3段，第6-10段，第6页第3段，图2,图4 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107315680A (en) | 2017-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107315680B (en) | Method and system for detecting Hybrid mobile application user interface | |
| KR102275413B1 (en) | Detecting and extracting image document components to create flow document | |
| Choudhary et al. | Crosscheck: Combining crawling and differencing to better detect cross-browser incompatibilities in web applications | |
| US20210295114A1 (en) | Method and apparatus for extracting structured data from image, and device | |
| CN111767228B (en) | Interface testing method, device, equipment and medium based on artificial intelligence | |
| WO2021017735A1 (en) | Smart contract formal verification method, electronic apparatus and storage medium | |
| US11321559B2 (en) | Document structure identification using post-processing error correction | |
| US9384389B1 (en) | Detecting errors in recognized text | |
| US9563611B2 (en) | Merging web page style addresses | |
| US10372980B2 (en) | Electronic form identification using spatial information | |
| US10528649B2 (en) | Recognizing unseen fonts based on visual similarity | |
| US8897598B1 (en) | Mosaicing documents for translation using video streams | |
| US9535888B2 (en) | System, method, software arrangement and computer-accessible medium for a generator that automatically identifies regions of interest in electronic documents for transcoding | |
| EP3891625A1 (en) | Method of detecting user interface layout issues for web applications | |
| Delamaro et al. | Using concepts of content‐based image retrieval to implement graphical testing oracles | |
| US11556610B2 (en) | Content alignment | |
| CN107294918B (en) | Phishing webpage detection method and device | |
| CN113221855B (en) | Small target detection method and system based on scale sensitive loss and feature fusion | |
| US9811505B2 (en) | Techniques to provide processing enhancements for a text editor in a computing environment | |
| CN109783805B (en) | Network community user identification method and device and readable storage medium | |
| Saar et al. | Browserbite: cross‐browser testing via image processing | |
| Xu et al. | Identifying semantic blocks in Web pages using Gestalt laws of grouping | |
| KR102490486B1 (en) | Method for generating table information based on the image | |
| CN117093386B (en) | Page screenshot method, device, computer equipment and storage medium | |
| TW201523421A (en) | Determining images of article for extraction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |