CN115168182A - Automatic positioning and repairing method and device for user interface display defects - Google Patents
Automatic positioning and repairing method and device for user interface display defects Download PDFInfo
- Publication number
- CN115168182A CN115168182A CN202210664508.6A CN202210664508A CN115168182A CN 115168182 A CN115168182 A CN 115168182A CN 202210664508 A CN202210664508 A CN 202210664508A CN 115168182 A CN115168182 A CN 115168182A
- Authority
- CN
- China
- Prior art keywords
- application program
- component
- user interface
- template
- source code
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000007547 defect Effects 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 31
- 230000008439 repair process Effects 0.000 claims abstract description 19
- 238000001514 detection method Methods 0.000 claims abstract description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 claims description 2
- 230000003628 erosive effect Effects 0.000 claims description 2
- 230000003044 adaptive effect Effects 0.000 claims 5
- 238000012360 testing method Methods 0.000 abstract description 6
- 230000002950 deficient Effects 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 3
- 239000012634 fragment Substances 0.000 description 3
- 238000010998 test method Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000013522 software testing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration using local operators
- G06T5/30—Erosion or dilatation, e.g. thinning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20036—Morphological image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method and a device for automatically positioning and repairing a display defect of a user interface. The repairing method comprises the following steps: 1) Detecting a screenshot of a user interface of a target application program by using a computer vision technology, and acquiring a display defect area on the screenshot according to a detection result; 2) Denoising the image of the display defect area by utilizing binarization and extracting a coordinate of the display defect area; 3) Traversing the view layout file of the target application program, and positioning a corresponding component id in the view layout file according to the coordinate of the display defect area; 4) And traversing the source code of the target application program according to each component id, acquiring the source code corresponding to each component id and repairing the source code. The method and the device can help developers to quickly locate and repair the display defects of the user interface of the application program, save repair time and reduce test cost.
Description
Technical Field
The invention belongs to the technical field of computers, relates to a software testing technology, in particular to a UI (user interface) test, and is used for automatically positioning and repairing display defects of a user interface of an application program. The invention constructs a rich automatic positioning and repairing strategy for the display defects of the user interface of the application program. Based on the strategy, the method automatically repairs the display defects of the user interface of the application program by using the information retrieval method, helps developers to quickly position and repair the display defects of the user interface of the application program, and saves the repair time.
Background
With the rapid development of mobile devices, the variety and style of mobile applications in the world today are becoming more and more abundant. Given that there are 300 million Android applications available for different tasks such as reading, shopping, banking and chatting, cell phones and applications have now become an integral part of our daily lives. The importance of mobile applications necessitates thorough testing by the development team to ensure the quality of the mobile application. In order to ensure the quality of the application program, the common mobile application program user interface tests mainly include a model-based automatic test method, a probability-based automatic test method and a deep learning-based automatic test method.
Research has shown that repairing a known application user interface display defect takes approximately two days, which is both time consuming and labor intensive. Developers must understand the problem, reproduce the problem, locate its root cause in the source code, and speculate on strategies that may fix the problem. The difficulty of all of these tasks is exacerbated by the software complexity, code inconsistencies, and amount of code of legacy code, particularly in highly fragmented front-end code. In particular, by presenting an average of 8 XML files, a single user display page may be generated, which makes it difficult to determine suspect XML files for defective components on the page. Furthermore, the variety of components involved in application user interface display issues and the variety of design styles in which application user display pages are designed make the situation worse. Although there are many automatic problem-repairing methods, search methods, repair patterns, and semantic information based on genetic algorithms, none of these methods can repair the display problem of a User Interface (UI).
In summary, the existing methods for automatically repairing the defects mainly aim at the defects which can cause the application program to crash. There is no suitable solution for the user interface display problem.
Disclosure of Invention
In order to solve the above problems, an object of the present invention is to provide a method and an apparatus for automatically locating and repairing a display defect of an application user interface. According to the method, a rich automatic positioning and repairing strategy for the display defects of the user interface of the application program is constructed, and the display defects of the user interface of the application program are automatically repaired by using an information retrieval method based on the strategy, so that developers are helped to quickly position and repair the display defects of the user interface of the application program, the repairing time is saved, and the testing cost is reduced.
The technical scheme of the invention is as follows:
an automatic positioning and repairing method and device for application program user interface display defects comprises the following steps:
1) Firstly, detecting a screenshot of a user interface by using a computer vision technology (CNN) to determine whether a user interface display defect exists on the screenshot;
2) Reversely acquiring information of the last layer of the convolution layer according to the detection result to complete image positioning of the display defect of the user interface;
3) And according to the image positioning result, performing noise reduction on the positioned image with the display defect by utilizing binarization, and extracting the coordinates of the defect area.
4) Traversing a view layout file of an application program, and positioning a corresponding component id according to the coordinates of the defect area;
5) Traversing an application program source code according to the component id and acquiring a source code corresponding to the component id;
6) And selecting a proper repair scheme to complete the repair of the defect code by using a search-based method.
Further, defect detection is performed by:
1) Analyzing the user interface screenshot of the application program by using a convolutional neural network;
2) Completing the detection of the application user interface screenshot and completing the classification of the defect category.
Further, the defect positioning of the user interface screenshot of the application program is carried out through the following steps:
1) Acquiring the coordinates of the defect area of the screenshot of the application program by reversely inputting the convolutional neural network;
2) Traversing an interface view layout file of an application program according to the defect coordinates, and acquiring coordinates of each component;
3) And positioning the defective component in the code by taking the intersection of the defective area coordinate and the coordinates of each component as a judgment basis.
Further, defect repair is performed by:
1) Obtaining the id of the component through the obtained defective component;
2) Summarizing to obtain a repair template;
a first template: setting the self-adaptation of the height of the component;
and (2) template II: setting text size self-adaptation;
and (3) template III: setting boundary self-adaptation;
and (4) template IV: setting layout self-adaptation;
and a fifth template: setting font size self-adaptation;
and a sixth template: setting a coding format;
3) Matching a proper repair template by utilizing a search-based method according to the component id;
4) And completing the repair of the assembly.
A storage medium having a computer program stored therein, wherein the computer program is arranged to perform the above-mentioned method when executed.
An electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer to perform the method as described above.
Compared with the prior art, the invention has the following beneficial effects:
the invention detects the user interface display defects through automatic extraction and constructs a rich automatic positioning and repairing strategy of the application program user interface display defects. Based on the strategy, the method for information retrieval is used for automatically repairing the display defects of the user interface of the application program, so that developers are helped to quickly locate and repair the display defects of the user interface of the application program, the repairing time is saved, and the testing cost is reduced.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The method of the invention has the flow as shown in figure 1, and comprises the following specific steps:
step 1, firstly, simulating the operation of clicking and the like of a user to automatically run an application program to obtain screenshot information and view layout file information of the application program.
Step 2, inputting the screenshot of the application program into a convolutional neural network (reference: zhe Liu, chunyang Chen, junjie Wang, yuekai Huang, jun Hu, and Qing Wang.2020.Owl eyes: mapping UI display visual understanding. Proceedings of the 35th IEEE/ACM International Conference on Automated Software engineering. Association for Computing Machinery, new York, NY, USA,398-409.DOI https:// doi. Org/10.1145/3324884.3416547), completing the detection of the display defect of the user interface, inputting the detection result reversely into the neural network, visualizing the characteristics of the last layer of the convolutional layer (i.e. the defect region) and obtaining the defect region of the convolutional layer.
And 3, denoising the visualized defect area by adopting a computer vision processing method. Firstly, the thermodynamic diagram of the defect area is binarized to obtain a gray level image, and the gray level image is binarized, namely, the image part with the threshold value larger than 233 is set to be white and is regarded as a potential vehicle area, and the rest part is set to be black. The erosion in mathematical morphology is then used to eliminate noise to reduce white areas and facilitate accurate identification of defect areas, completing the precise location.
And 4, completing the defect positioning of the code level according to the picture coordinate information. To further locate the defective fragments from the source code, we need to obtain the component coordinates in the run-time xml file. The run-time xml file is a detailed view hierarchy of the UI page obtained during the running of the application, including information such as id, coordinates, etc. of all components appearing on the UI page. In the running process of App, the hierarchy information and the corresponding screenshots of all spaces on the current application screen are obtained by running the uiautomator dump proposition of Android Debug Bridge (ADB). Each screenshot (UI page) corresponds to a run-time XML, and the coordinates of UI elements on the screenshot correspond to the coordinates of components in the run-time XML one by one.
And 5, traversing and searching the source code according to the component coordinates acquired in the step 4 to retrieve the code segment corresponding to the id. For successfully matched code fragments, we then check if the component type can also successfully match the retrieved code fragment. For some components that do not have a component id, we use the textual description of the component to retrieve.
And 6, according to the defect code segment obtained in the step 5, using the positioned bug source code as input for defect repair, matching error codes by using a predefined template, and repairing problems. We define six templates. The method will check if the error code matches any of the six templates and generate a corresponding patch.
A first template: setting the self-adaptation of the height of the component;
and (2) template II: setting text size self-adaptation;
and (3) template III: setting boundary self-adaptation;
and (4) template IV: setting layout self-adaptation;
and (5) template five: setting font size self-adaptation;
and (6) template six: setting an encoding format.
The above examples are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications can be made without departing from the spirit and principles of the invention, and are intended to be included within the scope of the invention.
Claims (7)
1. An automatic positioning and repairing method for a user interface display defect comprises the following steps:
1) Detecting a screenshot of a user interface of a target application program by using a computer vision technology, and acquiring a display defect area on the screenshot according to a detection result;
2) Denoising the image of the display defect area by utilizing binarization and extracting a coordinate of the display defect area;
3) Traversing the view layout file of the target application program, and positioning a corresponding component id in the view layout file according to the coordinate of the display defect area;
4) And traversing the source code of the target application program according to each component id, acquiring the source code corresponding to each component id and repairing the source code.
2. The method according to claim 1, wherein a corresponding repair template is set for the source code corresponding to each component id of the view layout file; and searching the matched repair template according to the component id to repair the source code corresponding to the component id.
3. The method of claim 2, wherein modifying the template comprises: a component height adaptive template, a text size adaptive template, a boundary adaptive template, a layout adaptive template, a font size adaptive template, and an encoding format template.
4. The method of claim 1, 2 or 3, wherein the user screenshot and the view layout file of the target application program are obtained by running the target application program.
5. The method according to claim 1, 2 or 3, characterized in that in step 2), the thermodynamic diagram showing the defect region is binarized, and a grayscale image is binarized; the noise is then removed using erosion in mathematical morphology to reduce the white areas.
6. An electronic apparatus, comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program comprising instructions for carrying out the steps of the method according to any one of claims 1 to 5.
7. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210664508.6A CN115168182A (en) | 2022-06-13 | 2022-06-13 | Automatic positioning and repairing method and device for user interface display defects |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210664508.6A CN115168182A (en) | 2022-06-13 | 2022-06-13 | Automatic positioning and repairing method and device for user interface display defects |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115168182A true CN115168182A (en) | 2022-10-11 |
Family
ID=83485508
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210664508.6A Pending CN115168182A (en) | 2022-06-13 | 2022-06-13 | Automatic positioning and repairing method and device for user interface display defects |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115168182A (en) |
-
2022
- 2022-06-13 CN CN202210664508.6A patent/CN115168182A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10585780B2 (en) | Enhancing software development using bug data | |
| CN109144882B (en) | Software fault positioning method and device based on program invariants | |
| US10191889B2 (en) | Systems, apparatuses and methods for generating a user interface by performing computer vision and optical character recognition on a graphical representation | |
| CN107145445B (en) | Automatic analysis method and system for error log of software automation test | |
| CN111767228A (en) | Interface testing method, device, equipment and medium based on artificial intelligence | |
| CN110309073B (en) | Method, system and terminal for automatically detecting user interface errors of mobile application program | |
| JP7404839B2 (en) | Identification of software program defect location | |
| CN107862327B (en) | Security defect identification system and method based on multiple features | |
| CN112148595A (en) | Software change level defect prediction method for removing repeated change | |
| US9563541B2 (en) | Software defect detection identifying location of diverging paths | |
| Sun et al. | Ui components recognition system based on image understanding | |
| CN112015634A (en) | Page structure information generation method and device and electronic equipment | |
| CN113032279A (en) | Web application testing and repairing method based on semantic path search | |
| CN115168182A (en) | Automatic positioning and repairing method and device for user interface display defects | |
| US11960862B2 (en) | Source code correction assistance apparatus and source code correction assistance method | |
| CN113778852B (en) | Code analysis method based on regular expression | |
| WO2023275887A1 (en) | System and method for automated software testing | |
| CN114638597A (en) | Intelligent government affair handling application system, method, terminal and medium | |
| CN110633204B (en) | Program defect detection method and device | |
| CN111008140B (en) | Cross-platform UI (user interface) automatic testing method and device | |
| US20130080137A1 (en) | Conversion method and system | |
| Watanabe et al. | Layout cross-platform and cross-browser incompatibilities detection using classification of DOM elements | |
| CN113284141A (en) | Model determination method, device and equipment for defect detection | |
| CN111797007A (en) | Automatic detection and positioning method for application program user interface defects and electronic device | |
| CN113515588A (en) | Form data detection method, computer device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |