CN110515835B

CN110515835B - Test method based on machine vision and DOM tree structure

Info

Publication number: CN110515835B
Application number: CN201910695205.9A
Authority: CN
Inventors: 刘春刚
Original assignee: Shanghai Yunda Information Technology Co ltd
Current assignee: Shanghai Yunda Information Technology Co ltd
Priority date: 2019-07-30
Filing date: 2019-07-30
Publication date: 2023-05-23
Anticipated expiration: 2039-07-30
Also published as: CN110515835A

Abstract

The invention discloses a testing method based on machine vision and DOM tree structure, comprising the following steps: s100, extracting and selecting characteristics, and identifying characteristics of target elements in a test page; s200, classifying and generalizing elements with similar characteristics in the page according to the characteristics of the target elements to obtain generalized elements; s300, performing automatic test according to the generalization element and recording. The invention can greatly reduce the test cost and test time, can increase the robustness of the test scheme and the automatic generation of the test scheme, and the flow robot actively understands the user behavior. The invention can be widely used in web page testing links, flow robots and the like, after the invention is used, a tester can not need to further update a test script because of the update of a developer to a web page, and can operate all similar elements by recording once or one script. Meanwhile, the flow robot has the function of actively understanding the user behavior.

Description

Test method based on machine vision and DOM tree structure

Technical Field

The invention relates to a webpage testing technology, in particular to a testing method based on machine vision and DOM tree structures.

Background

The automation requirements of the current software, web pages and mobile terminal application development or testing are higher and higher, and the importance of the user behavior understanding is self-evident. Such as a flow-based robot or an automated testing field, for example, in the field of automated testing of web pages, there are already a lot of testing software that can record the operation of a tester and then can be continuously executed. However, the problem is that the steps in the testing process can be generalized, for example, north China 1, east China 2 and … …, which are equivalent to the testing process and are all required to be tested, so that a problem is easily raised that when a tester selects the north China 1 server, a testing platform or tool cannot understand the selection and generalize, and when the testing platform or tool executes the step, servers at other places can be randomly selected or all servers can be tested in parallel, so that the time for the tester to record or write scripts is saved, and the comprehensiveness of the test is increased.

In order to achieve the aim of generalization, the applicant uses two technologies of machine vision and DOM tree structure analysis to achieve the aim, wherein the machine vision is mainly used for carrying out similar matching on image features of icons and buttons in a recognition webpage, and the DOM tree analysis method is used for searching similar elements through positions of target elements in HTML or XML of the webpage and the structure of a DOM tree to generalize the elements.

At the same time, another less obvious benefit is increased robustness, i.e. fault tolerance, especially in the field of testing. When the test can be continued after the version iterating, for example, during the test process performed by the original test tool or test script, the north China 1 server is disconnected, the whole test is interrupted, the test time is wasted, meanwhile, unnecessary losses such as the reduction of the experience of a customer are possibly caused, and the modification by a tester or the writing or recording of the script from the beginning are inevitable. The technology in the invention can automatically select other servers, and log records, so that a tester is left to judge whether the situation is normal or wrong, thereby greatly improving the testing efficiency (such as the situation that the tester is not left aside in the middle of the night) and reducing the workload of the tester.

Disclosure of Invention

In view of the above-mentioned drawbacks of the prior art, the present invention provides a testing method based on machine vision and DOM tree structure, which has generalization property and can greatly improve the web page testing efficiency.

In order to achieve the above purpose, the invention provides a testing method based on machine vision and DOM tree structure, comprising the following steps:

s100, extracting and selecting characteristics, and identifying characteristics of target elements in a test page;

s200, classifying and generalizing elements with similar characteristics according to the characteristics of the target elements to obtain generalized elements;

s300, performing automatic test according to the generalization element and recording.

Preferably, in S100, two methods, that is, S110, a machine vision method, S120, and a DOM tree analysis method are included, and the two methods may be performed simultaneously, compared with each other, or may be used singly;

under the conditions that the use environment of the machine vision method is clear in webpage and the edges of the buttons of the icons are clear, the aim of generalization is achieved by identifying the approximate shape of a target element and finding similar icons or buttons in a certain area;

the DOM tree analysis method is performed through the structure of the DOM tree, and analysis obeys the following assumptions:

suppose 1. The hierarchy of generalizable like elements must be similar;

suppose 2. When homogeneous elements all find the nearest same parent element, non-homogeneous elements must not be this parent element. Under the premise of the two assumptions, the same kind of elements can be found, and the generalization purpose is achieved.

Preferably, the machine vision method comprises the following steps:

s111, selecting a target element, wherein the target element is an icon or a button, and identifying the characteristics of the target element, and if the target element is a square frame, identifying the area, the inclination angle, the length and the like of the square frame; the circle center and the radius of the circle are identified for the circle; identifying the focus, major axis, minor axis, etc. for an ellipse;

s112, identifying edges of all elements in the page;

s113, screening all elements according to the characteristics of the target elements to obtain graphic elements with characteristics similar to the characteristics of the target elements, and obtaining generalized elements.

The S111-S112 may be implemented by an existing OCR technology, or may be implemented by the following technologies:

s112.1, marginalizing a target page, firstly changing the image of the test page into a binary image, and better detecting the edges of icons or buttons because important features of the image can be enhanced, and then detecting the edges, wherein the edge detection formula is as follows:

wherein:

gx is a horizontal Gradient, gy is a vertical Gradient, edge_gradient (G) is an Edge Gradient, and Angle (θ) is an Edge inclination; all edges in the page can be detected through the step, but the edges of the characters are also included;

s112.2 selects the obtained edge according to the characteristics of the target element. If the target element is a rectangle square frame, knowing the length and width of the square frame, and the length and width of the square frame are horizontal and vertical, filtering the edges with abnormal non-horizontal, non-vertical, length and height (width), so that most characters and unconventional edges can be removed, and the coordinates of a plurality of linear starting points and ending points are obtained;

s112.3, taking intersection points, and selecting intersection points of all edges;

and S112.4, connecting all the intersection points, selecting a shape similar to the target element, removing the shape inconsistent with the characteristics of the target element through the area, the radius, the aspect ratio and the like of the target element, and remaining the same elements of the target element, namely, performing good generalization on the target element, and classifying the generalization elements into the same class. This method is very simple and efficient, but when the icon or button features are not obvious, a DOM tree analysis method is needed to get the desired generalization result.

S120, a DOM tree analysis method, which comprises the following steps:

s121, searching a path of a target element;

s122, searching elements under the same-level path according to the found path;

s123, eliminating irrelevant elements, and generalizing the target elements to obtain generalized elements; the rejection method is to find the parent element of the target element (the previous level element or the class to which the target element belongs), and then to exclude irrelevant elements according to the fact that when similar elements all find the same parent element, non-similar elements are not necessarily the parent element.

The beneficial effects of the invention are as follows:

the invention can greatly reduce the test cost and test time, can increase the robustness of the test scheme and the automatic generation of the test scheme, and the flow robot actively understands the user behavior.

The invention can be widely used in web page testing links, flow robots and the like, after the invention is used, a tester can not need to further update a test script because of the update of a developer to a web page, and can operate all similar elements by recording once or one script. Meanwhile, the flow robot has the function of actively understanding the user behavior. When the method is used, elements can be automatically generalized according to the normal recording or script flow, and a series of test flows or flow robot workflows can be automatically generated.

Drawings

FIG. 1 is a schematic diagram of a test page in an embodiment.

FIG. 2 is a schematic diagram of an exemplary embodiment after marginalizing a test page.

FIG. 3 illustrates the extraction of regions of interest after marginalizing a test page in an embodiment.

Fig. 4 is a schematic diagram of an edge selection in an embodiment.

FIG. 5 is a schematic diagram of an embodiment in which selected edges are intersected.

FIG. 6 is a schematic diagram of selecting a rectangle similar to the target element in the embodiment.

FIG. 7 is a schematic diagram of a DOM tree structure in an embodiment.

Detailed Description

The invention is further illustrated by the following examples in conjunction with the accompanying drawings:

currently, the main prior art for performing page testing is as follows:

1、Selenium

selenium is probably the most popular open source automated test framework in web applications. In 2 thousand years, a history of development has been over ten years so far, and Selenium has become the choice for many Web automation testers, especially those with advanced programming and scripting skills.

The Selenium supports multiple system environments (Windows, mac, linux) and multiple browsers (Chrome, fireFox, IE and headless browser (no interface)). Its script may be written in a variety of programming languages such as Java, groovy, python, C#, PHP, ruby, and Perl.

Because of the flexibility of Selenium, testers can write various complex, advanced test scripts to address various complex problems, which require advanced programming skills and effort to build automated test frameworks and libraries that meet their own needs.

Certificate: open source

2、Katalon Studio

Katalon Studio is a powerful automated test solution in web applications, mobile and web services. Based on the Selenium and Applium frameworks, katalon Studio integrates the advantages of these frameworks in terms of software automation.

This tool supports different levels of test skill sets. The non-programmer can also quickly start an automatic test project (such as recording test scripts by using spy objects), and meanwhile, the time for constructing a new library and maintaining scripts by the programmer and the advanced testers is saved.

Katalon Studio may be integrated into the CI/CD process and is compatible with popular quality processing tools, including qTest, JIRA, jenkins, and Git. It provides a very good function called Katalon analysis, providing a comprehensive test report to the user through indexes and graphs.

Certificate: free of charge

3、Uipath

UiPath Studio is a complete solution for application integration and automated execution of third party applications, administrative IT tasks and business IT flows.

The item is a graphical representation of the business process. It enables you to automatically execute rule-based flows by letting you have full control of the relation between execution order and a set of custom steps (also called activities in the UiPath Studio). Each activity contains a small action, such as clicking a button, reading a file or writing to a log panel.

The main types of supported items are:

sequence-apply to linear procedure, allowing you to smoothly go from one activity to another without confusing items.

Flow chart-applicable to more complex business logic, enabling you to integrate decisions and connect activities in a more diversified way through multiple branching logic operators.

State machine-applicable to very large items; they use a limited number of states in the execution, which are triggered by conditions (transitions) or activities.

The disadvantages of Selenium are: the element positioning mode is single and not intelligent, and requires advanced programming skills of technicians. Katalon Studio is a Selenium-based integrated toolkit that uses a recording interface that is user-friendly. Both of these have the disadvantage of being less robust-any page update or iteration with a high probability will result in test failure, while the tester will have much mechanically repeated effort if it is required to test for the same class of elements, which can be completely solved by this solution.

In terms of flow robots, the software such as uipath is still based on the definition of rules by the user, rather than actively understanding the user behavior. If the machine vision method and the DOM tree analysis method of the invention are used, the target elements can be generalized, and like elements can be generated, so that the robustness of the test flow can be greatly increased, and even if the webpage is updated in an iterative manner, the test flow can be still performed, and the user operation can be better understood. And meanwhile, repeated labor of testers or users is greatly reduced.

The invention is further explained below in connection with fig. 1-6:

s110 machine vision method:

s111, selecting the figure 1 as a test page, which is a web page to be tested, wherein the displayed options in regions are all one type of element in human view, but the selected southern Asian 1 (Monbout) is completely different in test scripts, and all regions are expected to be tested.

S112.1, changing the test page into a black-and-white binary image, wherein the edges of the icons or buttons can be better detected, and the later image is the result of marginalization.

All edges are detected by the formula of edge detection, which is as follows:

wherein: gx, horizontal gradient; gy: vertical gradient, fig. 2 was obtained. The region of interest to be extracted is then identified by the rectangular edges of the target element, as shown in fig. 3 below.

S112.2, through fig. 3, all the frames have been found, but it can be seen that the edge detection detects not only the frames but also the edges of the text, which is what we do not want. It is necessary to add some decision conditions such as length, whether horizontal, whether vertical, marked in the original as shown in fig. 4 below.

S112.3, through S112.2, obtains coordinates of the start point and the end point of a plurality of straight lines, and thus it is not possible to determine which straight lines form a rectangle. In order to obtain the precise positions of the icons and buttons, it is necessary to take the intersection points of the respective straight lines (take the intersection points), as shown in fig. 5.

And S112.4, connecting all the intersection points, then selecting a rectangular frame, and removing the rectangle with too much area difference from the target graph and the rectangle with inconsistent length-width ratio. As shown in fig. 6, the result of image feature generalization is well obtained. However, when the icon or button feature is not obvious, an additional method is needed to obtain the required generalization result, namely a DOM tree analysis method.

Referring to fig. 7, it can be seen by a simple observation: the DOM tree analysis method can find the target element mainly through proper div- > span- > div … - > span, and the process is defined as a path in the DOM tree. In general, homogeneous, generalizable element paths should be consistent.

S121, searching a path of a target element, and assuming that the target element is:

Huadong 2

；

The path of the target element can be obtained:

['div','span','div','div','div','div','div','div','div',

'dd','dl','form','div','div','div','div','div','div','di

v','div','div','div','div','div','div','div','body','htm

l','[document]']

；

s122, according to the path of the target element, searching the element in turn:

North China 2

Huadong 1

Huadong 2

south China 1

southeast Asian 3 (Jilong slope)

North China 5

Mexi 1 (silicon valley)

singapore span)

south asia 1 (Montbai)

southeast Asian 5 (Atcoada)

southeast Asian 2 (Sydney)

Meidong 1 (Virginia)

North China 3

uk (london)

calculation storage integral version: span-

proprietary network ]

classical network ]

1C SSD

2C SSD

16C SSD

2

；

S123, find some irrelevant elements according to the path, so a second assumption can be used: when homogeneous elements all find the same parent element, non-homogeneous elements must not be this parent element to exclude irrelevant elements. The pseudo code is expressed as follows:

input candidate element (including target element), target element-in this example is (Huadong 2)

Initialization: dic [ parent element of candidate element ] =candidate element

● Only the target element is in the dic that the while contains the target element, i.e., dic [ parent of target element..parent element ] = target element:

■dicTmp＝{}

■candiParents＝[]

■ foriin candidate:

ocandiParents.append(i.parent)

oif i.parent not in dicTmp.keys():

odicTmp[i.parent]+＝dic[i]

■dic＝dicTmp

■ Candidate = candiparants

；

The results obtained are as follows:

North China 2

Huadong 1

Huadong 2

south China 1

North China 5

Mexi 1 (silicon valley)

singapore span)

south asia 1 (Montbai)

southeast Asian 5 (Atcoada)

southeast Asian 2 (Sydney)

Meidong 1 (Virginia)

North China 3

uk (london)

The present invention is not described in detail in the present application, and is well known to those skilled in the art.

The foregoing describes in detail preferred embodiments of the present invention. It should be understood that numerous modifications and variations can be made in accordance with the concepts of the invention by one of ordinary skill in the art without undue burden. Therefore, all technical solutions which can be obtained by logic analysis, reasoning or limited experiments based on the prior art by the person skilled in the art according to the inventive concept shall be within the scope of protection defined by the claims.

Claims

1. The testing method based on the machine vision and the DOM tree structure is characterized by comprising the following steps of:

s300, performing automatic test according to the generalization element and recording;

in S100, two modes are included, namely S110, a machine vision method, S120 and a DOM tree analysis method, wherein the machine vision method and the DOM tree analysis method are used alternatively or in combination;

the DOM tree analysis method is performed by the structure of the DOM tree, and the analysis obeys the following assumptions:

suppose 1. The hierarchy of generalizable like elements must be similar;

suppose 2. When homogeneous elements all find the nearest same parent element, non-homogeneous elements must not be this parent element;

s110, a machine vision method, which comprises the following steps:

s111, selecting a target element, wherein the target element is an icon or a button, and detecting characteristic points of the target element to obtain characteristics of the target element;

s112, identifying edges of all elements in the page;

s113, screening all elements according to the characteristics of the target elements to obtain graphic elements with characteristics similar to the characteristics of the target elements, and obtaining generalized elements;

s120, a DOM tree analysis method, which comprises the following steps:

s121, searching a path of a target element;

s122, searching all elements under the same-level path according to the found path;

s123, eliminating irrelevant elements, and generalizing the target element to obtain a generalization element.

2. The test method of claim 1, wherein in S112, the steps of:

s112.1, marginalizing a target page, firstly preprocessing an image of a test page into a binary image, and then detecting edges, wherein the formula of edge detection is as follows:

in formula (1): gx is a horizontal Gradient, gy is a vertical Gradient, edge_gradient (G) is an Edge Gradient, and Angle (θ) is an Edge inclination;

s112.2, screening the obtained edges according to the characteristics of the target elements;

s112.3, taking intersection points, and selecting intersection points of the residual edges;

and S112.4, connecting all the intersection points, selecting a shape similar to the target element, screening through the characteristics of the target element, and obtaining the generalization element by the similar elements of the target element.

3. The test method as set forth in claim 1, wherein in S123, the culling is performed by finding a parent element of the target element, and then culling according to the condition that "when the same kind of element finds the same parent element, the non-same kind of element is not necessarily the same parent element to exclude irrelevant elements".