WO2015074521A1

WO2015074521A1 - Devices and methods for positioning based on image detection

Info

Publication number: WO2015074521A1
Application number: PCT/CN2014/091244
Authority: WO
Inventors: Sen Lin; Jiong LI; Qingyu TANG
Original assignee: Tencent Technology (Shenzhen) Company Limited
Priority date: 2013-11-19
Filing date: 2014-11-17
Publication date: 2015-05-28
Also published as: HK1200939A1; CN104133665A; CN104133665B

Abstract

Devices and methods are provided for positioning based on image detection. For example, a first image of a target control widget is captured from a first interface associated with a program; one or more layout parameters of the target control widget are determined based on at least information associated with a layout of the first interface; a location area of the target control widget is determined in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget; a second image of the target control widget is captured from the location area of the target control widget in the second interface; and whether the target control widget is positioned is detected based on at least information associated with the first image and the second image.

Description

DEVICES AND METHODS FOR POSITIONING BASED ON IMAGE DETECTION

CROSS-REFERENCES TO RELATED APPLICATIONS

The application claims priority to Chinese Patent Application No. 201310585139.2, filed November 19, 2013, incorporated by reference herein for all purposes.

BACKGROUND OF THE INVENTION

Certain embodiments of the present invention are directed to computer technology. More particularly, some embodiments of the invention provide devices and methods for positioning technology. Merely by way of example, some embodiments of the invention have been applied to image detection. But it would be recognized that the invention has a much broader range of applicability.

Program operation automation for graphical user interface (GUI) is usually applied to automatic testing to automatically accomplish some complicated input operations for a user and reduce a usage threshold of the user.

The program operation automation for GUI often includes interface control positioning, and user operation simulation. The user operation simulation is often accomplished by Windows input event simulation. The difference between the user operation simulation and the Windows input event simulation is often small. The interface control positioning can be achieved through many methods due to different interface implementation manners, including positioning based on interface control detection and positioning based on image detection. A method based on the interface control detection usually requires that an interface engine discloses interfaces to enable an automation program to acquire a handle of a control widget and further acquire detailed information of the control widget. When the interface engine does not disclose the interfaces to a third party, e.g., applications realized based on a private DirectUI engine, the interface engine cannot be used at all. On the other hand, positioning based on image detection may solve the problems encountered by the method based on the interface control detection because positioning based on image detection usually does not depend on an interface engine. A control widget can be detected based on final visual effects, which is closer to human eye recognition and more intelligent.

The current positioning technology based on image detection pre-captures images appearing in an operation process, then performs screenshots of a software execution interface in an automatic testing process, and searches the pre-captured images in the interface screenshots in order to position a target control widget.

The above-noted conventional positioning technology based on image detection has some disadvantages. For example, a current positioning method based on image detection usually pre-captures an entire interface, and then searches small images in the large pre-captured interface image. It often needs high resolution of the images, and usually requires that the resolution (e.g., size) during the execution of the program is the same as that during the pre-capturing of the images. Otherwise, the zoomed image may not be searched. In addition, if several pre-captured small images of control widgets appear during the execution of the program, the conventional positioning method based on image detection cannot distinguish which image corresponds to a target control widget.

Hence it is highly desirable to improve the techniques for positioning based on image detection.

BRIEF SUMMARY OF THE INVENTION

According to one embodiment, a method is provided for positioning based on image detection. For example, a first image of a target control widget is captured from a first interface associated with a program； one or more layout parameters of the target control widget are determined based on at least information associated with a layout of the first interface； a location area of the target control widget is determined in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget； a second image of the target control widget is captured from the location area of the target control widget in the second interface； and whether the target control widget is positioned is detected based on at least information associated with the first image and the second image.

According to another embodiment, a device for positioning based on image detection includes: a capture module configured to capture a first image of a target control widget from a first interface associated with a program； a determination module configured to determine one or more layout parameters of the target control widget based on at least information associated with a layout of the first interface and determine a location area of the target control widget in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget； wherein the capture module is further configured to capture a second image of the target control widget from the location area of the target control widget in the second interface； and a detection module configured to detect whether the target control widget is positioned based on at least information associated with the first image and the second image.

According to yet another embodiment, a non-transitory computer readable storage medium includes programming instructions for positioning based on image detection. The programming instructions are configured to cause one or more data processors to execute certain operations. For example, a first image of a target control widget is captured from a first interface associated with a program； one or more layout parameters of the target control widget are determined based on at least information associated with a layout of the first interface； a location area of the target control widget is determined in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget； a second image of the target control widget is captured from the location area of the target control widget in the second interface； and whether the target control widget is positioned is detected based on at least information associated with the first image and the second image.

Depending upon embodiment, one or more benefits may be achieved. These benefits and various additional objects, features and advantages of the present invention can be fully appreciated with reference to the detailed description and accompanying drawings that follow.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a simplified diagram showing a method for positioning based on image detection according to one embodiment of the present invention.

Figure 2 is a simplified diagram showing an interface layout according to one embodiment of the present invention.

Figure 3 (A) and Figure 3 (B) are simplified diagrams showing a method for positioning based on image detection according to some embodiments of the present invention.

Figure 4 is a simplified diagram showing a first interface for positioning based on image detection according to one embodiment of the present invention.

Figure 5 is a simplified diagram showing a first image for positioning based on image detection according to one embodiment of the present invention.

Figure 6 is a simplified diagram showing a second interface for positioning based on image detection according to one embodiment of the present invention.

Figure 7 is a simplified diagram showing a second image for positioning based on image detection according to one embodiment of the present invention.

Figure 8 is a simplified diagram showing a histogram of the first image as shown in Figure 5 according to some embodiments of the present invention, and Figure 9 is a simplified diagram showing a histogram of the second image as shown in Figure 7 according to certain embodiments of the present invention.

Figure 10 is a simplified diagram showing a third interface for positioning based on image detection according to one embodiment of the present invention.

Figure 11 is a simplified diagram showing a third image for positioning based on image detection according to one embodiment of the present invention.

Figure 12 is a simplified diagram showing a histogram of the third image as shown in Figure 11 according to one embodiment of the present invention.

Figure 13 is a simplified diagram showing a device for positioning based on image detection according to one embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Figure 1 is a simplified diagram showing a method for positioning based on image detection according to one embodiment of the present invention. The diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. The method 1 includes processes 100-104. For example, the method 1 is executed by a positioning device which is integrated via software.

According to one embodiment, the process 100 includes: capturing a first image of a target control widget from a first interface associated with a program. For example, the first interface corresponds to a default size when the program is executed. As an example, the program is opened, and runs on the first interface at the default size. As another example, the first interface includes a plurality of control widgets, and relevant spaces exist in a visual graphic format. As yet another example, the control widgets include a button, a file edit box, etc. The target control widget is one of the control widgets in the first interface according to some embodiments.

According to another embodiment, the process 101 includes: determining one or more layout parameters of the target control widget based on at least information associated with a layout of the first interface. For example, the layout parameters of the target control widget may be learned based on layouts of all the control widgets included in the first interface. As an example, the layout parameters of the target control widget include all parameters characterizing the target control widget in the first interface. As another example, the layout parameters of the target control widget include a size of the target control widget in the first interface, an alignment manner of the target control widget to a parent container in the first interface, a margin of the target control widget relative to the parent container (e.g., in the first interface) , an attribute value indicating whether the target control widget zooms with the parent container, etc. As yet another example, the parent container includes a main window of the first interface, or a part of the main window. In one example, the alignment manner includes top, bottom, left, right, top left, top right, bottom left, bottom right, center, etc. In another example, the attribute value indicating whether the target control widget zooms with the parent container is True or False. In yet another example, if the attribute value is True, it indicates that the target control widget zooms with the parent container. If the attribute value is False, it indicates that the target control widget does not zoom with the parent container.

Figure 2 is a simplified diagram showing an interface layout according to one embodiment of the present invention. The diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications.

As shown in Figure 2, a window A is a main window of a first interface, a window B is a part of the main window, and a control C is a part of the window B, according to some embodiments. For example, the window B is a parent container of the control C, and the window A is a parent container of the window B. As an example, one or more layout parameters of the target control widget relative to the parent container are determined as the layout parameters of the target control widget. In another example, one or more layout parameters of the parent container relative to a parent container of the parent container are determined in a similar manner till the parent container is the main window. A location of each target control widget in the main window can be determined according to the target control widget, the layout parameters of the target control widget relative to the parent container, and the layout parameters of the parent container relative to a parent container of the parent container, according to certain embodiments. For example, the layout location of the target control widget C in the main window A is determined based on the layout parameters of the target control widget C relative to the window B, and the layout parameters of the window B relative to the main window A.

Referring back to Figure 1, the process 102 includes: determining a location area of the target control widget in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget, according to some embodiments. For example, the second interface corresponds to a zoom-in interface or a zoom-out interface associated with the first interface. In another example, if the first interface is an interface in the default size, the second interface is obtained by zooming in or zooming out the first interface in the default size. As an example, the location area of the target control widget in the second interface is determined based on a size of the first interface, a size of the second interface, the size of the target control widget in the first interface, the alignment manner of the target control widget to the parent container in the first interface, the margin of the target control widget relative to the parent container (e.g., in the first interface) , and the attribute value indicating whether the target control widget zooms with the parent container.

According to yet another embodiment, the process 103 includes: capturing a second image of the target control widget from the location area of the target control widget in the second interface. For example, the location area of the target control widget in the second interface is determined accurately during the process 102. As an example, the process 103 captures the second image of the target control widget in the second interface from the location area in the second interface determined in the process 102.

In one embodiment, the process 104 includes: detecting whether the target control widget is positioned based on at least information associated with the first image and the second image. For example, the first image and the second image are images of the target control widget in the first interface and the second interface respectively, whether the target control widget is positioned may be determined by detecting the first image and the second image.

According to some embodiments, the method 1 is executed to capture the first image of the target control widget from the first interface associated with the program； determine the layout parameters of the target control widget based on the layout of the first interface； determine the location area of the target control widget in the second interface based on the first interface and the second interface associated with the program, and the layout parameters of the target control widget； capture the second image of the target control widget from the location area of the target control widget in the second interface； and detect whether the target control widget is positioned based on the first image and the second image. For example, the method 1 is executed to overcome deficiency of the conventional technology that cannot accurately locate a small image of a target control widget when a size of an interface during the execution of a program is inconsistent with a preset size. As an example, the method 1 is executed to accurately position the location area of the target control widget in the second interface, accurately acquire the image of the target control widget in the second interface by particularly capturing the second image of the target control widget in the second interface from the location area, and detect whether the target control widget is positioned based on the first image and second image. As another example, application scenarios of the technology of positioning based on image detection may be extended effectively by employing the method 1.

According to some embodiments, the process 102 includes: determining a second size of the target control widget in the second interface based on at least information associated with the attribute value indicating whether the target control widget zooms with the parent container, the first size of the target control widget in the first interface, an original size of the first interface, and a target size of the second interface. For example, when the attribute value indicating whether the target control widget zooms with the parent container is False, the first size of the target control widget in the first interface is the same as the second size of the target control widget in the second interface. In another example, when the attribute value indicating whether the target control widget zooms with the parent container is True, the second size of the target control widget in the second interface is determined based on an original size of the first interface, a target size of the second interface and the first size of the target control widget in the first interface.

According to certain embodiments, the process 102 includes: determining a second margin of the target control widget relative to the parent container in the second interface based on at least information associated with the first margin of the target control widget relative to the parent container in the first interface, the original size of the first interface, and the target size of the second interface. For example, a zoom-in scale or a zoom-out scale of the second interface relative to the first interface is determined based on the original size of the first interface and the target size of the second interface, the second margin of the target control widget relative to the parent container in the second interface is determined based on the first margin of the target control widget relative to the parent container in the first interface.

According to certain embodiments, the process 102 includes: determining the location area of the target control widget in the second interface based on at least information associated with the alignment manner of the target control widget to the parent container in the first interface, the second size of the target control widget in the second interface, and the second margin of the target control widget relative to the parent container in the second interface. For example, though layout parameters of each target control widget are relative to the parent container of the target control widget, layout parameters of each parent container may be determined in a similar manner. As an example, if the parent container is not the main window, the location area of the target control widget can be determined. As another example, the location area of the target control widget in the second interface is determined based on the alignment manner of the target control widget to the parent container in the first interface, the second size of the target control widget in the second interface, and the second margin of the target control widget relative to the parent container in the second interface.

According to some embodiments, the process 104 includes: detecting whether the target control widget is positioned based on at least information associated with a first text included in the first image and a second text included in the second image； and/or detecting whether the target control widget is positioned based on at least information associated with a first color of the first image and a second color of the second image. For example, whether the target control widget is positioned is determined by detecting the first text included in the first image and the second text included in the second image, or by detecting the first color of the first image and the second color of the second image. In another example, the first text included in the first image and the second text included in the second image, as well as the first color of the first image and the second color of the second image are detected simultaneously. It is determined that the target control widget is positioned when the detection of the first text and the second text and the detection of the first color and the second color both pass. Otherwise, the target control widget is not considered as being positioned, according to some embodiments.

According to certain embodiments, the process for detecting whether the target control widget is positioned based on at least information associated with a first text included in the first image and a second text included in the second image includes: extracting the first text included in the first image and the second text included in the second image from the first image and the second image respectively. For example, the first text and/or the second text include words formed by letters, numbers, Chinese characters, etc.

According to some embodiments, the process for detecting whether the target control widget is positioned based on at least information associated with a first text included in the first image and a second text included in the second image includes: determining whether the first text included in the first image is the same as the second text included in the second image； in response to the first text included in the first image being the same as the second text included in the second image, determining that the target control widget is positioned； and in response to the first text included in the first image not being the same as the second text included in the second image, determining that the target control widget is not positioned. For example, when the first text included in the first image is the same as the second text included in the second image, the second image of the target control widget acquired is accurate. That is, the target control widget is positioned. Otherwise, the target control widget is not positioned.

According to certain embodiments, the process for detecting whether the target control widget is positioned based on at least information associated with a first color of the first image and a second color of the second image includes: extracting a first color histogram of the first image and a second color histogram of the second image, e.g., separately. For example, the first color histogram and the second color histogram include color information of the first image and color information of the second image, respectively, where the first color histogram and/or the second color histogram are expressed by red (R) , green (G) and blue (B) information.

According to some embodiments, the process for detecting whether the target control widget is positioned based on at least information associated with a first color of the first image and a second color of the second image includes: extracting a first color value of the first image and a second color value of the second image from the first color histogram of the first image and the second color histogram of the second image respectively. For example, an R value, a G value and a B value corresponding to the first image are acquired from the first image. Similarly, an R value, a G value and a B value corresponding to the second image are acquired from the second image.

According to some embodiments, the process for detecting whether the target control widget is positioned based on at least information associated with a first color of the first image and a second color of the second image includes: determining whether a difference between the first color value and the second color value is less than or equal to a preset threshold； in response to the difference between the first color value and the second color value being less than or equal to the preset threshold, determining that the target control widget is positioned； and in response to the difference between the first color value and the second color value being larger than the preset threshold, determining that the target control widget is not positioned. For example, whether a difference between the R value of the first image and the R value of the second image is less than or equal to the preset threshold, whether a difference between the G value of the first image and the G value of the second image is less than or equal to the preset threshold, and whether a difference between the B value of the first image and the B value of the second image is less than or equal to the preset threshold are determined separately. As an example, the target control widget is determined as being positioned if the three differences are all less than or equal to the preset threshold. Otherwise, the target control widget is not determined as being positioned. In one example, the preset threshold is set based on actual demands. As an example, the preset threshold corresponds to 5％ of the first color value.

According to certain embodiments, colors, brightness and contrasts of the first image and/or the second image are compared. For example, when the differences between colors, brightness and contrast of the first image and of the second image are less than or equal to a preset threshold respectively, the target control widget is determined as being positioned. Otherwise, the target control widget is not determined as being positioned.

Figure 3 (A) and Figure 3 (B) are simplified diagrams showing a method for positioning based on image detection according to some embodiments of the present invention. The diagrams are merely examples, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. The method 2 includes processes 200-210.

According to one embodiment, the process 200 includes: opening a program, and capturing a first image of a target control widget from a first interface in a default size associated with the program. For example, the first interface includes a plurality of control widgets, and one control widget is selected as the target control widget. As an example, the first image of the target control widget is captured from the first interface.

According to another embodiment, the process 201 includes: determining, based on a layout of the first interface, a first size of the target control widget in the first interface, an alignment manner of the target control widget to a parent container in the first interface, a first margin of the target control widget relative to the parent container (e.g., in the first interface) , and an attribute value indicating whether the target control widget zooms with the parent container. For example, a location of the target control widget relative to a main window may not be described accurately. To accurately show the location of the target control widget, the first interface is divided, according to some embodiments. For example, parent windows and child windows are divided layer upon layer from a main window, and the target control widget is placed in a smaller window to describe accurately the relationship between the target control widget and the smaller window. As an example, a parent window of each window is a parent container (e.g., as shown in Figure 2) . Parameter values related to the first size of the target control widget in the first interface, the alignment manner of the target control widget to the parent container in the first interface, the first margin of the target control widget relative to the parent container (e.g., in the first interface) , and the attribute value indicating whether the target control widget zooms with the parent container are similar to those described in Figure 1–Figure 2.

According to yet another embodiment, the process 202 includes: determining a second size of the target control widget in a second interface based on the attribute value indicating whether the target control widget zooms with the parent container, the first size of the target control widget in the first interface, an original size of the first interface, and a target size of the second interface. For example, a size of the target control widget is expressed by width and height, e.g., in a format of x, y (width, height) .

According to some embodiments, the process 203 includes: determining a second margin of the target control widget relative to the parent container in the second interface based on the first margin of the target control widget relative to the parent container in the first interface, the original size of the first interface, and the target size of the second interface. In one embodiment, the first margin of the target control widget relative to the parent container in the first interface and the second margin of the target control widget relative to the parent container in the second interface are expressed by four parameters. For example, a margin format includes left, top, right, and bottom, corresponding to left, top, right, and bottom locations of the target control widget relative to the parent container. In another embodiment, the margin format is expressed in combination with the alignment manner of the target control widget to the parent container in the first interface. For example, the right and the bottom in the margin format are set as zero for a top left alignment, and the bottom and the left in the margin format are set as zero for a top right alignment. In another example, the left and the right may not be assigned a value simultaneously for a center alignment. As an example, one of the left and the right is set as zero for the convenience of positioning. As another example, the top and the bottom may not be assigned a value simultaneously, and one of the top and the bottom is set as zero.

According to certain embodiments, the process 204 includes: determining a location area of the target control widget in the second interface based on the alignment manner of the target control widget to the parent container in the first interface, the second size of the target control widget in the second interface, and the second margin of the target control widget relative to the parent container in the second interface. For example, the alignment manner of the target control widget to the parent container in the first interface is the same as an alignment manner of the target control widget to the parent container in the second interface, so that the location area of the target control widget in the second interface may be determined accurately based on the alignment manner of the target control widget to the parent container in the second interface, the second size of the target control widget in the second interface, and the second margin of the target control widget relative to the parent container in the second interface.

According to some embodiments, the process 205 includes: capturing a second image of the target control widget from the location area of the target control widget in the second interface. For example, the second image is an image of the target control widget in the second interface. As an example, the process 206 includes: determining whether the first image and the second image include texts. As another example, if the first image and the second image include texts, the process 207 is executed. Otherwise, the process 208 is executed.

According to certain embodiments, the process 207 includes: determining whether the first text included in the first image is the same as the second text included in the second image； if the first text included in the first image is the same as the second text included in the second image, executing the process 208； and if the first text included in the first image is not the same as the second text included in the second image, determining that the target control widget is not positioned and ending the method 2. For example, the first text and/or the second text include numbers, character strings, Chinese characters, a combination thereof, etc.

According to some embodiments, the process 208 includes: extracting a first color histogram of the first image and a second color histogram of the second image separately. For example, the process 209 includes: extracting a first color value of the first image and a second color value of the second image from the first color histogram of the first image and the second color histogram of the second image respectively. As an example, R, G and B values of the first image, and R, G and B values of the second image are extracted. As another example, brightness, colors, and contrasts may be acquired from the first color histogram and/or the second color histogram for providing conditions for determination.

According to certain embodiments, the process 210 includes: determining whether a difference between the first color value of the first image and the second color value of the second image is less than or equal to a preset threshold； if the difference between the first color value of the first image and the second color value of the second image is less than or equal to the preset threshold, determining that the target control widget is positioned； and if the difference between the first color value of the first image and the second color value of the second image is larger than the preset threshold, determining that the target control widget is not positioned and ending the method 2.

According to some embodiments, the color values include the R values, the G values and the B values. For example, first R values, first G values and first B values of the first image and second R values, second G values and second B values of the second image are detected. As an example, if first differences between the first R values and the second R values, second differences between the first G values and the second G values, and third differences between the first B values and the second B values are all less than or equal to the preset threshold, the target control widget is determined as being positioned. Otherwise, the target control widget is determined as not being positioned. As another example, one or more other parameters, such as brightness, color, and contrast, are added as conditions for determination. When the differences between these parameters of the two images are determined as all being less than or equal to the preset threshold, the target control widget is determined as being positioned. Otherwise, the target control widget is determined as not being positioned.

Figure 4 is a simplified diagram showing a first interface for positioning based on image detection according to one embodiment of the present invention. The diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications.

Figure 5 is a simplified diagram showing a first image for positioning based on image detection according to one embodiment of the present invention. The diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications.

Figure 6 is a simplified diagram showing a second interface for positioning based on image detection according to one embodiment of the present invention. The diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications.

Figure 7 is a simplified diagram showing a second image for positioning based on image detection according to one embodiment of the present invention. The diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Figure 4–Figure 7 are related to a “League of Legends” client.

For example, (1) the program is started, and a default size of the program is recorded as 1280*800. As shown in Figure 4, a first interface corresponds to a sectional view of the “League of Legends” client in the default size of 1280*800, as an example. In another example, (2) a first image of a play button as a target control widget (e.g., as shown in Figure 5) is captured from the first interface (e.g., as shown in Figure 4) . In yet another example, (3) the first interface is zoomed. One or more layout parameters of the play button are defined based on the layout change of the target control widget – the play button, when the program interface is zoomed. For example, the defined layout parameters are as shown in Table 1.

Table 1

According to the definitions in Table 1, the location of the play button relative to the top left corner of the main window is fixed, and the button is 571 pixels from the left of the main window, 17 pixels from the top of the main window, and 142*56 in size, according to some embodiments. For example, the button zooms with the main window, and the screenshot is play_normal.bmp by default. As an example, with the adoption of the layout description combination, layout information of the control widget is described via a certain hierarchical layout nest as long as the control widget does not appear in the program interface randomly.

According to one embodiment, (4) the interface of the program is moved. For example, the interface of the program is zoomed in to 1024*640, and subjected to screenshot again at the moment to obtain a second interface (e.g., as shown in Figure 6) . As an example, the second interface has a pixel size of 1024*640. According to another embodiment, (5) a zooming ratio of a current interface is calculated as: 1024/1280＝0.8. For example, a layout attribute of the play button defines zoom＝true, and the calculated actual layout information of the play button includes: size＝113,44, and margin＝456,13,0,0. In another example, after the second interface is captured, the second image of the target control widget is captured from the second interface, where the second image of the target control widget in the second interface starts from (x: 456, y: 13) and has (width: 113, height: 44) , e.g., as shown in Figure 7.

According to another embodiment, (6) image comparison is performed. For example, a PLAY button 1 with an original size (e.g., as shown in Figure 5) is compared with a PLAY button 2 (e.g., as shown in Figure 7) that obtained during the actual execution of the program. According to yet another embodiment, the processes (1) , (2) , (3) , (4) , (5) and (6) as described above are performed sequentially. According to yet another embodiment, the processes (1) , (2) , (3) , (4) , (5) and (6) as described above are performed non-sequentially (e.g., simultaneously) . Table 2 shows an example comparison between the two images.

Table 2

	First image	Second image
	First image	Second image	size	142,56	113,44
margin	571,17,0,0	456,13,0,0	size	142,56	113,44
margin	571,17,0,0	456,13,0,0	Screenshot	Figure 5	Figure 7
Histogram	Figure 8	Figure 9	Screenshot	Figure 5	Figure 7

Figure 8 is a simplified diagram showing a histogram of the first image as shown in Figure 5 according to some embodiments of the present invention, and Figure 9 is a simplified diagram showing a histogram of the second image as shown in Figure 7 according to certain embodiments of the present invention. The diagrams are merely examples, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications.

As shown in Figure 5 and Figure 7, the two images are determined as one with the naked eye, according to some embodiments. However, after zooming, the two images may not be compared directly at pixel level due to the effects of a zooming algorithm and location errors caused by the zoom scale, according to certain embodiments. For example, the respective color histograms of the two images are extracted, and four dimensions of R/G/B/Gray are compared, where Gray represents brightness of the images. As an example, the two images are considered approximately the same if an average error does not exceed a preset threshold (e.g., 5％) . As another example, the preset threshold may be adjusted according to the actual execution of the program.

Figure 10 is a simplified diagram showing a third interface for positioning based on image detection according to one embodiment of the present invention. The diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications.

Figure 11 is a simplified diagram showing a third image for positioning based on image detection according to one embodiment of the present invention. The diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications.

According to one embodiment, a current program is not in a main interface, and instead the program is in the third interface as shown in Figure 10. For example, a screenshot of a target control widget in the third interface is captured to obtain the third image as shown in Figure 11. As an example, image comparison is performed. Table 3 shows an example comparison between the first image as shown in Figure 5 and the third image as shown in Figure 11.

Table 3

	First image	Third image
	First image	Third image	size	142,56	113,44
margin	571,17,0,0	456,13,0,0	size	142,56	113,44
margin	571,17,0,0	456,13,0,0	Screenshot	Figure 5	Figure 11
Histogram	Figure 8	Figure 12	Screenshot	Figure 5	Figure 11

Figure 12 is a simplified diagram showing a histogram of the third image as shown in Figure 11 according to one embodiment of the present invention. The diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications.

According to one embodiment, one or more control widgets (other than the PLAY button) is determined easily in the third image by comparing color information of the screenshot and a preset screenshot. According to certain embodiments, whether the target control widget on the program interface is displayed currently is determined, and then the target control widget is operated by simulating keyboard and mouse events to further analyze and determine one or more subsequent interfaces, so that automatic operation of the program is accomplished. As described above, an interface description definition for a fixed length-width ratio. For programs with a flexible length-width ratio, zoom attributes can be set for an x axis and a y axis respectively, according to certain embodiments.

For image comparison, if a destination area where the target control widget is located displays other images with similar color features as the target control widget, mistaken determination may be caused, in some embodiments. The destination area may be divided into a plurality of blocks (e.g., four blocks) which are compared one by one to improve the accuracy, according to certain embodiments. If obvious texts and textures appear in the destination area, text detection may be introduced, and the detected texts are compared to avoid negative effects of scaling on image comparison, according to some embodiments.

Figure 13 is a simplified diagram showing a device for positioning based on image detection according to one embodiment of the present invention. The diagram is merely an example, which should not unduly limit the scope of the claims. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. The device 3 includes a capture module 10, a determination module 11 and a detection module 12.

According to one embodiment, the capture module 10 is configured to capture a first image of a target control widget from a first interface associated with a program. For example, the determination module 11 is configured to determine one or more layout parameters of the target control widget based on at least information associated with a layout of the first interface and determine a location area of the target control widget in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget. As an example, the capture module 10 is connected with the target control widget 11. As another example, the capture module 10 is further configured to capture a second image of the target control widget from the location area of the target control widget in the second interface. As yet another example, the detection module 12 is connected with the capture module 10. As yet another example, the detection module 12 is configured to detect whether the target control widget is positioned based on at least information associated with the first image and the second image. The device 3 is configured to execute the method 1 and/or the method 2, according to certain embodiments.

According to some embodiments, the device 3 is configured to capture the first image of the target control widget from the first interface associated with the program； determine the layout parameters of the target control widget based on the layout of the first interface； determine the location area of the target control widget in the second interface based on the first interface and the second interface associated with the program, and the layout parameters of the target control widget； capture the second image of the target control widget from the location area of the target control widget in the second interface； and detect whether the target control widget is positioned based on the first image and the second image. For example, the device 3 is configured to overcome deficiency of the conventional technology that cannot accurately locate a small image of a target control widget when a size of an interface during the execution of a program is inconsistent with a preset size. As an example, the device 3 is configured to accurately position the location area of the target control widget in the second interface, accurately acquire the image of the target control widget in the second interface by particularly capturing the second image of the target control widget in the second interface from the location area, and detect whether the target control widget is positioned based on the first image and second image. As another example, application scenarios of the technology of positioning based on image detection may be extended effectively by employing the device 3.

According to certain embodiments, the first interface corresponds to a default size associated with the program. For example, the second interface corresponds to a zoom-in interface or a zoom-out interface associated with the first interface. As an example, the layout parameters include a first size of the target control widget in the first interface, an alignment manner of the target control widget to a parent container in the first interface, a first margin of the target control widget relative to the parent container in the first interface, and an attribute value indicating whether the target control widget zooms with the parent container. In another example, the parent container includes a main window of the first interface, or a part of the main window.

According to some embodiments, the determination module 11 is further configured to: determine a second size of the target control widget in the second interface based on at least information associated with the attribute value indicating whether the target control widget zooms with the parent container, the first size of the target control widget in the first interface, an original size of the first interface, and a target size of the second interface； determine a second margin of the target control widget relative to the parent container in the second interface based on at least information associated with the first margin of the target control widget relative to the parent container in the first interface, the original size of the first interface, and the target size of the second interface； and determine the location area of the target control widget in the second interface based on at least information associated with the alignment manner of the target control widget to the parent container in the first interface, the second size of the target control widget in the second interface, and the second margin of the target control widget relative to the parent container in the second interface.

According to some embodiments, the detection module 12 is further configured to detect whether the target control widget is positioned based on at least information associated with a first text included in the first image and a second text included in the second image, and/or detect whether the target control widget is positioned based on at least information associated with a first color of the first image and a second color of the second image. For example, the detection module 12 is further configured to: extract the first text included in the first image from the first image； extract the second text included in the second image from the second image； determine whether the first text included in the first image is the same as the second text included in the second image； in response to the first text included in the first image being the same as the second text included in the second image, determine that the target control widget is positioned； and in response to the first text included in the first image not being the same as the second text included in the second image, determine that the target control widget is not positioned. As an example, the detection module 12 is further configured to: extract a first color histogram of the first image； extract a second color histogram of the second image； extract a first color value of the first image from the first color histogram of the first image； extract a second color value of the second image from the second color histogram of the second image； determine whether a difference between the first color value and the second color value is less than or equal to a preset threshold； in response to the difference between the first color value and the second color value being less than or equal to the preset threshold, determine that the target control widget is positioned； and in response to the difference between the first color value and the second color value being larger than the preset threshold, determine that the target control widget is not positioned.

According to one embodiment, a method is provided for positioning based on image detection. For example, a first image of a target control widget is captured from a first interface associated with a program； one or more layout parameters of the target control widget are determined based on at least information associated with a layout of the first interface； a location area of the target control widget is determined in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget； a second image of the target control widget is captured from the location area of the target control widget in the second interface； and whether the target control widget is positioned is detected based on at least information associated with the first image and the second image. For example, the method is implemented according to at least Figure 1 and/or Figure 3 (A)–Figure 3 (B) .

According to another embodiment, a device for positioning based on image detection includes: a capture module configured to capture a first image of a target control widget from a first interface associated with a program； a determination module configured to determine one or more layout parameters of the target control widget based on at least information associated with a layout of the first interface and determine a location area of the target control widget in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget； wherein the capture module is further configured to capture a second image of the target control widget from the location area of the target control widget in the second interface； and a detection module configured to detect whether the target control widget is positioned based on at least information associated with the first image and the second image. For example, the device is implemented according to at least Figure 13.

According to yet another embodiment, a non-transitory computer readable storage medium includes programming instructions for positioning based on image detection. The programming instructions are configured to cause one or more data processors to execute certain operations. For example, a first image of a target control widget is captured from a first interface associated with a program； one or more layout parameters of the target control widget are determined based on at least information associated with a layout of the first interface； a location area of the target control widget is determined in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget； a second image of the target control widget is captured from the location area of the target control widget in the second interface； and whether the target control widget is positioned is detected based on at least information associated with the first image and the second image. For example, the storage medium is implemented according to at least Figure 1 and/or Figure 3 (A)–Figure 3 (B) .

The above only describes several scenarios presented by this invention, and the description is relatively specific and detailed, yet it cannot therefore be understood as limiting the scope of this invention. It should be noted that ordinary technicians in the field may also, without deviating from the invention’s conceptual premises, make a number of variations and modifications, which are all within the scope of this invention. As a result, in terms of protection, the patent claims shall prevail.

For example, some or all components of various embodiments of the present invention each are, individually and/or in combination with at least another component, implemented using one or more software components, one or more hardware components, and/or one or more combinations of software and hardware components. In another example, some or all components of various embodiments of the present invention each are, individually and/or in combination with at least another component, implemented in one or more circuits, such as one or more analog circuits and/or one or more digital circuits. In yet another example, various embodiments and/or examples of the present invention can be combined.

Additionally, the methods and systems described herein may be implemented on many different types of processing devices by program code comprising program instructions that are executable by the device processing subsystem. The software program instructions may include source code, object code, machine code, or any other stored data that is operable to cause a processing system to perform the methods and operations described herein. Other implementations may also be used, however, such as firmware or even appropriately designed hardware configured to perform the methods and systems described herein.

The systems’ and methods’ data (e.g., associations, mappings, data input, data output, intermediate data results, final data results, etc.) may be stored and implemented in one or more different types of computer-implemented data stores, such as different types of storage devices and programming constructs (e.g., RAM, ROM, EEPROM, Flash memory, flat files, databases, programming data structures, programming variables, IF-THEN (or similar type) statement constructs, application programming interface, etc. ) . It is noted that data structures describe formats for use in organizing and storing data in databases, programs, memory, or other computer-readable media for use by a computer program.

The systems and methods may be provided on many different types of computer-readable media including computer storage mechanisms (e.g., CD-ROM, diskette, RAM, flash memory, computer’s hard drive, DVD, etc. ) that contain instructions (e.g., software) for use in execution by a processor to perform the methods’ operations and implement the systems described herein. The computer components, software modules, functions, data stores and data structures described herein may be connected directly or indirectly to each other in order to allow the flow of data needed for their operations. It is also noted that a module or processor includes a unit of code that performs a software operation, and can be implemented for example as a subroutine unit of code, or as a software function unit of code, or as an object (as in an object-oriented paradigm) , or as an applet, or in a computer script language, or as another type of computer code. The software components and/or functionality may be located on a single computer or distributed across multiple computers depending upon the situation at hand.

The computing system can include client devices and servers. A client device and server are generally remote from each other and typically interact through a communication network. The relationship of client device and server arises by virtue of computer programs running on the respective computers and having a client device-server relationship to each other.

This specification contains many specifics for particular embodiments. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations, one or more features from a combination can in some cases be removed from the combination, and a combination may, for example, be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Although specific embodiments of the present invention have been described, it is understood by those of skill in the art that there are other embodiments that are equivalent to the described embodiments. Accordingly, it is to be understood that the invention is not to be limited by the specific illustrated embodiments, but only by the scope of the appended claims.

Claims

A method for positioning based on image detection, the method comprising:

capturing a first image of a target control widget from a first interface associated with a program；

determining one or more layout parameters of the target control widget based on at least information associated with a layout of the first interface；

determining a location area of the target control widget in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget；

capturing a second image of the target control widget from the location area of the target control widget in the second interface； and

detecting whether the target control widget is positioned based on at least information associated with the first image and the second image.
The method of claim 1, wherein:

the first interface corresponds to a default size associated with the program； and

the second interface corresponds to a zoom-in interface or a zoom-out interface associated with the first interface.
The method of claim 1, wherein:

the layout parameters include a first size of the target control widget in the first interface, an alignment manner of the target control widget to a parent container in the first interface, a first margin of the target control widget relative to the parent container in the first interface, and an attribute value indicating whether the target control widget zooms with the parent container； and

the parent container includes a main window of the first interface, or a part of the main window.
The method of claim 3, wherein the determining a location area of the target control widget in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget includes:

determining a second size of the target control widget in the second interface based on at least information associated with the attribute value indicating whether the target control widget zooms with the parent container, the first size of the target control widget in the first interface, an original size of the first interface, and a target size of the second interface；

determining a second margin of the target control widget relative to the parent container in the second interface based on at least information associated with the first margin of the target control widget relative to the parent container in the first interface, the original size of the first interface, and the target size of the second interface； and

determining the location area of the target control widget in the second interface based on at least information associated with the alignment manner of the target control widget to the parent container in the first interface, the second size of the target control widget in the second interface, and the second margin of the target control widget relative to the parent container in the second interface.
The method of any of claims 1-4, wherein the detecting whether the target control widget is positioned based on at least information associated with the first image and the second image includes: detecting whether the target control widget is positioned based on at least information associated with a first text included in the first image and a second text included in the second image.
The method of claim 5, wherein the detecting whether the target control widget is positioned based on at least information associated with a first text included in the first image and a second text included in the second image includes:

extracting the first text included in the first image from the first image；

extracting the second text included in the second image from the second image；

determining whether the first text included in the first image is the same as the second text included in the second image；

in response to the first text included in the first image being the same as the second text included in the second image, determining that the target control widget is positioned； and

in response to the first text included in the first image not being the same as the second text included in the second image, determining that the target control widget is not positioned.
The method of any of claims 1-4, wherein the detecting whether the target control widget is positioned based on at least information associated with the first image and the second image includes: detecting whether the target control widget is positioned based on at least information associated with a first color of the first image and a second color of the second image.
The method of claim 7, wherein the detecting whether the target control widget is positioned based on at least information associated with a first color of the first image and a second color of the second image includes:

extracting a first color histogram of the first image；

extracting a second color histogram of the second image；

extracting a first color value of the first image from the first color histogram of the first image；

extracting a second color value of the second image from the second color histogram of the second image；

determining whether a difference between the first color value and the second color value is less than or equal to a preset threshold；

in response to the difference between the first color value and the second color value being less than or equal to the preset threshold, determining that the target control widget is positioned； and

in response to the difference between the first color value and the second color value being larger than the preset threshold, determining that the target control widget is not positioned.
A device for positioning based on image detection, the device comprising:

a capture module configured to capture a first image of a target control widget from a first interface associated with a program；

a determination module configured to determine one or more layout parameters of the target control widget based on at least information associated with a layout of the first interface and determine a location area of the target control widget in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget；

wherein the capture module is further configured to capture a second image of the target control widget from the location area of the target control widget in the second interface； and

a detection module configured to detect whether the target control widget is positioned based on at least information associated with the first image and the second image.
The device of claim 9, wherein:

the first interface corresponds to a default size and is associated with the program； and

the second interface corresponds to a zoom-in interface or a zoom-out interface associated with the first interface.
The device of claim 9, wherein the layout parameter includes a size of the target control widget in the first interface, an alignment manner of the target control widget to a parent container in the first interface, amargin of the target control widget relative to the parent container, and an attribute value indicating whether the target control widget zooms with the parent container, and the parent container is a main window of the first interface, or a part of the main window.
The device of claim 11, wherein the determination module is further configured to:

determine a second size of the target control widget in the second interface based on at least information associated with the attribute value indicating whether the target control widget zooms with the parent container, the first size of the target control widget in the first interface, an original size of the first interface, and a target size of the second interface；

determine a second margin of the target control widget relative to the parent container in the second interface based on at least information associated with the first margin of the target control widget relative to the parent container in the first interface, the original size of the first interface, and the target size of the second interface； and

determine the location area of the target control widget in the second interface based on at least information associated with the alignment manner of the target control widget to the parent container in the first interface, the second size of the target control widget in the second interface, and the second margin of the target control widget relative to the parent container in the second interface.
The device of any of claims 9-12, wherein the detection module is further configured to detect whether the target control widget is positioned based on at least information associated with a first text included in the first image and a second text included in the second image.
The device of claim 13, wherein the detection module is further configured to:

extract the first text included in the first image from the first image；

extract the second text included in the second image from the second image；

determine whether the first text included in the first image is the same as the second text included in the second image；

in response to the first text included in the first image being the same as the second text included in the second image, determine that the target control widget is positioned； and

in response to the first text included in the first image not being the same as the second text included in the second image, determine that the target control widget is not positioned.
The device of any of claims 9-12, wherein the detection module is further configured to detect whether the target control widget is positioned based on at least information associated with a first color of the first image and a second color of the second image.
The device of claim 15, wherein the detection module is further configured to:

extract a first color histogram of the first image；

extract a second color histogram of the second image；

extract a first color value of the first image from the first color histogram of the first image；

extract a second color value of the second image from the second color histogram of the second image；

determine whether a difference between the first color value and the second color value is less than or equal to a preset threshold；

in response to the difference between the first color value and the second color value being less than or equal to the preset threshold, determine that the target control widget is positioned； and

in response to the difference between the first color value and the second color value being larger than the preset threshold, determine that the target control widget is not positioned.
The device of claim 9, further comprising:

one or more data processors； and

a computer-readable storage medium；

wherein one or more of the capture module, the determination module, and the detection module are stored in the storage medium and configured to be executed by the one or more data processors.
A non-transitory computer readable storage medium comprising programming instructions for positioning based on image detection, the programming instructions configured to cause one or more data processors to execute operations comprising:

capturing a first image of a target control widget from a first interface associated with a program；

determining one or more layout parameters of the target control widget based on at least information associated with a layout of the first interface；

determining a location area of the target control widget in a second interface based on at least information associated with the second interface, the first interface, and the layout parameters of the target control widget；

capturing a second image of the target control widget from the location area of the target control widget in the second interface； and

detecting whether the target control widget is positioned based on at least information associated with the first image and the second image.