CN102830958A - Method and system for obtaining interface control information - Google Patents
Method and system for obtaining interface control information Download PDFInfo
- Publication number
- CN102830958A CN102830958A CN201110161998XA CN201110161998A CN102830958A CN 102830958 A CN102830958 A CN 102830958A CN 201110161998X A CN201110161998X A CN 201110161998XA CN 201110161998 A CN201110161998 A CN 201110161998A CN 102830958 A CN102830958 A CN 102830958A
- Authority
- CN
- China
- Prior art keywords
- control
- selection area
- image
- area image
- detected
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000000052 comparative effect Effects 0.000 claims description 12
- 238000013519 translation Methods 0.000 claims description 4
- 238000012360 testing method Methods 0.000 abstract description 26
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000000295 complement effect Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000004615 ingredient Substances 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention provides a method and a system for obtaining interface control information. The method comprises the following steps of: according to a pre-set strategy, selecting an area in an interface, and capturing an image of the area, wherein the selected area is consistent to a pre-obtained area with a control to be detected; obtaining a pixel characteristic of the image of the selected area according to the pre-set strategy through a pixel combination of the image of the selected area; and determining information of the interface control in the selected area according to a comparing result of the pixel characteristic of the image of the selected area, and a pixel characteristic of an image of the area which is pre-obtained according to the pre-set strategy and includes the control to be detected. With the adoption of the method and system for obtaining the interface control information provided by the invention, the pixel characteristic of the image of the control of a non-standard window is analyzed, and is compared with the pixel characteristic of the image of the area which is pre-obtained according to the pre-set strategy and includes the control to be detected, so that whether the desired control appears in the selected area in the window or not can be judged, and state information of the control in the selected area can be obtained. Therefore, the purpose of automatically testing the control of the non-standard window with a uniform method can be realized.
Description
Technical field
The present invention relates to field of computer technology, relate in particular to a kind of method and system of obtaining interface control information.
Background technology
The widespread use of the develop rapidly of Along with computer technology and Windows system; The various application programs that run in the Windows system are also more and more; In the Windows software automated testing; Need replace the people to operate tested software with program,, again it carried out operations such as mouse or keyboard so program need obtain the position of certain control in the window.
Windows provides the method for obtaining standard control information; Can obtain the customizing messages of a control like system API (API) such as FindWindow, GetWindow, GetWindowRect, Windows window or standard control have attributes such as class name, text.
The handle that the FindWindow function can obtain window through the text and the class name of window; The handle that the GetWindow function can be through parent window and the text of control obtain the handle of control; The GetWindowRect function can obtain the coordinate information of control with respect to screen through the handle of control; Screen coordinate information has been arranged, operations such as the automatic test program just can be clicked it, focus switching, button.
Yet the control in the Windows graphical interfaces is not necessarily to create with standard mode; This non-standard control can't or not handled the standard windows messaging; Also possibly there are not attributes such as control text, so just can't satisfy the demands as this class function of FindWindow.Non-standard control all has unique real mode, can go into seriously its inner implementation, or all obtain the support of control exploitation side, makes control accept specific message, thereby returns corresponding information.But this method is general inadequately, and a kind of different non-standard control of every processing will use a kind of specific method, reduces the development efficiency to the system automation test.
Graphical interfaces as shown in Figure 1 is exactly to be made up of one group of off-gauge control, can't confirm the wherein position of check box and button with conventional method.
Because the implementation method of non-standard control is had nothing in common with each other, so a kind of technical scheme of choosing to non-standard window control in the Windows graphical interfaces of current needs, to reach the purpose of under the Windows system, carrying out automatic test.
Summary of the invention
Technical matters to be solved by this invention provides a kind of method and system of obtaining interface control information; Solve problem how in the Windows graphical interfaces, to choose the non-standard window control that needs, reached the purpose of under the Windows system, carrying out automatic test.
In order to address the above problem, the invention provides a kind of method of obtaining interface control information, comprising:
According to preset strategy, a selected zone in the interface, and the image that intercepting should the zone, wherein said selection area and acquisition in advance comprise the regional consistent of control to be detected;
According to the combination of pixels of the image of said selection area, obtain the pixel characteristic of this selection area image by preset strategy;
According to the pixel characteristic of said selection area image comparative result, confirm the information of the interface control in this selection area with the pixel characteristic of the area image that comprises control to be detected that obtains in advance by preset strategy.
Further, said method also can comprise, saidly obtains the step of the pixel characteristic of this area image by preset strategy, comprising:
The color of the control that detects according to desire is confirmed target pixel value;
Calculate the ratio in the combination of pixels that said target pixel value accounts for said selection area image.
Further, said method also can comprise, said calculating target pixel value accounts for the step of the ratio in the combination of pixels of said selection area image, comprising:
The value of each pixel in the said selection area image is kept in the array, travels through this array, the target pixel value that calculates in this array accounts for the total ratio of array.
Further, said method also can comprise, saidly confirms to comprise the step of the information of the interface control in this selection area according to comparative result:
The target pixel value proportion that comprises according to prior acquisition in the area image of control to be detected is confirmed proportion threshold value;
When said target pixel value accounts for ratio in the combination of pixels of said selection area image and reaches said threshold value, then confirm control to be detected to have occurred in this selection area.
Further, said method also can comprise, calculates the distribution characteristics of said target pixel value in said selection area image;
According to said distribution characteristics, confirm the position of said control to be detected in said selection area.
Further, said method also can comprise, saidly obtains the step of the pixel characteristic of this area image by preset strategy, comprising:
Data to the combination of pixels of said selection area image forms are carried out the uniqueness computing, obtain the uniqueness result.
Further, said method can comprise that also the step that the said data that the combination of pixels of said selection area image is formed are carried out the uniqueness computing comprises:
The value of each pixel in the said selection area image is kept in the array, the data in the array are carried out the uniqueness computing as a binary data string.
Further, said method also can comprise, saidly confirms to comprise the step of the information of the interface control in this selection area according to comparative result:
Obtain in advance to comprise the data that the combination of pixels in the area image of control to be detected forms according to preset strategy and carry out the uniqueness computing, and obtain the uniqueness result;
When the corresponding uniqueness result of said selection area image is consistent with the said corresponding uniqueness result of area image who comprises control to be detected, then confirm control to be detected to have occurred in this selection area.
Further, said method also can comprise, according to the control position in the area image that comprises control to be detected of said prior acquisition, confirms the control position in the said selection area image.
Further; Said method also can comprise; If confirm control to be detected not occur in the current selected zone, then with the current selected zone line by line or by after pixel of row translation as new current selected zone, obtain the pixel characteristic of this selection area image once more.
Further; Said method also can comprise, and is said according to preset strategy, a selected zone in the interface; And intercepting should the zone the step of image; Comprise: judge if control to be detected reaches predetermined threshold value in frequency that certain zone occurs, then according to selected zone in the interface, this zone, and the image that intercepting should the zone.
Further, said method can comprise that also the said step of confirming the information of the interface control in this selection area further comprises: the status information of judging the selection area inner boundary control that is obtained.
Further, said method can comprise that also the status information of selection area inner boundary control is obtained in said judgement, comprising: whether check box is chosen or control information available whether.
The present invention also provides a kind of system that obtains interface control information, comprising:
Chosen module is used for according to preset strategy, selected zone in the interface, and the image interception in should the zone comes out, wherein said selection area and prior acquisition comprise the regional consistent of control to be detected;
The characteristic acquisition module is used for the combination of pixels according to the image of said selection area, obtains the pixel characteristic of this selection area image by preset strategy;
Comparison module is used for the comparative result according to pixel characteristic with the pixel characteristic of the area image that comprises control to be detected that obtains in advance by preset strategy of said selection area image, confirms the information of the interface control in this selection area.
Further, said system also can comprise, said characteristic acquisition module, and the color of the control that detects according to desire is confirmed target pixel value, and the ratio in the combination of pixels that said target pixel value accounts for said selection area image calculated is as said pixel characteristic.
Further; Said system also can comprise; Said comparison module; The ratio that is used for accounting for the said area image that comprises control to be detected at said target pixel value, is then confirmed control to be detected to have occurred in this selection area in preset range the time with the difference that accounts for the ratio in the said selection area image.
Further, said system also can comprise, said characteristic acquisition module is used for the data that the combination of pixels of said selection area image forms are carried out the uniqueness computing, obtains the uniqueness result as pixel characteristic.
Further; Said system also can comprise; Said comparison module; If the consistance of uniqueness operation result of combination of pixels of uniqueness operation result and said selection area image of combination of pixels that is used for the area image that comprises control to be detected of more said prior acquisition is unanimity then confirm control to be detected to have occurred in this selection area.
Compared with prior art; Use the present invention; Through analyzing the pixel characteristic of non-standard window control, and compare, and then can judge the control that whether has occurred needs in the window in the selection area with the pixel characteristic of the area image that comprises control to be detected that obtains in advance by preset strategy in the images displayed of screen; And can obtain the status information of control in the selection area; Realized non-standard window control being carried out the purpose of automatic test, reduced people's working strength, had realistic meaning through unified method.
Description of drawings
Fig. 1 is the synoptic diagram of the graphical interfaces of current one group of non-standard control;
Fig. 2 is the schematic flow sheet that obtains the method for interface control information of the present invention;
Fig. 3 is the schematic flow sheet of an embodiment of the present invention " picture element density scan method ";
Fig. 4 is the synoptic diagram of the graphical interfaces that is characterized as certain special color of non-standard control of the present invention;
Fig. 5 is in the same area of the present invention and the synoptic diagram of the graphical interfaces of the identical non-standard control of main body color;
Fig. 6 be of the present invention in a bigger zone that comprises selected window area traversal search the process synoptic diagram in the zone identical with the digital digest of required control;
Fig. 7 is the synoptic diagram that obtains the system of interface control information of the present invention.
Embodiment
Below in conjunction with accompanying drawing and embodiment the present invention is described further.
The application's method is based upon screen coordinate system and window coordinates are on the basis.
Relate to following notion among the application:
Screen coordinate system is a two-dimensional coordinate system, and the screen upper left corner is coordinate origin, and origin is designated as (0; 0), abscissa axis overlaps with the screen upper side frame, and from left to right coordinate figure increases progressively; Coordinate unit is a pixel; Axis of ordinates overlaps with the right frame, and coordinate figure increases progressively from top to bottom, and coordinate unit is a pixel.
Window coordinates system is similar with screen coordinate system, and horizontal ordinate all is parallel to the screen frame, and the window upper left corner is coordinate origin; Origin is designated as (0,0), and abscissa axis overlaps with the upper side frame of window; From left to right coordinate figure increases progressively, and coordinate unit is a pixel, and axis of ordinates overlaps with the window left frame; Coordinate figure increases progressively from top to bottom, and coordinate unit is a pixel.Obviously window coordinates can convert screen coordinate to.
What the application related to all is the rectangular area, and the zone that hereinafter is mentioned all refers to the rectangular area.Can describe a rectangular area with the coordinate of two points in the window coordinates system, these two points are respectively the regional upper left corner and the lower right corner.Each zone can be expressed as shape as for (x1, y1, x2, four-tuple y2), wherein (x1 y1) be upper left corner coordinate, and (x2 y2) be lower right corner coordinate, and establishing each zone is the wide X of being of unit with the pixel, and height is Y, and afterwards civilian false code will be used these settings.
Test macro the selection area record in the configuration file of system that will analyze because the position of window on screen is uncertain, so what in configuration file, write down is the position of control in window coordinates system.When test macro moves, convert actual screen coordinate again to.
MD5: English full name Message Digest Algorithm 5, Chinese is defined as Message Digest 5 the 5th edition, and the MD5 algorithm through its irreversible character string mapping algorithm, has produced the unique MD5 informative abstract of this segment information with a piece of digital information.Different information is not the same through the summary that MD5 calculates, specifically can be with reference to RFC 1321 agreements.
Control: the ingredient of Windows graphical interfaces; Can accept the input of input equipments such as keyboard or mouse; Thereby accomplish certain particular functionality, common control has button, OptionButton, edit box, list box, scroll bar and static control etc. in the Windows system.
Pixel: the minimum of image in computing machine formed unit, and pixel is appreciated that the numerical value into certain any color in the image.
Because the implementation method of non-standard window control is had nothing in common with each other, so can not start with from the internal operation mechanism of control, the application looks for another way, and obtains information in the control through analyzing non-standard window control in the images displayed of screen.
The application can obtain information in the non-standard window control of Windows graphical interfaces, and then can judge the control that whether has occurred needs in the rectangular area selected in the window; Perhaps judge the status information obtain control in the selection area, comprise like check box whether being chosen or information such as whether available control.
As shown in Figure 2, a kind of method of obtaining interface control information of the present invention may further comprise the steps:
The selection area record in the configuration file of system that will analyze because the position of window on screen is uncertain, so what in configuration file, write down is the position of control in window coordinates system.
Specifically comprise: a selected zone in the interface converts the positional information of the required control that writes down in the configuration file in window coordinates system the positional information of actual screen coordinate to, and stores.
Said according to preset strategy, a selected zone in the interface, and the step of the image that intercepting should the zone; Specifically comprise: judge if control to be detected reaches predetermined threshold value in the frequency that certain zone occurs; Then according to selected zone in the interface, this zone, and the image that intercepting should the zone, wherein predetermined threshold value for example is set to: the frequency that control to be detected occurs is 50 times or 100 inferior; Decide with actual conditions, the present invention does not limit.
It should be noted that: the scope of said selection area is consistent with the scope in the zone that comprises control to be detected of said prior acquisition, and promptly can to show as area identical in two zones.
Wherein, the pixel characteristic of selection area image can be the pixel characteristic of certain position in this selection area image, or the pixel characteristic of whole selection area image.
Particularly, can may further comprise the steps:
The color of the control that detects according to desire is confirmed target pixel value;
Calculate the ratio in the combination of pixels that said target pixel value accounts for said selection area image.
Further comprise:
Calculate the distribution characteristics of said target pixel value in said selection area image;
According to said distribution characteristics, confirm the position of said control to be detected in said selection area.
Doing when detecting to control whether occurring, also can detect the particular location of appearance, can confirm the control position in the said selection area image according to the control position in the area image that comprises control to be detected of said prior acquisition.
Wherein, said calculating target pixel value accounts for the step of the ratio in the combination of pixels of said selection area image, comprising:
The value of each pixel in the said selection area image is kept in the array, travels through this array, the target pixel value that calculates in this array accounts for the total ratio of array.
Perhaps, saidly obtaining the pixel characteristic of this area image by preset strategy, is to carry out the uniqueness computing through the data that the combination of pixels to said selection area image forms, and obtains the uniqueness result.
Wherein, the step that the said data that the combination of pixels of said selection area image is formed are carried out the uniqueness computing comprises:
The value of each pixel in the said selection area image is kept in the array, the data in the array are carried out the uniqueness computing as a binary data string.
Saidly confirm to comprise the step of the information of the interface control in this selection area according to comparative result:
The target pixel value proportion that comprises according to prior acquisition in the area image of control to be detected is confirmed proportion threshold value;
When said target pixel value accounts for ratio in the combination of pixels of said selection area image and reaches said threshold value, then confirm control to be detected to have occurred in this selection area.
Perhaps saidly confirm to comprise the step of the information of the interface control in this selection area according to comparative result:
Obtain in advance to comprise the data that the combination of pixels in the area image of control to be detected forms according to preset strategy and carry out the uniqueness computing, and obtain the uniqueness result;
When the corresponding uniqueness result of said selection area image is consistent with the said corresponding uniqueness result of area image who comprises control to be detected, then confirm control to be detected to have occurred in this selection area.
Also can further comprise:
If confirm control to be detected not occur in the current selected zone, then with the current selected zone line by line or by after pixel of row translation as new current selected zone, obtain the pixel characteristic of this selection area image once more.
Method one embodiment that obtains interface control information of the present invention, as shown in Figure 3, may further comprise the steps:
Step 310, the selected window area of test macro;
Test macro the selection area record in the configuration file of system that will analyze because the position of window on screen is uncertain, so what in configuration file, write down is the position of control in window coordinates system.
Specifically comprise: said test macro is selected window area, and the positional information of the required control that writes down in the configuration file with said test macro in window coordinates system converts the positional information of actual screen coordinate to, and stores.
The image of the window area that step 320, extraction are selected is analyzed each control color of pixel value in this image;
The interior image in zone test macro in the window is selected through the mode of the similar Flame Image Process class libraries of Python Image Library extracts.
The image that extracts is made up of pixel, a point in each pixel correspondence image, this color of the value representation of pixel; Like yellow corresponding value; Red corresponding another value, specifically referring to RFC 797 agreements, can be with 32 values that integer comes recording pixel; The present invention is kept at the pixel that these are extracted out in the array, so that subsequent analysis.
Step 330, judge whether certain color value reaches the preset ratio value in certain control pixel of this image, if then selected this control is required control.
Concrete steps are following: test macro is kept at each color of pixel value in the image of said selected window area in the array; Travel through this array; When the ratio that accounts for array sum when certain color value in this array reaches the preset ratio value, confirm that then this control is selected control.
In the reality test,, just need to judge in the window of paying close attention to if test macro need carry out certain operation to certain window control; Whether there is this control in the zone that control should occur; If this control be characterized as certain special color, as shown in Figure 4, yellow color is that " agreeing also to install " button is different from the color of other control; Test macro should its zone that should occur of first intercepting image; Each color of pixel value in the image is kept in the array, and this array of test macro traversal is when the ratio that accounts for the array sum when the yellow color value in the array acquires a certain degree; Just think " agreeing and installation " control in this zone, to have occurred, otherwise this button does not occur.
This procedural mode is following:
This method is called " picture element density scan method " in the present invention; Its parameter rect is the selected zone of test macro; FeatureColor refers to the main body color of the control that test macro is paid close attention to, and threshold is that the ratio of all pixel counts in the main body colored pixels is counted the territory, occupied area can be confirmed the threshold value that control exists for how many times.
Further, the limitation of " picture element density scan method " is to distinguish and can appears in the same area and the identical again control of main body color, and in order to address this problem, the present invention proposes the embodiment of " MD5 digest determination methods " again.
As shown in Figure 5: " unlatching " button appears on the identical position with the Close button; And be equally with the black of literal as the sign color, so take on same position that " picture element density scan method " just can not judge current appearance be X button or release the button.
The step of " MD5 digest determination methods " is following: if in same area and the identical again control of main body color when being a plurality of; Promptly judge certain color value in a plurality of control pixels of this image if reach the preset ratio value; Then can discern selected required control according to the digital digest of different controls to control;
To above-mentioned situation, according to the digital digest of different controls control to be discerned, selected required control specifically is to realize through following steps:
After test macro obtains the array of pixels of certain control regions in the said selected window; With the data in the array as a binary data string; Through the serial data that obtains is calculated, obtain the digital digest of this serial data, according to digital digest different controls is discerned.
Wherein mainly be MD5 digest through available control region image as the sign of control, test macro has been obtained in the window after certain regional array of pixels, with the data in the array as a binary data string; With the MD5 digest method this serial data is calculated, can draw the digital digest of serial data, because the image of different controls is different; Therefore its digital digest also is different; Can they be made a distinction thus, selected required control is taked to control accordingly.
Wherein the digital digest of control image in selection area can be calculated in advance, and the sign summary as control is kept in the database of test macro.
Test macro is made a summary the image digitization summary of selection area and is compared with the sign of control, if consistent, explain that then control appears in the selected zone.
It should be noted that: the same control that controls possibly be shown with multiple different appearance, so the sign of control is plucked if it were not for single, can be one group of summary.
Further; The limitation of above-mentioned " MD5 digest determination methods " is when the position of control changes in selection area; Even control has appeared in the selected zone really, this regional image digitization summary also can't be complementary with the digital digest of control.For this situation, the invention provides solution, promptly in a bigger zone that comprises selection area, search the subregion identical with the digital digest of control, as shown in Figure 6:
This method comprises: if the digital digest of control image can't be complementary with the digital digest of required control in the image of the window area that the judgement of said test macro is selected; Then traversal is searched the zone identical with the digital digest of required control in a bigger zone that comprises said selected window area; If find, then confirm should the zone in control be required control.
Concrete grammar: said test macro enlarges certain scope with former selected window area, and this extended area scope can comprise former selected window area, begins from the extended area upper left corner; Resample area to identical with the length and width of former selected window area is calculated digital digest; If judge that the digital digest of resample area is identical with the sign summary of required control, confirm that then required control exists, otherwise; With resample area to pixel of left; Overlap with extended area if judge the right margin of resample area, then with pixel of the downward translation of resample area, end coincides up to the lower right corner of the lower right corner of resample area and extended area.
If the digital digest of resample area is identical with the digital digest of control in this process, confirm that then control exists, otherwise this process will last till that the lower right corner of resample area and the lower right corner of extended area coincide and could finish.
This procedural mode is following:
Wherein, rect is former selection area, and ctrlMd5 is the sign summary of control.
" picture element density scanning method " and " MD5 digest determining method " can be judged the state of control equally.
When confirmed that control occurs in selected zone after, judge the state of control according to the MD5 digest in zone.The value current like drop-down tabulation, whether check box is selected medium.
Radio button before selected with selected after image be different; As the image before choosing is zero ZIP; Its MD5 digest is e1fb6b1310405f94cc92ce5ea2a8347d, and the image after choosing is ⊙ ZIP, its MD5 digest: 68c5bf356ad8a4f7508df4040acdff18.
It should be noted that: in the present invention, mainly be, can adopt several different methods to come the eigenwert of computed image through to after image pixel is analyzed." picture element density scan method " and " MD5 digest determination methods " is two kinds of implementations wherein; Can certainly obtain interface control information through other embodiments, such as: calculate embodiments such as array HASH, edge pixel coupling or pixel inversion, the present invention does not limit this.Aforesaid way all is through analysis and comparison to the interface image characteristic, finally obtains interface control information.
As shown in Figure 7, the present invention also provides a kind of system that obtains interface control information, comprising: chosen module 701, characteristic acquisition module 702 and comparison module 703, wherein,
Chosen module 701 is used for according to preset strategy, selected zone in the interface, and the image interception in should the zone comes out, wherein said selection area and prior acquisition comprise the regional consistent of control to be detected;
Comparison module 703 is used for the comparative result according to pixel characteristic with the pixel characteristic of the area image that comprises control to be detected that obtains in advance by preset strategy of said selection area image, confirms the information of the interface control in this selection area.
Wherein, said characteristic acquisition module 702, the color of the control that detects according to desire is confirmed target pixel value, and the ratio in the combination of pixels that said target pixel value accounts for said selection area image calculated is as said pixel characteristic.
Said comparison module 703, the ratio that is used for accounting for the said area image that comprises control to be detected at said target pixel value, are then confirmed control to be detected to have occurred in this selection area in preset range the time with the difference that accounts for the ratio in the said selection area image.
In this embodiment, the present invention is described comparing process from another angle.
Said characteristic acquisition module 702 is used for the data that the combination of pixels of said selection area image forms are carried out the uniqueness computing, obtains the uniqueness result as pixel characteristic.
Said comparison module 703; If the consistance of uniqueness operation result of combination of pixels of uniqueness operation result and said selection area image of combination of pixels that is used for the area image that comprises control to be detected of more said prior acquisition is unanimity then confirm control to be detected to have occurred in this selection area.
The above; Be merely the preferable embodiment of the present invention, but protection scope of the present invention is not limited thereto, anyly is familiar with this technological people in the technical scope that the present invention disclosed; The variation that can expect easily or replacement all should be encompassed within protection scope of the present invention.Therefore, protection scope of the present invention should be as the criterion with the protection domain of claim.
Claims (18)
1. a method of obtaining interface control information is characterized in that, comprising:
According to preset strategy, a selected zone in the interface, and the image that intercepting should the zone, wherein said selection area and acquisition in advance comprise the regional consistent of control to be detected;
According to the combination of pixels of the image of said selection area, obtain the pixel characteristic of this selection area image by preset strategy;
According to the pixel characteristic of said selection area image comparative result, confirm the information of the interface control in this selection area with the pixel characteristic of the area image that comprises control to be detected that obtains in advance by preset strategy.
2. the method for claim 1 is characterized in that, saidly obtains the step of the pixel characteristic of this area image by preset strategy, comprising:
The color of the control that detects according to desire is confirmed target pixel value;
Calculate the ratio in the combination of pixels that said target pixel value accounts for said selection area image.
3. method as claimed in claim 2 is characterized in that, said calculating target pixel value accounts for the step of the ratio in the combination of pixels of said selection area image, comprising:
The value of each pixel in the said selection area image is kept in the array, travels through this array, the target pixel value that calculates in this array accounts for the total ratio of array.
4. method as claimed in claim 2 is characterized in that, saidly confirms to comprise the step of the information of the interface control in this selection area according to comparative result:
The target pixel value proportion that comprises according to prior acquisition in the area image of control to be detected is confirmed proportion threshold value;
When said target pixel value accounts for ratio in the combination of pixels of said selection area image and reaches said threshold value, then confirm control to be detected to have occurred in this selection area.
5. method as claimed in claim 2 is characterized in that, further comprises:
Calculate the distribution characteristics of said target pixel value in said selection area image;
According to said distribution characteristics, confirm the position of said control to be detected in said selection area.
6. according to claim 1 or claim 2 method is characterized in that, saidly obtains the step of the pixel characteristic of this area image by preset strategy, comprising:
Data to the combination of pixels of said selection area image forms are carried out the uniqueness computing, obtain the uniqueness result.
7. method as claimed in claim 6 is characterized in that, the step that the said data that the combination of pixels of said selection area image is formed are carried out the uniqueness computing comprises:
The value of each pixel in the said selection area image is kept in the array, the data in the array are carried out the uniqueness computing as a binary data string.
8. method as claimed in claim 6 is characterized in that, saidly confirms to comprise the step of the information of the interface control in this selection area according to comparative result:
Obtain in advance to comprise the data that the combination of pixels in the area image of control to be detected forms according to preset strategy and carry out the uniqueness computing, and obtain the uniqueness result;
When the corresponding uniqueness result of said selection area image is consistent with the said corresponding uniqueness result of area image who comprises control to be detected, then confirm control to be detected to have occurred in this selection area.
9. like claim 4 or 8 described methods, it is characterized in that, further comprise:
According to the control position in the area image that comprises control to be detected of said prior acquisition, confirm the control position in the said selection area image.
10. like claim 4 or 8 described methods, it is characterized in that, further comprise:
If confirm control to be detected not occur in the current selected zone, then with the current selected zone line by line or by after pixel of row translation as new current selected zone, obtain the pixel characteristic of this selection area image once more.
11. the method for claim 1; It is characterized in that, said according to preset strategy, a selected zone in the interface; And intercepting should the zone the step of image; Comprise: judge if control to be detected reaches predetermined threshold value in frequency that certain zone occurs, then according to selected zone in the interface, this zone, and the image that intercepting should the zone.
12. the method for claim 1 is characterized in that, the said step of confirming the information of the interface control in this selection area further comprises: the status information of judging the selection area inner boundary control that is obtained.
13. method as claimed in claim 12 is characterized in that, the status information of selection area inner boundary control is obtained in said judgement, comprising: whether check box is chosen or control information available whether.
14. a system that obtains interface control information is characterized in that, comprising:
Chosen module is used for according to preset strategy, selected zone in the interface, and the image interception in should the zone comes out, wherein said selection area and prior acquisition comprise the regional consistent of control to be detected;
The characteristic acquisition module is used for the combination of pixels according to the image of said selection area, obtains the pixel characteristic of this selection area image by preset strategy;
Comparison module is used for the comparative result according to pixel characteristic with the pixel characteristic of the area image that comprises control to be detected that obtains in advance by preset strategy of said selection area image, confirms the information of the interface control in this selection area.
15. system as claimed in claim 14; It is characterized in that; Said characteristic acquisition module, the color of the control that detects according to desire is confirmed target pixel value, and the ratio in the combination of pixels that said target pixel value accounts for said selection area image calculated is as said pixel characteristic.
16. system as claimed in claim 15; It is characterized in that; Said comparison module; The ratio that is used for accounting for the said area image that comprises control to be detected at said target pixel value, is then confirmed control to be detected to have occurred in this selection area in preset range the time with the difference that accounts for the ratio in the said selection area image.
17., it is characterized in that said characteristic acquisition module is used for the data that the combination of pixels of said selection area image forms are carried out the uniqueness computing, obtains the uniqueness result as pixel characteristic like claim 14 or 15 described systems.
18. system as claimed in claim 17; It is characterized in that; Said comparison module; If the consistance of uniqueness operation result of combination of pixels of uniqueness operation result and said selection area image of combination of pixels that is used for the area image that comprises control to be detected of more said prior acquisition is unanimity then confirm control to be detected to have occurred in this selection area.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110161998.XA CN102830958B (en) | 2011-06-16 | 2011-06-16 | A kind of method and system for obtaining interface control information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110161998.XA CN102830958B (en) | 2011-06-16 | 2011-06-16 | A kind of method and system for obtaining interface control information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102830958A true CN102830958A (en) | 2012-12-19 |
| CN102830958B CN102830958B (en) | 2017-11-24 |
Family
ID=47334110
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110161998.XA Active CN102830958B (en) | 2011-06-16 | 2011-06-16 | A kind of method and system for obtaining interface control information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102830958B (en) |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103135986A (en) * | 2013-01-25 | 2013-06-05 | 中国人民解放军装甲兵工程学院 | Method and system for operation collection |
| CN103645890A (en) * | 2013-11-29 | 2014-03-19 | 北京奇虎科技有限公司 | Method and device for positioning control part in graphical user interface |
| CN104252410A (en) * | 2013-06-27 | 2014-12-31 | 阿里巴巴集团控股有限公司 | Method and equipment for testing control in page |
| CN104391797A (en) * | 2014-12-09 | 2015-03-04 | 北京奇虎科技有限公司 | GUI (graphical user interface) widget identification method and device |
| CN104462979A (en) * | 2014-12-26 | 2015-03-25 | 深圳数字电视国家工程实验室股份有限公司 | Automatic dynamic detection method and device of application program |
| CN104598282A (en) * | 2015-02-05 | 2015-05-06 | 广州华多网络科技有限公司 | Control capture method and equipment |
| CN104615543A (en) * | 2015-02-12 | 2015-05-13 | 腾讯科技（深圳）有限公司 | Abnormal window program response detecting method and device |
| CN104636029A (en) * | 2014-12-31 | 2015-05-20 | 魅族科技（中国）有限公司 | Display control method and system for control |
| CN104766198A (en) * | 2015-03-03 | 2015-07-08 | 包敬海 | Method for rapidly recognizing mail attachment |
| CN104899146A (en) * | 2015-06-19 | 2015-09-09 | 安一恒通（北京）科技有限公司 | Image matching technology based software stability test method and device |
| WO2015165245A1 (en) * | 2014-04-30 | 2015-11-05 | 广州市动景计算机科技有限公司 | Webpage data processing method and device |
| CN105095236A (en) * | 2014-04-30 | 2015-11-25 | 优视科技有限公司 | Advertisement filtering method and device |
| CN105740145A (en) * | 2016-01-27 | 2016-07-06 | 广州酷狗计算机科技有限公司 | Method and device for locating element in control |
| CN106228191A (en) * | 2016-07-28 | 2016-12-14 | 努比亚技术有限公司 | A kind of control text overlength detection device and method |
| CN106998489A (en) * | 2017-03-30 | 2017-08-01 | 武汉斗鱼网络科技有限公司 | A kind of focus is crossed the border searching method and device |
| CN109522086A (en) * | 2018-11-29 | 2019-03-26 | 金蝶软件（中国）有限公司 | To the operating method and device of window, computer installation and readable storage medium storing program for executing |
| CN109684225A (en) * | 2018-12-29 | 2019-04-26 | 广州云测信息技术有限公司 | A kind of method for testing software and device |
| CN109753435A (en) * | 2018-12-29 | 2019-05-14 | 广州云测信息技术有限公司 | A kind of method for testing software and device |
| CN110688308A (en) * | 2019-09-19 | 2020-01-14 | 广州虎牙科技有限公司 | Memory storage redundancy detection method and device, electronic equipment and readable storage medium |
| CN110765825A (en) * | 2018-07-27 | 2020-02-07 | 北京京东尚科信息技术有限公司 | Method and system for acquiring article placement state |
| CN111488499A (en) * | 2020-03-06 | 2020-08-04 | 北京健康之家科技有限公司 | Public opinion data processing method and device |
| CN112163579A (en) * | 2020-09-30 | 2021-01-01 | 江苏安全技术职业学院 | Student thought report analysis system and method based on text semantic feature analysis |
| CN114020267A (en) * | 2021-11-04 | 2022-02-08 | 四川启睿克科技有限公司 | Method for detecting RPA window control |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006268666A (en) * | 2005-03-25 | 2006-10-05 | Fujitsu Ltd | Automatic test system with correction function, automatic test method, and program |
| CN101630286A (en) * | 2009-08-07 | 2010-01-20 | 金蝶软件（中国）有限公司 | Automatic testing method and automatic testing device |
| CN101770578A (en) * | 2010-03-24 | 2010-07-07 | 上海交通大学 | Image characteristic extraction method |
| CN101802824A (en) * | 2007-09-20 | 2010-08-11 | 诺基亚公司 | Method, apparatus and computer program product for providing a visual search interface |
| CN101853157A (en) * | 2010-05-14 | 2010-10-06 | 中国科学院软件研究所 | Automatic identification method for application software GUI object |
| CN101866352A (en) * | 2010-05-28 | 2010-10-20 | 广东工业大学 | Design patent retrieval method based on analysis of image content |
| US20110099499A1 (en) * | 2009-10-26 | 2011-04-28 | Ayelet Pnueli | Graphical user interface component identification |
| US20110126158A1 (en) * | 2009-11-23 | 2011-05-26 | University Of Washington | Systems and methods for implementing pixel-based reverse engineering of interface structure |
| US20110131551A1 (en) * | 2009-12-02 | 2011-06-02 | Nitsan Amichai | Graphical user interface input element identification |
-
2011
- 2011-06-16 CN CN201110161998.XA patent/CN102830958B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006268666A (en) * | 2005-03-25 | 2006-10-05 | Fujitsu Ltd | Automatic test system with correction function, automatic test method, and program |
| CN101802824A (en) * | 2007-09-20 | 2010-08-11 | 诺基亚公司 | Method, apparatus and computer program product for providing a visual search interface |
| CN101630286A (en) * | 2009-08-07 | 2010-01-20 | 金蝶软件（中国）有限公司 | Automatic testing method and automatic testing device |
| US20110099499A1 (en) * | 2009-10-26 | 2011-04-28 | Ayelet Pnueli | Graphical user interface component identification |
| US20110126158A1 (en) * | 2009-11-23 | 2011-05-26 | University Of Washington | Systems and methods for implementing pixel-based reverse engineering of interface structure |
| US20110131551A1 (en) * | 2009-12-02 | 2011-06-02 | Nitsan Amichai | Graphical user interface input element identification |
| CN101770578A (en) * | 2010-03-24 | 2010-07-07 | 上海交通大学 | Image characteristic extraction method |
| CN101853157A (en) * | 2010-05-14 | 2010-10-06 | 中国科学院软件研究所 | Automatic identification method for application software GUI object |
| CN101866352A (en) * | 2010-05-28 | 2010-10-20 | 广东工业大学 | Design patent retrieval method based on analysis of image content |
Non-Patent Citations (4)
| Title |
|---|
| TOM YEH ET AL.: "Sikuli: using GUI screenshots for search and automation", 《PROCEEDING UIST "09 PROCEEDINGS OF THE 22ND ANNUAL ACM SYMPOSIUM ON USER INTERFACE SOFTWARE AND TECHNOLOGY》 * |
| 厉旻: "自动化测试中基于GUI非标准控件的识别和应用", 《微计算机信息》 * |
| 叶志伟 等: "一种基于MD5的近似图像检测算法研究", 《计算机应用研究》 * |
| 毛力 等: "基于颜色内容的图像检索原理与方法", 《情报科学》 * |
Cited By (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103135986A (en) * | 2013-01-25 | 2013-06-05 | 中国人民解放军装甲兵工程学院 | Method and system for operation collection |
| CN103135986B (en) * | 2013-01-25 | 2015-11-25 | 中国人民解放军装甲兵工程学院 | A kind of operation acquisition method and system |
| CN104252410A (en) * | 2013-06-27 | 2014-12-31 | 阿里巴巴集团控股有限公司 | Method and equipment for testing control in page |
| CN103645890A (en) * | 2013-11-29 | 2014-03-19 | 北京奇虎科技有限公司 | Method and device for positioning control part in graphical user interface |
| CN103645890B (en) * | 2013-11-29 | 2017-12-01 | 北京奇虎科技有限公司 | A kind of method and apparatus for being used for orient control in graphical user interfaces |
| CN105095236A (en) * | 2014-04-30 | 2015-11-25 | 优视科技有限公司 | Advertisement filtering method and device |
| WO2015165245A1 (en) * | 2014-04-30 | 2015-11-05 | 广州市动景计算机科技有限公司 | Webpage data processing method and device |
| CN104391797A (en) * | 2014-12-09 | 2015-03-04 | 北京奇虎科技有限公司 | GUI (graphical user interface) widget identification method and device |
| CN104391797B (en) * | 2014-12-09 | 2017-11-10 | 北京奇虎科技有限公司 | The recognition methods of GUI controls and device |
| CN104462979A (en) * | 2014-12-26 | 2015-03-25 | 深圳数字电视国家工程实验室股份有限公司 | Automatic dynamic detection method and device of application program |
| CN104636029A (en) * | 2014-12-31 | 2015-05-20 | 魅族科技（中国）有限公司 | Display control method and system for control |
| CN104598282A (en) * | 2015-02-05 | 2015-05-06 | 广州华多网络科技有限公司 | Control capture method and equipment |
| CN104598282B (en) * | 2015-02-05 | 2017-10-27 | 广州华多网络科技有限公司 | A kind of control method for catching and equipment |
| CN104615543A (en) * | 2015-02-12 | 2015-05-13 | 腾讯科技（深圳）有限公司 | Abnormal window program response detecting method and device |
| CN104766198A (en) * | 2015-03-03 | 2015-07-08 | 包敬海 | Method for rapidly recognizing mail attachment |
| CN104899146A (en) * | 2015-06-19 | 2015-09-09 | 安一恒通（北京）科技有限公司 | Image matching technology based software stability test method and device |
| CN104899146B (en) * | 2015-06-19 | 2018-04-24 | 安一恒通（北京）科技有限公司 | Software stability test method and device based on image matching technology |
| CN105740145A (en) * | 2016-01-27 | 2016-07-06 | 广州酷狗计算机科技有限公司 | Method and device for locating element in control |
| CN105740145B (en) * | 2016-01-27 | 2018-09-18 | 广州酷狗计算机科技有限公司 | The method and device of element in orient control |
| CN106228191A (en) * | 2016-07-28 | 2016-12-14 | 努比亚技术有限公司 | A kind of control text overlength detection device and method |
| CN106998489B (en) * | 2017-03-30 | 2018-07-06 | 武汉斗鱼网络科技有限公司 | A kind of focus is crossed the border searching method and device |
| CN106998489A (en) * | 2017-03-30 | 2017-08-01 | 武汉斗鱼网络科技有限公司 | A kind of focus is crossed the border searching method and device |
| CN110765825A (en) * | 2018-07-27 | 2020-02-07 | 北京京东尚科信息技术有限公司 | Method and system for acquiring article placement state |
| US11861910B2 (en) | 2018-07-27 | 2024-01-02 | Beijing Jingdong Shangke Information Technology Co., Ltd. | Method and system for acquiring item placing state |
| CN109522086A (en) * | 2018-11-29 | 2019-03-26 | 金蝶软件（中国）有限公司 | To the operating method and device of window, computer installation and readable storage medium storing program for executing |
| CN109522086B (en) * | 2018-11-29 | 2022-07-12 | 金蝶软件（中国）有限公司 | Operation method and device for window, computer device and readable storage medium |
| CN109684225A (en) * | 2018-12-29 | 2019-04-26 | 广州云测信息技术有限公司 | A kind of method for testing software and device |
| CN109753435A (en) * | 2018-12-29 | 2019-05-14 | 广州云测信息技术有限公司 | A kind of method for testing software and device |
| CN110688308A (en) * | 2019-09-19 | 2020-01-14 | 广州虎牙科技有限公司 | Memory storage redundancy detection method and device, electronic equipment and readable storage medium |
| CN111488499A (en) * | 2020-03-06 | 2020-08-04 | 北京健康之家科技有限公司 | Public opinion data processing method and device |
| CN112163579A (en) * | 2020-09-30 | 2021-01-01 | 江苏安全技术职业学院 | Student thought report analysis system and method based on text semantic feature analysis |
| CN114020267A (en) * | 2021-11-04 | 2022-02-08 | 四川启睿克科技有限公司 | Method for detecting RPA window control |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102830958B (en) | 2017-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102830958A (en) | Method and system for obtaining interface control information | |
| EP3480735B1 (en) | Inspection apparatus, data generation apparatus, data generation method, and data generation program | |
| Dixon et al. | Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure | |
| US10725892B2 (en) | Method and system for automated testing of a graphical user interface for customer devices | |
| CN102419663B (en) | A kind of infrared touch screen multi-point recognition method and system | |
| CN101964062B (en) | Image searching method based on user interface | |
| US9804955B2 (en) | Method and apparatus for creating reference images for an automated test of software with a graphical user interface | |
| CN110851299A (en) | Automatic flow exception eliminating method, device, equipment and storage medium | |
| EP2966652B1 (en) | System and method for nuclear fuel assembly deformation measurement | |
| CN108121648B (en) | Interface error monitoring method | |
| CN116052193B (en) | RPA interface dynamic form picking and matching method and system | |
| CN105354138A (en) | Method for determining specific control attribute of mobile phone page | |
| CN102473278A (en) | Image processing apparatus, image processing method, and storage medium | |
| CN106447656A (en) | Rendering flawed image detection method based on image recognition | |
| CN113378839A (en) | Information processing apparatus, information processing method, and computer readable medium | |
| KR102653177B1 (en) | Apparatus and method for extracting object information | |
| CN110858302A (en) | System and method for editing monitoring script in picture image | |
| CN105446676A (en) | Method and apparatus for carrying out large screen display | |
| US11308352B2 (en) | Apparatus for processing labeled data to be used in learning of discriminator, method of controlling the apparatus, and non-transitory computer-readable recording medium | |
| CN103688528B (en) | By width and the method for position of image information detection document in large-scale scanning system | |
| CN113434072B (en) | Mobile terminal application control identification method based on computer vision | |
| US20220383483A1 (en) | Segmentation of Continuous Dynamic Scans | |
| WO2012153456A1 (en) | Defect review apparatus | |
| CN109445913B (en) | Resource object leakage detection method and device | |
| CN114187658A (en) | Input operation identification method and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 100015 rooms 301-306, 2 / F and 3 / F, block B, building 2, yard 6, Jiuxianqiao Road, Chaoyang District, Beijing Patentee after: Beijing Qizhi Business Consulting Co.,Ltd. Address before: 100016 East unit, 4th floor, Zhaowei building, 14 Jiuxianqiao Road, Chaoyang District, Beijing Patentee before: Qizhi software (Beijing) Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220209 Address after: Room 801, 8th floor, No. 104, floors 1-19, building 2, yard 6, Jiuxianqiao Road, Chaoyang District, Beijing 100015 Patentee after: BEIJING QIHOO TECHNOLOGY Co.,Ltd. Address before: 100015 rooms 301-306, 2 / F and 3 / F, block B, building 2, yard 6, Jiuxianqiao Road, Chaoyang District, Beijing Patentee before: Beijing Qizhi Business Consulting Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220802 Address after: 300450 No. 9-3-401, No. 39, Gaoxin 6th Road, Binhai Science Park, Binhai New Area, Tianjin Patentee after: 3600 Technology Group Co.,Ltd. Address before: Room 801, 8th floor, No. 104, floors 1-19, building 2, yard 6, Jiuxianqiao Road, Chaoyang District, Beijing 100015 Patentee before: BEIJING QIHOO TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230627 Address after: 1765, floor 17, floor 15, building 3, No. 10 Jiuxianqiao Road, Chaoyang District, Beijing 100015 Patentee after: Beijing Hongxiang Technical Service Co.,Ltd. Address before: 300450 No. 9-3-401, No. 39, Gaoxin 6th Road, Binhai Science Park, Binhai New Area, Tianjin Patentee before: 3600 Technology Group Co.,Ltd. |