CN109857673A - Control recognition methods and device - Google Patents
Control recognition methods and device Download PDFInfo
- Publication number
- CN109857673A CN109857673A CN201910138832.2A CN201910138832A CN109857673A CN 109857673 A CN109857673 A CN 109857673A CN 201910138832 A CN201910138832 A CN 201910138832A CN 109857673 A CN109857673 A CN 109857673A
- Authority
- CN
- China
- Prior art keywords
- control
- identified
- information
- icon
- tree
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention discloses a kind of control recognition methods, method comprises determining that target interface locating for control to be identified；Based on the control information of the target interface, the control to be identified is identified；Wherein, the control information includes at least two in control tree, text information and icon information.Using the embodiment of the present invention, the accuracy of control identification can be improved.
Description
Technical field
The present invention relates to field of terminal more particularly to a kind of control recognition methods and devices.
Background technique
A application program (Application, APP) needs to survey the function and performance of APP before publication
Examination needs to test the operating condition of APP in more terminals, checks whether the function of APP meets expection, and in different terminals
On compatibility.
A series of test script step is needed to be implemented in Application testing, executing test script step is specifically to each
The relevant operations such as the corresponding control of test script step clicked, slided, long-pressing, when due to identifying to control, exist
The not high problem of control identification accuracy, and then influence Application testing.
Therefore, it is necessary to propose a kind of control recognition methods that can improve control identification accuracy.
Summary of the invention
The embodiment of the present invention provides a kind of control recognition methods, to solve the problems, such as that control identification accuracy is not high.
In order to solve the above-mentioned technical problem, the present invention is implemented as follows:
In a first aspect, a kind of control recognition methods is provided, this method comprises:
Determine target interface locating for control to be identified；
Based on the control information of the target interface, the control to be identified is identified；
Wherein, the control information includes at least two in control tree, text information and icon information.
Second aspect, provides a kind of control identification device, which includes:
Determining module, for determining target interface locating for control to be identified；
Identification module identifies the control to be identified for the control information based on the target interface；
Wherein, the control information includes at least two in control tree, text information and icon information.
The third aspect provides a kind of terminal device characterized by comprising memory, processor and is stored in described
It is real when the computer program is executed by the processor on memory and the computer program that can run on the processor
Now the step of method as described in relation to the first aspect.
Fourth aspect provides a kind of computer readable storage medium, which is characterized in that the computer-readable storage medium
Computer program is stored in matter, and the step of method as described in relation to the first aspect is realized when the computer program is executed by processor
Suddenly.
In embodiments of the present invention, by the control information based on target interface locating for control to be identified, described in identification
Control to be identified can be improved the accuracy of control identification.
Detailed description of the invention
The drawings described herein are used to provide a further understanding of the present invention, constitutes a part of the invention, this hair
Bright illustrative embodiments and their description are used to explain the present invention, and are not constituted improper limitations of the present invention.In the accompanying drawings:
Fig. 1 is a kind of schematic diagram for application scenarios that one embodiment of the present of invention provides；
Fig. 2 is a kind of flow diagram for control recognition methods that one embodiment of the present of invention provides；
Fig. 3 is a kind of method flow schematic diagram based on control tree identification control that one embodiment of the present of invention provides；
Fig. 4 is a kind of method flow signal based on text information identification control that one embodiment of the present of invention provides
Figure；
Fig. 5 is a kind of method flow signal based on icon information identification control that one embodiment of the present of invention provides
Figure；
Fig. 6 be another embodiment of the present invention provides a kind of control recognition methods flow diagram；
Fig. 7 is a kind of structural schematic diagram for control identification device that one embodiment of the present of invention provides；
Fig. 8 is a kind of structural schematic diagram for terminal device that one embodiment of the present of invention provides.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall within the protection scope of the present invention.
Application scenarios of the invention are illustrated referring to Fig. 1.
It include: the terminal device 102 of test platform 100 and bearing test object in this application scene, on terminal device 102
A user interface on show the first control 104 and the second control 106 to be identified to be identified, in which:
In automatic test course, test platform 100 parses test script, is determined as the target widget of test object
Mark；Based on control tree, text information and icon information, the target widget is recognized from user interface；The control tree
In include identifier or textual form control to be identified, include the control (example to be identified of textual form in the text information
Such as: the second control 106 to be identified), in the icon information including form of icons control to be identified (such as: first is to be identified
Control 104).
Target widget can be constantly recognized during automatic test as a result, execute foot corresponding with target widget
This step, to achieve the purpose that execute entire automatic test script.
Wherein, automatic test generally refers to the automation of software test, and software test is exactly to run on a preset condition based
System or application program assess operation result, and preparatory condition should include normal condition and exceptional condition；Automatic test script is
The series of instructions of specified automatic test, each instruction correspond to each test script step；Control refers to data and method
Encapsulation；Control tree can be with the hierarchical relationship between each control of visual representation；Text information is all texts in present user interface
This；Icon information is all icons in present user interface.
Fig. 2 is a kind of flow diagram for icon-based programming method that one embodiment of the present of invention provides, and this method can be by
Terminal device in the corresponding application scenarios of Fig. 1 executes, and referring to fig. 2, this method can specifically include following steps:
Step 202: determining target interface locating for control to be identified.
Wherein, target interface is user interface locating for the control to be identified on terminal device, and test platform can determine
User interface locating for control to be identified.
Step 204: the control information based on the target interface identifies the control to be identified；
Wherein, the control information includes at least two in control tree, text information and icon information.
It should be noted that when the control information includes control tree and text information, a kind of implementation of step 204
It can be with are as follows:
Step S1: being based on the control tree, identifies the control to be identified, includes identifier or text in the control tree
The control to be identified of form；
Wherein, identifier can be default ID (identifier, identifier), and the control to be identified of forms of identifier can be with
It is " 124 " etc. that the control to be identified of textual form can be " closing " or " Close " etc., control tree can also include icon shape
The control to be identified of formula, the control to be identified of form of icons can be " < " orDeng.
Further, referring to Fig. 3, a kind of implementation of step S1 can be with are as follows:
Step S102: target widget is searched according to preset control tree search strategy in the control tree of the target interface.
Wherein, target widget is the corresponding control of certain testing procedure in automatic test course, and preset control tree searches plan
Somewhat body can be exemplified are as follows:
From multiple attributes of target widget, the preferred attribute of target widget is determined；Mesh is searched according to the preferred attribute
Mark control；Priority can be preselected by tester and be specified, and the attribute of target widget can be the identifier of control type, control
Deng.
Step S104: determine whether to find a control to be identified.
If so, thening follow the steps S106；If it is not, thening follow the steps S108；
It should be noted that since the attribute of control may lack or different controls possesses identical attribute, because
This, it is possible that can not find control, finds one when searching target widget according to the preferred attribute of target widget in control tree
A control, or the case where find multiple controls.For ease of description, the one or more controls found can be known as control to be identified
Part.
Step S104 specifically can be exemplified are as follows:
Target widget can be " search button ", and the identifier of the search button can be " 1234 ", and icon can beControl type can be A class control, and text can be " search "；It can be according to target in the control tree of target interface
The identifier of control searches target widget, since the identifier " 1234 " of target widget may lack, then possibly can not pass through mesh
The identifier for marking control searches target widget；It can also be looked into according to the control type of target widget in the control tree of target interface
Target widget is looked for, since A class control there may be one or more in control tree, then may find one or more controls to be identified
Part identifies control to be identified when finding one or more controls to be identified.
Step S106: determine whether this control to be identified is forms of identifier；
If so, thening follow the steps S110；If it is not, thening follow the steps S114；
Wherein, forms of identifier is a kind of display format of the control in control tree, and the display format of control can also be
Textual form or form of icons, such as return push-button can be shown as " 223 " or the textual form of forms of identifier
" " of " Close " or " closing " or form of icons.
Step S110: determine the control to be identified of this forms of identifier and forms of identifier target widget whether phase
Match.
If so, thening follow the steps S112；If it is not, thening follow the steps S114；
Step S110 specifically can be exemplified are as follows:
Determine whether the matching degree of the control to be identified of forms of identifier and the target widget of forms of identifier is not less than
One preset matching threshold value matches if being not less than the first preset matching threshold value；If being lower than the first preset matching threshold value, no
Match.
Step S112: the control to be identified for identifying this forms of identifier is target widget；
Step S114: determine whether this control to be identified is textual form；
If so, thening follow the steps S116；If it is not, thening follow the steps S120；
Step S116: whether the target widget of the control to be identified and textual form that determine this textual form matches；
If so, thening follow the steps S118；If it is not, thening follow the steps S120；
Step S116 specifically can be exemplified are as follows:
Determine whether the matching degree of the control to be identified of textual form and the target widget of textual form is not less than second in advance
If matching threshold, if being not less than the second preset matching threshold value, match；If being lower than the second preset matching threshold value, not phase
Match.
Step S118: the control to be identified for identifying this textual form is target widget；
Step S120: determine whether the target widget of the control and form of icons to be identified of this form of icons matches；
If so, thening follow the steps S122；
Step S120 specifically can be exemplified are as follows:
It is pre- to determine whether the matching degree of the target widget of the control and form of icons to be identified of form of icons is not less than third
If matching threshold, if being not less than third preset matching threshold value, match；If being lower than third preset matching threshold value, not phase
Match.
Step S122: the control to be identified for identifying this form of icons is target widget；
Step S108: it determines in the control to be identified of multiple form of icons with the presence or absence of the target widget phase with form of icons
Matched control to be identified；
If so, thening follow the steps S124；
Step S108 specifically can be exemplified are as follows:
When finding multiple controls to be identified, determine each control to be identified in the control to be identified of multiple form of icons with
The matching degree of the target widget of form of icons, matching degree is not less than third preset matching threshold value if it exists, then exists and target control
The control to be identified that part matches, otherwise there is no the controls to be identified to match with target widget.
Step S124: identifying in the control to be identified to match, the control to be identified of the highest form of icons of matching degree
For target widget.
Step S124 specifically can be exemplified are as follows:
If third preset matching threshold value is 90%, the matching degree of the control to be identified to match is respectively 92%, 95%,
98%, then the control to be identified that matching degree is 98% is target widget.
Step S2: if the control to be identified can not be identified, it is based on the text information, identifies the control to be identified
Part includes the control to be identified of textual form in the text information.
Further, referring to Fig. 4, a kind of implementation of step S2 can be with are as follows:
Step S202: in the text information, using OCR (Optical Character Recognition, optics
Character recognition) mode identifies control to be identified, it include the control to be identified of textual form in the text information；
Step S204: whether the file destination of the control to be identified and textual form that determine textual form matches；
If so, thening follow the steps S206；
Step S204 specifically can be exemplified are as follows:
Determine whether the matching degree of the control to be identified of textual form and the target widget of textual form is not less than second in advance
If matching threshold, if being not less than the second preset matching threshold value, match；If being lower than the second preset matching threshold value, not phase
Match.
Step S206: the control to be identified for identifying this textual form is target widget.
In one embodiment, before step S202, further includes:
Step S201: whether the screenshot of the screenshot and reference interface that determine target interface matches；
If so, thening follow the steps S202；
Wherein, test script records separately system and executes two states, to be identified when target interface is that current script step executes
Interface locating for control；When reference interface is that current script is recorded, interface locating for target widget；
Step S201 specifically can be exemplified are as follows:
Determine whether the matching degree of the screenshot of target interface and the screenshot of reference interface is lower than the 4th preset matching threshold value；If
Not less than the 4th preset matching threshold value, then match；If being lower than the 4th default preset matching threshold value, do not match that.
This implementation based on step 204, can be in such a way that the control tree of target interface and text information combine
It identifies control to be identified, improves accuracy, diversity and the reliability of control identification.
It should be noted that when the control information includes control tree and icon information, another realization side of step 204
Formula can be with are as follows:
Step S1 ': being based on the control tree, identifies the control to be identified, includes identifier or text in the control tree
The control to be identified of this form；
Step S2 ': if the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified
Part includes the control to be identified of form of icons in the icon information.
This implementation based on step 204, can be in such a way that the control tree of target interface and icon information combine
It identifies control to be identified, improves accuracy, diversity and the reliability of control identification.
Further, referring to Fig. 5, a kind of implementation of step S2 ' can be with are as follows:
Step S202 ': in the icon information, control to be identified, the icon letter are identified using image recognition mode
It include the control to be identified of form of icons in breath；
Step S204 ': determine whether the target widget of the control and form of icons to be identified of form of icons matches；
If so, thening follow the steps S206 '；
Step S204 ' specifically can be exemplified are as follows:
It is pre- to determine whether the matching degree of the target widget of the control and form of icons to be identified of form of icons is not less than third
If matching threshold, if being not less than third preset matching threshold value, match；If being lower than third preset matching threshold value, not phase
Match.
Step S206 ': identify that this waits for that the identification control of form of icons is target widget.
In one embodiment, before step S202 ', further includes:
Step S201 ': whether the screenshot of the screenshot and reference interface that determine target interface matches；
If so, thening follow the steps S202 '；
Step S201 specifically can be exemplified are as follows:
Determine whether the matching degree of the screenshot of target interface and the screenshot of reference interface is lower than the 4th preset matching threshold value；If
Not less than the 4th preset matching threshold value, then match；If being lower than the 4th default preset matching threshold value, do not match that.
It should be noted that another of step 204 is realized when the control information includes text information and icon information
Mode can be with are as follows:
Based on the text information, the control to be identified is identified, include the described of textual form in the text information
Control to be identified；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the figure
Mark the control to be identified in information including form of icons.
This implementation based on step 204, the side that can be combined by the text information and icon information of target interface
Formula identifies control to be identified, improves accuracy, diversity and the reliability of control identification.
Further, a kind of implementation of step S2 or step S2 ' can be with are as follows:
Determine whether the complexity of the control to be identified of form of icons is lower than complexity threshold；
If so, being based on the icon information, father's control of the control to be identified is identified, to identify the control to be identified
Part includes father's control of form of icons in the icon information；
If it is not, being then based on the icon information, the control to be identified is identified, include form of icons in the icon information
The control to be identified.
This implementation specifically can be exemplified are as follows:
If the complexity of the control to be identified of form of icons is lower than complexity threshold, i.e., too simple, it is determined that figure
Whether the complexity of father's control of the control to be identified of mark form is lower than complexity threshold；
If the complexity of father's control of the control to be identified of form of icons is not less than complexity threshold, image is used
Identification method identifies father's control of control to be identified, determine father's control of control to be identified and target widget father's control whether phase
Matching；If matching, identify that father's control of control to be identified is father's control of target widget, so that it is determined that control to be identified
For target widget；
If the complexity of the control to be identified of form of icons is not less than complexity threshold, image recognition mode is used
It identifies control to be identified, determines whether control to be identified matches with target widget；If matching, control to be identified is identified
For target widget.
Based on this, by identifying father's control of control to be identified, to identify control to be identified, control knowledge is further improved
Other reliability and flexibility.
It should be noted that another implementation of step S2 or step S2 ' can be with are as follows:
If the complexity of father's control of the control to be identified of form of icons be lower than complexity threshold, it is determined that it is described to
Identify whether the complexity of father's control (father father's control of control i.e. to be identified) of father's control of control is lower than complexity threshold；
If it is not, then identifying father father's control of the control to be identified using image recognition mode, the control to be identified is determined
Whether father father's control of part and father father's control of target widget match；If matching, the father father of control to be identified is identified
Control is father father's control of target widget, so that it is determined that control to be identified is target widget.
Based on this, by identifying that father father's control of control to be identified further improves control to identify control to be identified
The reliability and flexibility of identification.
It should be noted that step 204 is again when the control information includes control tree, text information and icon information
A kind of implementation can be with are as follows:
Step S1 ": being based on the control tree, identifies the control to be identified, includes identifier or text in the control tree
The control to be identified of this form；
Step S2 ": if the control to be identified can not be identified, it is based on the text information, identifies the control to be identified
Part includes the control to be identified of textual form in the text information；
Step S3 ": if the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified
Part includes the control to be identified of form of icons in the icon information.
This implementation based on step 204 can pass through the control tree of target interface, text information and icon information knot
The mode of conjunction identifies control to be identified, improves accuracy, diversity and the reliability of control identification.
After step S3 ", step 204 further include:
If the control to be identified can not be identified, it is determined whether exceed the first default recognition cycle；
If it is not, being then based on the control tree, the control to be identified is identified, include identifier or text in the control tree
The control to be identified of form；
If the control to be identified can not be identified, it is based on the text information, identifies the control to be identified, the text
It include the control to be identified of textual form in this information；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the figure
Mark the control to be identified in information including form of icons.
The number of control identification is increased, the reliable of control identification is improved by the way that default recognition cycle is arranged based on this
Property and efficiency.
Further, after step S3 ", step 204 further include:
If exceeding the described first default recognition cycle, it is determined whether exceed the second default recognition cycle；
If it is not, being then based on the text information, the control to be identified is identified, include textual form in the text information
The control to be identified；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the figure
Mark the control to be identified in information including form of icons.
Based on this, recognition cycle is preset by being arranged, increases the number of control identification, and when the time is not abundant, it is excellent
Control first is identified based on one or more of control informations, improves flexibility, reliability and the efficiency of control identification.
Embodiment corresponding for Fig. 2 identifies institute by the control information based on target interface locating for control to be identified
Control to be identified is stated, can be improved the accuracy of control identification.
Fig. 6 be another embodiment of the present invention provides a kind of icon-based programming method flow diagram, this method can
It is executed by the terminal device in the corresponding application scenarios of Fig. 1, referring to Fig. 6, this method can specifically include following steps:
Step 602: when the control information includes control tree, text information and icon information, being based on the control tree, know
The not described control to be identified includes the control to be identified of identifier or textual form in the control tree；
Step 604: if the control to be identified can not be identified, being based on the text information, identify the control to be identified
Part includes the control to be identified of textual form in the text information；
Step 606: if the control to be identified can not be identified, being based on the icon information, identify the control to be identified
Part includes the control to be identified of form of icons in the icon information；
Step 608: if the control to be identified can not be identified, it is determined whether exceed the first default recognition cycle；
If it is not, thening follow the steps 602；If so, thening follow the steps 610；
Step 610: determining whether beyond the second default recognition cycle；
If it is not, thening follow the steps 604；If so, terminating.
Embodiment corresponding for Fig. 6, by control tree, text information and target information based on target interface extremely
Lack 2 kinds to identify control, improves accuracy, the reliability, flexibility of control identification；By the way that default recognition cycle is arranged, mention
The high efficiency of control identification.
In addition, for simple description, therefore, it is stated as a series of action groups for above method embodiment
It closes, but those skilled in the art should understand that, embodiment of the present invention is not limited by the described action sequence, because
Embodiment according to the present invention, some steps may be performed in other sequences or simultaneously.Secondly, those skilled in the art
It should be aware of, embodiment described in this description belongs to preferred embodiment, and related movement is not necessarily originally
Necessary to invention embodiment.
Fig. 7 is a kind of structural schematic diagram for control identification device that one embodiment of the present of invention provides, should referring to Fig. 7
Device can specifically include: determining module 702 and identification module 704, in which:
Determining module 702, for determining target interface locating for control to be identified；
Identification module 704 identifies the control to be identified for the control information based on the target interface；
Wherein, the control information includes at least two in control tree, text information and icon information.
Embodiment corresponding for Fig. 7 identifies institute by the control information based on target interface locating for control to be identified
Control to be identified is stated, can be improved the accuracy of control identification.
Optionally, when the control information includes control tree and text information, identification module includes:
First identification submodule identifies the control to be identified, includes in the control tree for being based on the control tree
The control to be identified of identifier or textual form；
If the control to be identified can not be identified, it is based on the text information, identifies the control to be identified, the text
It include the control to be identified of textual form in this information.
Optionally, when the control information includes control tree and icon information, identification module further include:
Second identification submodule identifies the control to be identified, includes in the control tree for being based on the control tree
The control to be identified of identifier or textual form；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the figure
Mark the control to be identified in information including form of icons.
Optionally, when the control information includes text information and icon information, identification module further include:
Third identifies submodule, for being based on the text information, identifies the control to be identified, in the text information
The control to be identified including textual form；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the figure
Mark the control to be identified in information including form of icons.
Optionally, the second identification submodule or third identify submodule further include:
Complexity determination unit, for determining whether the complexity of the control to be identified of form of icons is lower than complexity
Threshold value；
If so, being based on the icon information, father's control of the control to be identified is identified, to identify the control to be identified
Part includes father's control of form of icons in the icon information；
If it is not, being then based on the icon information, the control to be identified is identified, include form of icons in the icon information
The control to be identified.
Optionally, when the control information includes control tree, text information and icon information, identification module further include:
4th identification submodule identifies the control to be identified, includes in the control tree for being based on the control tree
The control to be identified of identifier or textual form；
If the control to be identified can not be identified, it is based on the text information, identifies the control to be identified, the text
It include the control to be identified of textual form in this information；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the figure
Mark the control to be identified in information including form of icons.
Optionally, the 4th identification submodule further include:
Period 1 determination unit, if for that can not identify the control to be identified, it is determined whether default beyond first
Recognition cycle；
If it is not, being then based on the control tree, the control to be identified is identified, include identifier or text in the control tree
The control to be identified of form；
If the control to be identified can not be identified, it is based on the text information, identifies the control to be identified, the text
It include the control to be identified of textual form in this information；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the figure
Mark the control to be identified in information including form of icons.
Optionally, period 1 determination unit further include:
Second round determines subelement, if for exceeding the described first default recognition cycle, it is determined whether exceed second
Default recognition cycle；
If it is not, being then based on the text information, the control to be identified is identified, include textual form in the text information
The control to be identified；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the figure
Mark the control to be identified in information including form of icons.
Device provided in an embodiment of the present invention can be realized each mistake that device is realized in the embodiment of the method for Fig. 2 to Fig. 6
Journey, to avoid repeating, which is not described herein again.Moreover, it should be noted that in all parts of the device of the invention, according to
Its function to be realized and logical partitioning has been carried out to component therein, still, the present invention is not only restricted to this, can according to need
All parts are repartitioned or are combined.
A kind of hardware structural diagram of Fig. 8 terminal device of each embodiment to realize the present invention,
The terminal device 800 includes but is not limited to: radio frequency unit 801, network module 802, audio output unit 803, defeated
Enter unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, processor
The components such as 810 and power supply 811.It will be understood by those skilled in the art that terminal device structure shown in Fig. 8 is not constituted
Restriction to terminal device, terminal device may include than illustrating more or fewer components, perhaps combine certain components or
Different component layouts.In embodiments of the present invention, terminal device include but is not limited to mobile phone, tablet computer, laptop,
Palm PC, car-mounted terminal, wearable device and pedometer etc..
Wherein, processor 810, for determining target interface locating for control to be identified；
Based on the control information of the target interface, the control to be identified is identified；
Wherein, the control information includes at least two in control tree, text information and icon information.
By the control information based on target interface locating for control to be identified, the control to be identified, Neng Gouti are identified
The accuracy of high control identification.
It should be understood that the embodiment of the present invention in, radio frequency unit 801 can be used for receiving and sending messages or communication process in, signal
Send and receive, specifically, by from base station downlink data receive after, to processor 810 handle；In addition, by uplink
Data are sent to base station.In general, radio frequency unit 801 includes but is not limited to antenna, at least one amplifier, transceiver, coupling
Device, low-noise amplifier, duplexer etc..In addition, radio frequency unit 801 can also by wireless communication system and network and other set
Standby communication.
Terminal device provides wireless broadband internet by network module 802 for user and accesses, and such as user is helped to receive
It sends e-mails, browse webpage and access streaming video etc..
Audio output unit 803 can be received by radio frequency unit 801 or network module 802 or in memory 809
The audio data of storage is converted into audio signal and exports to be sound.Moreover, audio output unit 803 can also provide and end
The relevant audio output of specific function that end equipment 800 executes is (for example, call signal receives sound, message sink sound etc.
Deng).Audio output unit 803 includes loudspeaker, buzzer and receiver etc..
Input unit 804 is for receiving audio or video signal.Input unit 804 may include graphics processor
(Graphics Processing Unit, GPU) 8041 and microphone 8042, graphics processor 8041 is in video acquisition mode
Or the image data of the static images or video obtained in image capture mode by image capture apparatus (such as camera) carries out
Reason.Treated, and picture frame may be displayed on display unit 806.Through graphics processor 8041, treated that picture frame can be deposited
Storage is sent in memory 809 (or other storage mediums) or via radio frequency unit 801 or network module 802.Mike
Wind 8042 can receive sound, and can be audio data by such acoustic processing.Treated audio data can be
The format output that mobile communication base station can be sent to via radio frequency unit 801 is converted in the case where telephone calling model.
Terminal device 800 further includes at least one sensor 805, such as optical sensor, motion sensor and other biographies
Sensor.Specifically, optical sensor includes ambient light sensor and proximity sensor, wherein ambient light sensor can be according to environment
The light and shade of light adjusts the brightness of display panel 8061, and proximity sensor can close when terminal device 800 is moved in one's ear
Display panel 8061 and/or backlight.As a kind of motion sensor, accelerometer sensor can detect in all directions (general
For three axis) size of acceleration, it can detect that size and the direction of gravity when static, can be used to identify terminal device posture (ratio
Such as horizontal/vertical screen switching, dependent game, magnetometer pose calibrating), Vibration identification correlation function (such as pedometer, tap)；It passes
Sensor 805 can also include fingerprint sensor, pressure sensor, iris sensor, molecule sensor, gyroscope, barometer, wet
Meter, thermometer, infrared sensor etc. are spent, details are not described herein.
Display unit 806 is for showing information input by user or being supplied to the information of user.Display unit 806 can wrap
Display panel 8061 is included, liquid crystal display (Liquid Crystal Display, LCD), Organic Light Emitting Diode can be used
Forms such as (Organic Light-Emitting Diode, OLED) configure display panel 8061.
User input unit 807 can be used for receiving the number or character information of input, and generate the use with terminal device
Family setting and the related key signals input of function control.Specifically, user input unit 807 include touch panel 8071 and
Other input equipments 8072.Touch panel 8071, also referred to as touch screen collect the touch operation of user on it or nearby
(for example user uses any suitable objects or attachment such as finger, stylus on touch panel 8071 or in touch panel 8071
Neighbouring operation).Touch panel 8071 may include both touch detecting apparatus and touch controller.Wherein, touch detection
Device detects the touch orientation of user, and detects touch operation bring signal, transmits a signal to touch controller；Touch control
Device processed receives touch information from touch detecting apparatus, and is converted into contact coordinate, then gives processor 810, receiving area
It manages the order that device 810 is sent and is executed.Furthermore, it is possible to more using resistance-type, condenser type, infrared ray and surface acoustic wave etc.
Seed type realizes touch panel 8071.In addition to touch panel 8071, user input unit 807 can also include other input equipments
8072.Specifically, other input equipments 8072 can include but is not limited to physical keyboard, function key (such as volume control button,
Switch key etc.), trace ball, mouse, operating stick, details are not described herein.
Further, touch panel 8071 can be covered on display panel 8061, when touch panel 8071 is detected at it
On or near touch operation after, send processor 810 to determine the type of touch event, be followed by subsequent processing device 810 according to touching
The type for touching event provides corresponding visual output on display panel 8061.Although in fig. 8, touch panel 8071 and display
Panel 8061 is the function that outputs and inputs of realizing terminal device as two independent components, but in some embodiments
In, can be integrated by touch panel 8071 and display panel 8061 and realize the function that outputs and inputs of terminal device, it is specific this
Place is without limitation.
Interface unit 808 is the interface that external device (ED) is connect with terminal device 800.For example, external device (ED) may include having
Line or wireless head-band earphone port, external power supply (or battery charger) port, wired or wireless data port, storage card end
Mouth, port, the port audio input/output (I/O), video i/o port, earphone end for connecting the device with identification module
Mouthful etc..Interface unit 808 can be used for receiving the input (for example, data information, electric power etc.) from external device (ED) and
By one or more elements that the input received is transferred in terminal device 800 or can be used in 800 He of terminal device
Data are transmitted between external device (ED).
Memory 809 can be used for storing software program and various data.Memory 809 can mainly include storing program area
The storage data area and, wherein storing program area can (such as the sound of application program needed for storage program area, at least one function
Sound playing function, image player function etc.) etc.；Storage data area can store according to mobile phone use created data (such as
Audio data, phone directory etc.) etc..In addition, memory 809 may include high-speed random access memory, it can also include non-easy
The property lost memory, a for example, at least disk memory, flush memory device or other volatile solid-state parts.
Processor 810 is the control centre of terminal device, utilizes each of various interfaces and the entire terminal device of connection
A part by running or execute the software program and/or module that are stored in memory 809, and calls and is stored in storage
Data in device 809 execute the various functions and processing data of terminal device, to carry out integral monitoring to terminal device.Place
Managing device 810 may include one or more processing units；Preferably, processor 810 can integrate application processor and modulatedemodulate is mediated
Manage device, wherein the main processing operation system of application processor, user interface and application program etc., modem processor is main
Processing wireless communication.It is understood that above-mentioned modem processor can not also be integrated into processor 810.
Terminal device 800 can also include the power supply 811 (such as battery) powered to all parts, it is preferred that power supply 811
Can be logically contiguous by power-supply management system and processor 810, to realize management charging by power-supply management system, put
The functions such as electricity and power managed.
In addition, terminal device 800 includes some unshowned functional modules, details are not described herein.
Preferably, the embodiment of the present invention also provides a kind of terminal device, including processor 810, and memory 809 is stored in
On memory 809 and the computer program that can run on the processor 810, the computer program are executed by processor 810
Each process of the above-mentioned control recognition methods embodiment of Shi Shixian, and identical technical effect can be reached, to avoid repeating, here
It repeats no more.
The embodiment of the present invention also provides a kind of computer readable storage medium, and meter is stored on computer readable storage medium
Calculation machine program, the computer program realize each process of above-mentioned control recognition methods embodiment, and energy when being executed by processor
Reach identical technical effect, to avoid repeating, which is not described herein again.Wherein, the computer readable storage medium, such as only
Read memory (Read-Only Memory, abbreviation ROM), random access memory (Random Access Memory, abbreviation
RAM), magnetic or disk etc..
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the device that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or device institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or device.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in a storage medium
In (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that a terminal (can be mobile phone, computer, service
Device, air conditioner or network equipment etc.) execute method described in each embodiment of the present invention.
The embodiment of the present invention is described with above attached drawing, but the invention is not limited to above-mentioned specific
Embodiment, the above mentioned embodiment is only schematical, rather than restrictive, those skilled in the art
Under the inspiration of the present invention, without breaking away from the scope protected by the purposes and claims of the present invention, it can also make very much
Form belongs within protection of the invention.
Claims (10)
1. a kind of control recognition methods characterized by comprising
Determine target interface locating for control to be identified；
Based on the control information of the target interface, the control to be identified is identified；
Wherein, the control information includes at least two in control tree, text information and icon information.
2. the method according to claim 1, wherein when the control information includes control tree and text information,
Based on the control information of the target interface, the control to be identified is identified, comprising:
Based on the control tree, the control to be identified is identified, include the described of identifier or textual form in the control tree
Control to be identified；
If the control to be identified can not be identified, it is based on the text information, identifies the control to be identified, the text envelope
It include the control to be identified of textual form in breath.
3. the method according to claim 1, wherein when the control information includes control tree and icon information,
Based on the control information of the target interface, the control to be identified is identified, comprising:
Based on the control tree, the control to be identified is identified, include the described of identifier or textual form in the control tree
Control to be identified；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the icon letter
It include the control to be identified of form of icons in breath.
4. the method according to claim 1, wherein the control information includes text information and icon information
When, based on the control information of the target interface, identify the control to be identified, comprising:
Based on the text information, the control to be identified is identified, include the described wait know of textual form in the text information
Other control；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the icon letter
It include the control to be identified of form of icons in breath.
5. the method according to claim 3 or 4, which is characterized in that be based on the icon information, identify the control to be identified
Part, comprising:
Determine whether the complexity of the control to be identified of form of icons is lower than complexity threshold；
If so, being based on the icon information, father's control of the control to be identified is identified, to identify the control to be identified,
It include father's control of form of icons in the icon information；
If it is not, being then based on the icon information, the control to be identified is identified, include the institute of form of icons in the icon information
State control to be identified.
6. the method according to claim 1, wherein the control information includes control tree, text information and figure
When marking information, based on the control information of the target interface, the control to be identified is identified, comprising:
Based on the control tree, the control to be identified is identified, include the described of identifier or textual form in the control tree
Control to be identified；
If the control to be identified can not be identified, it is based on the text information, identifies the control to be identified, the text envelope
It include the control to be identified of textual form in breath；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the icon letter
It include the control to be identified of form of icons in breath.
7. according to the method described in claim 6, it is characterized in that, identifying the control to be identified being based on the icon information
After part, the method also includes:
If the control to be identified can not be identified, it is determined whether exceed the first default recognition cycle；
If it is not, being then based on the control tree, the control to be identified is identified, include identifier or textual form in the control tree
The control to be identified；
If the control to be identified can not be identified, it is based on the text information, identifies the control to be identified, the text envelope
It include the control to be identified of textual form in breath；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the icon letter
It include the control to be identified of form of icons in breath.
8. the method according to the description of claim 7 is characterized in that if exceeding the described first default recognition cycle, it is determined that be
It is no to exceed the second default recognition cycle；
If it is not, being then based on the text information, the control to be identified is identified, include the institute of textual form in the text information
State control to be identified；
If the control to be identified can not be identified, it is based on the icon information, identifies the control to be identified, the icon letter
It include the control to be identified of form of icons in breath.
9. a kind of control identification device characterized by comprising
Determining module, for determining target interface locating for control to be identified；
Identification module identifies the control to be identified for the control information based on the target interface；
Wherein, the control information includes at least two in control tree, text information and icon information.
10. a kind of terminal device characterized by comprising memory, processor and be stored on the memory and can be in institute
The computer program run on processor is stated, such as claim 1 to 8 is realized when the computer program is executed by the processor
Any one of described in method the step of.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910138832.2A CN109857673B (en) | 2019-02-25 | 2019-02-25 | Control identification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910138832.2A CN109857673B (en) | 2019-02-25 | 2019-02-25 | Control identification method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109857673A true CN109857673A (en) | 2019-06-07 |
| CN109857673B CN109857673B (en) | 2022-02-15 |
Family
ID=66898921
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910138832.2A Active CN109857673B (en) | 2019-02-25 | 2019-02-25 | Control identification method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109857673B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113076257A (en) * | 2021-04-19 | 2021-07-06 | 西安闻泰信息技术有限公司 | Test script recording method, device, medium and computer equipment |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130031492A1 (en) * | 2011-07-29 | 2013-01-31 | Curtis Matthew C | Interface Wires for a Measurement System Diagram |
| CN103309797A (en) * | 2012-03-09 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Automatic testing method and device of user interface |
| US20140118239A1 (en) * | 2012-10-25 | 2014-05-01 | PixiOnCloud, Inc. | Visual-symbolic control of remote devices having display-based user interfaces |
| CN104391797A (en) * | 2014-12-09 | 2015-03-04 | 北京奇虎科技有限公司 | GUI (graphical user interface) widget identification method and device |
| CN105843494A (en) * | 2015-01-15 | 2016-08-10 | 中兴通讯股份有限公司 | Method and device for realizing region screen capture, and terminal |
| CN108363599A (en) * | 2018-01-12 | 2018-08-03 | 深圳壹账通智能科技有限公司 | User interface shows recognition methods and terminal device |
| CN108399124A (en) * | 2018-02-07 | 2018-08-14 | 深圳壹账通智能科技有限公司 | Application testing method, device, computer equipment and storage medium |
| CN108595965A (en) * | 2018-04-28 | 2018-09-28 | 北京字节跳动网络技术有限公司 | Prevent the method and its device that the attribute information of interface control is stolen |
| CN109002759A (en) * | 2018-06-07 | 2018-12-14 | Oppo广东移动通信有限公司 | text recognition method, device, mobile terminal and storage medium |
-
2019
- 2019-02-25 CN CN201910138832.2A patent/CN109857673B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130031492A1 (en) * | 2011-07-29 | 2013-01-31 | Curtis Matthew C | Interface Wires for a Measurement System Diagram |
| CN103309797A (en) * | 2012-03-09 | 2013-09-18 | 阿里巴巴集团控股有限公司 | Automatic testing method and device of user interface |
| US20140118239A1 (en) * | 2012-10-25 | 2014-05-01 | PixiOnCloud, Inc. | Visual-symbolic control of remote devices having display-based user interfaces |
| CN104391797A (en) * | 2014-12-09 | 2015-03-04 | 北京奇虎科技有限公司 | GUI (graphical user interface) widget identification method and device |
| CN105843494A (en) * | 2015-01-15 | 2016-08-10 | 中兴通讯股份有限公司 | Method and device for realizing region screen capture, and terminal |
| CN108363599A (en) * | 2018-01-12 | 2018-08-03 | 深圳壹账通智能科技有限公司 | User interface shows recognition methods and terminal device |
| CN108399124A (en) * | 2018-02-07 | 2018-08-14 | 深圳壹账通智能科技有限公司 | Application testing method, device, computer equipment and storage medium |
| CN108595965A (en) * | 2018-04-28 | 2018-09-28 | 北京字节跳动网络技术有限公司 | Prevent the method and its device that the attribute information of interface control is stolen |
| CN109002759A (en) * | 2018-06-07 | 2018-12-14 | Oppo广东移动通信有限公司 | text recognition method, device, mobile terminal and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 李爱国 等: "基于ActiveX控件与Office对象模型的技术文档管理系统", 《计算机应用与软件》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113076257A (en) * | 2021-04-19 | 2021-07-06 | 西安闻泰信息技术有限公司 | Test script recording method, device, medium and computer equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109857673B (en) | 2022-02-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108255378A (en) | A kind of display control method and mobile terminal | |
| CN108363524B (en) | A kind of application icon display methods and mobile terminal | |
| CN109871246A (en) | A kind of message display method and terminal device | |
| CN109947650A (en) | Script step process methods, devices and systems | |
| CN108271143A (en) | A kind of connection method of Bluetooth device and mobile terminal | |
| CN110049185A (en) | Image processing method and terminal device | |
| CN110109593A (en) | A kind of screenshotss method and terminal device | |
| CN108345474A (en) | Startup method, starter and the mobile terminal of application program | |
| CN110231900A (en) | A kind of application icon display methods and terminal | |
| CN109600297A (en) | A kind of mark sweep-out method and terminal device | |
| CN107908770A (en) | A kind of photo searching method and mobile terminal | |
| CN108469968A (en) | A kind of applied program processing method and mobile terminal | |
| CN106200897B (en) | A kind of method and apparatus of display control menu | |
| CN110366027A (en) | A kind of video management method and terminal device | |
| CN109857297A (en) | Information processing method and terminal device | |
| CN108897482A (en) | A kind of information display method and terminal device | |
| CN108712321A (en) | A kind of reminding method and mobile terminal of message | |
| CN108510267A (en) | A kind of account information acquisition methods, mobile terminal | |
| CN109885490A (en) | A kind of picture control methods and device | |
| CN110069675A (en) | A kind of search method and mobile terminal | |
| CN110096203A (en) | A kind of screenshot method and mobile terminal | |
| CN110069407A (en) | A kind of function test method and device of application program | |
| CN109726303A (en) | A kind of image recommendation method and terminal | |
| CN109284146A (en) | A kind of light application open method and mobile terminal | |
| CN109088811A (en) | A kind of method for sending information and mobile terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210721 Address after: 100016 no.4301, No.52 Jiuxianqiao hospital, Chaoyang District, Beijing Applicant after: BEIJING TESTIN INFORMATION TECHNOLOGY Co.,Ltd. Address before: Room 2016, building 2, No.8, Fenghuang Third Road, Zhongxin Guangzhou Knowledge City, Guangzhou 510260, Guangdong Province Applicant before: GUANGZHOU TESTIN INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |