CN110457105A - Interface operation method, device, equipment and storage medium - Google Patents

Abstract

The embodiment of the present application provides a kind of interface operation method, device, equipment and storage medium；Method includes: to encode according to pre-arranged code rule to the element that operates of current interface on electronic equipment, obtains the mark that can operate element；The mark for operating element is shown and is operating the corresponding position of element with described；Identification includes the phonetic order of the mark that can operate element；The phonetic order is responded, respective operations are executed to the current interface.By the application, the software compatibility can be improved, and can reduce user's operation difficulty and user's learning cost, so that speech control process is easy to learn, improve user experience.

Description

Interface operation method, device, equipment and storage medium

Technical field

The invention relates to field of electronic device, relate to, but are not limited to a kind of interface operation method, device, equipment and Storage medium.

Background technique

For the electronic equipment with display unit, when operating to electronic equipment, usually pass through finger or touching The operating bodies such as control pen operate electronic equipment, alternatively, the key by electronic equipment is operated, these modes of operation are all It is not available family release both hands.But many electronic equipments are that inconvenience is operated with hand or user is currently inconvenient It is operated with hand, therefore the interface operation of voice control electronic equipment is a kind of good alternative solution.

Each of good interface of current voice control technology, usually predefined can operate the corresponding voice of element and refer to It enables, then user is by input semantic instructions, and then realizes to the voice-controlled operations that can operate element.

But the current this voice control technology that the corresponding phonetic order of element can be operated by predefined, it can not The mark that can operate element is intuitively shown to user, carrys out extra cost to user speech operation learning tape.

Summary of the invention

The embodiment of the present application provides a kind of interface operation method, device, equipment and storage medium, can be compatible with any tradition Software and system, and user operation process is easy to learn.

The technical solution of the embodiment of the present application is achieved in that

The embodiment of the present application provides a kind of interface operation method, comprising:

According to pre-arranged code rule, the element that operates of current interface on electronic equipment is encoded, obtain it is described can Operate the mark of element；

The mark for operating element is shown and is operating the corresponding position of element with described；

Identification includes the phonetic order of the mark that can operate element；

The phonetic order is responded, respective operations are executed to the current interface.

The embodiment of the present application provides a kind of interface operating device, comprising:

Coding module, for being compiled to the element that operates of current interface on electronic equipment according to pre-arranged code rule Code obtains the mark that can operate element；

Display module is operating the corresponding position of element with described for showing the mark for operating element；

Identification module includes the phonetic order of the mark that can operate element for identification；

Respond module executes respective operations to the current interface for responding the phonetic order.

The embodiment of the present application provides a kind of interface operation equipment, comprising:

Memory, for storing executable instruction；

Processor when for executing the executable instruction stored in the memory, is realized provided by the embodiments of the present application Interface operation method.

The embodiment of the present application provides a kind of storage medium, is stored with executable instruction, real when for causing processor to execute Existing interface operation method provided by the embodiments of the present application.

The embodiment of the present application has the advantages that

According to pre-arranged code rule, the element that operates of current interface on electronic equipment is encoded, obtain it is described can The mark for operating element is shown and is operating the corresponding position of element with described by the mark for operating element.In this way, can So that the method for the embodiment of the present application suitable for any software and system, improves the software compatibility；And user can be reduced Operation difficulty and user's learning cost improve user experience so that speech control process is easy to learn.

Detailed description of the invention

Fig. 1 is the interface schematic diagram of voice control in the related technology；

Fig. 2 is an optional configuration diagram of interface operation system provided by the embodiments of the present application；

Fig. 3 is the structural schematic diagram of terminal provided by the embodiments of the present application；

Fig. 4 is an optional flow diagram of interface operation method provided by the embodiments of the present application；

Fig. 5 A is that the embodiment of the present application shows the optional interface schematic diagram that can operate component identification；

Fig. 5 B is that the embodiment of the present application shows the optional interface schematic diagram that can operate component identification；

Fig. 5 C is that the embodiment of the present application shows the optional interface schematic diagram that can operate component identification；

Fig. 5 D is that the embodiment of the present application shows the optional interface schematic diagram that can operate component identification；

Fig. 5 E is that the embodiment of the present application shows the optional interface schematic diagram that can operate component identification；

Fig. 5 F is that the embodiment of the present application electronic equipment responds the interface schematic diagram after the phonetic order；

Fig. 6 is an optional flow diagram of interface operation method provided by the embodiments of the present application；

Fig. 7 is to show that predetermined cursor reminds the interface schematic diagram of label in the embodiment of the present application in current interface；

Fig. 8 is an optional flow diagram of interface operation method provided by the embodiments of the present application；

Fig. 9 is an optional flow diagram of interface operation method provided by the embodiments of the present application；

Figure 10 is the embodiment of the present application interface operation method application scenarios schematic diagram；

Figure 11 is an optional flow diagram of interface operation method provided by the embodiments of the present application；

Figure 12 is the structural schematic diagram of interface operating device provided by the embodiments of the present application；

Figure 13 is an optional flow diagram of interface operation method provided by the embodiments of the present application；

Figure 14 is an optional flow diagram of interface operation method provided by the embodiments of the present application；

Figure 15 is an optional flow diagram of interface operation method provided by the embodiments of the present application；

Figure 16 is an optional flow diagram of interface operation method provided by the embodiments of the present application.

Specific embodiment

In order to keep the purposes, technical schemes and advantages of the application clearer, below in conjunction with attached drawing to the application make into It is described in detail to one step, described embodiment is not construed as the limitation to the application, and those of ordinary skill in the art are not having All other embodiment obtained under the premise of creative work is made, shall fall in the protection scope of this application.

In the following description, it is related to " some embodiments ", which depict the subsets of all possible embodiments, but can To understand, " some embodiments " can be the same subsets or different subsets of all possible embodiments, and can not conflict In the case where be combined with each other.

Unless otherwise defined, technical and scientific term all used in the embodiment of the present application is implemented with the application is belonged to The normally understood meaning of those skilled in the art of example is identical.Term used in the embodiment of the present application is intended merely to describe The purpose of the embodiment of the present application, it is not intended that limitation the application.

The interface operation method provided in the embodiment of the present application in order to better understand, first to voice control in the related technology The scheme of interface operation processed carries out analytic explanation.

In the related technology, for the electronic equipment with voice control function, voice is usually installed on an electronic device Assistant (for example, small love classmate, Siri and small E etc.) etc. is used for the voice application (Application, APP) of speech recognition.It is logical Often, the electronic equipment semantic task that the pre-configured electronic equipment of meeting can be identified and be executed before factory, for example, inquiry is current The semantic task of time, the semantic task of call contact, request play semantic task of music etc..Alternatively, in the related technology Can also in such a way that code buries a little, need statistical data place implantation N line code, the critical behavior of counting user, Or control point is stated in such a way that code buries a little.

As shown in Figure 1, user carries out video using electronic equipment and broadcasts for the interface schematic diagram of voice control in the related technology It puts, when user wants pause video playing, phonetic order (broadcasting please be suspend) can be said to electronic equipment, then electronics is set Standby voice application can parse phonetic order, determine the phonetic order, and determine corresponding with the phonetic order Operation, last electronic equipment realizes the operation for clicking pause button, suspends video playing.

It is that can to operate element predefined based on interface good it can thus be seen that sound control method in the related technology Label go identification that can operate the control of element, exactly " suspend " for example, the correspondence voice label of element " pause " 110 can be operated, That is, the corresponding label of the control that can operate element is exactly the name of the control.This can operate the label " pause " of element, in electronics It has been defined before equipment factory or playout software installation.

So, it is just not difficult to obtain, in the voice control scheme of the relevant technologies, the prior art has at least the following problems:

1) label that interface can operate element pre-defines, that is, carries out relevant exploitation, then, it is right In some history APP or program, minority's and without related voice operation exploitation APP, if had in these APP It is some it is not predetermined operate element, then, voice control scheme in the related technology then can not achieve to these APP into Row voice control, that is, it is incompatible or be unable to control that voice control scheme in the related technology can have software and voice control The problem of.

2) control point is stated by way of bury a little to interface, it is inflexible, it is frequently present of omission or change Lead to incompatible problem.

3) phonetic order flexibly can not accurately navigate to the interface of software, for example, there are have at two in current interface There is when operating element of same names, electronic equipment is unable to judge accurately out that phonetic order is corresponding to operate that element is practical to be referred to Which, thus there are problems that position inaccurate.

4) since the label that interface can operate element pre-defines, user needs when executing voice control It will be clear that each phonetic order for operating element, in this way, extra cost can be carried out to user's operation learning tape, it can for new user It can not know immediately that the corresponding phonetic order of each function of software.

A kind of interface operation side is provided based at least one above-mentioned problem, the embodiment of the present application present in the relevant technologies Method, device, equipment and storage medium can improve the software compatibility, and can reduce use suitable for any software and system Family operation difficulty and user's learning cost improve user experience so that speech control process is easy to learn.

Illustrate the exemplary application of interface operation equipment provided by the embodiments of the present application below, it is provided by the embodiments of the present application Equipment may be embodied as screen sound equipment, laptop, tablet computer, desktop computer, mobile device (for example, mobile phone, Portable music player, personal digital assistant, specific messages equipment, portable gaming device) etc. various types of users it is whole End.In the following, by exemplary application when illustrating that equipment is embodied as terminal.

Referring to fig. 2, Fig. 2 is an optional configuration diagram of interface operation system 20 provided by the embodiments of the present application, A voice control application is supported to realize, terminal 200 (illustrating terminal 200-1 and terminal 200-2) passes through network 300 connection servers 400, network 300 can be wide area network or local area network, or be combination.

Terminal 200 is shown on graphical interfaces 210 (illustrating graphical interfaces 210-1 and graphical interfaces 210-2) The current interface of APP operates element for determine current interface on electronic equipment；According to pre-arranged code rule, to described Element can be operated to be encoded, the mark that can operate element is obtained, by the mark for operating element show with institute The corresponding position of element can be operated by stating；And acquire the phonetic order including the mark that can operate element；Server 400 is used for The phonetic order sent to terminal parses, and returns to parsing result to terminal, so that terminal 200 can be based on parsing knot Fruit responds the phonetic order, to realize the operation to the current interface.

It is the structural schematic diagram of terminal 200 provided by the embodiments of the present application referring to Fig. 3, Fig. 3, terminal 200 shown in Fig. 3 is wrapped It includes: at least one processor 310, memory 350, at least one network interface 320 and user interface 330.It is each in terminal 200 A component is coupled by bus system 340.It is understood that bus system 340 is for realizing the connection between these components Communication.Bus system 340 further includes power bus, control bus and status signal bus in addition in addition to including data/address bus.But For the sake of clear explanation, various buses are all designated as bus system 340 in Fig. 3.

Processor 310 can be a kind of IC chip, the processing capacity with signal, such as general processor, number Word signal processor (DSP, Digital Signal Processor) either other programmable logic device, discrete gate or Transistor logic, discrete hardware components etc., wherein general processor can be microprocessor or any conventional processing Device etc..

User interface 330 include make it possible to present one or more output devices 331 of media content, including one or Multiple loudspeakers and/or one or more visual display screens.User interface 330 further includes one or more input units 332, packet Include the user interface component for facilitating user's input, for example keyboard, mouse, microphone, touch screen display screen, camera, other are defeated Enter button and control.

Memory 350 can be it is removable, it is non-removable or combinations thereof.Illustrative hardware device includes that solid-state is deposited Reservoir, hard disk drive, CD drive etc..Memory 350 optionally includes one geographically far from processor 310 A or multiple storage equipment.

Memory 350 includes volatile memory or nonvolatile memory, may also comprise volatile and non-volatile and deposits Both reservoirs.Nonvolatile memory can be read-only memory (ROM, Read Only Memory), and volatile memory can To be random access memory (RAM, Random Access Memory).The memory 350 of the embodiment of the present application description is intended to Memory including any suitable type.

In some embodiments, memory 350 can storing data to support various operations, the example of these data includes Program, module and data structure or its subset or superset, below exemplary illustration.

Operating system 351, including for handle various basic system services and execute hardware dependent tasks system program, Such as ccf layer, core library layer, driving layer etc., for realizing various basic businesses and the hardware based task of processing；

Network communication module 352, for reaching other calculating via one or more (wired or wireless) network interfaces 320 Equipment, illustrative network interface 320 include: bluetooth, Wireless Fidelity (WiFi) and universal serial bus (USB, Universal Serial Bus) etc.；

Input processing module 353, for one to one or more from one of one or more input units 332 or Multiple user's inputs or interaction detect and translate input or interaction detected.

In some embodiments, device provided by the embodiments of the present application can realize that Fig. 3, which is shown, to be deposited using software mode The interface operating device 354 in memory 350 is stored up, can be the software of the forms such as program and plug-in unit, including following software Module: coding module 3541, display module 3542, identification module 3543 and respond module 3544, these modules be in logic, Therefore it can be combined arbitrarily according to the function of being realized or further split.The function of modules will be described hereinafter Energy.

In further embodiments, device provided by the embodiments of the present application can be realized using hardware mode, as an example, Device provided by the embodiments of the present application can be the processor using hardware decoding processor form, be programmed to perform this Shen Please embodiment provide interface operation method, for example, the processor of hardware decoding processor form can using one or more Application specific integrated circuit (ASIC, Application Specific Integrated Circuit), DSP, programmable logic Device (PLD, Programmable Logic Device), Complex Programmable Logic Devices (CPLD, Complex Programmable Logic Device), field programmable gate array (FPGA, Field-Programmable Gate ) or other electronic components Array.

Below in conjunction with the exemplary application and implementation of terminal provided by the embodiments of the present application, illustrate that the embodiment of the present application mentions The interface operation method of confession.

Referring to fig. 4, Fig. 4 is an optional flow diagram of interface operation method provided by the embodiments of the present application, will The step of showing in conjunction with Fig. 4 is illustrated.

Step S401, determine current interface on electronic equipment operates element.

It here, include that at least one can operate element in the electronic equipment current interface, the element that operates is energy Enough carry out the interface element of operation processing, that is to say, that the operation processing is that can appoint by mouse, stylus and finger etc. It anticipates a kind of processing mode that operating body is operated.For example, the interface element of clicking operation can be carried out, dragging behaviour can be carried out The interface element of work can carry out interface element of long press operation etc..

The current interface of the electronic equipment can be any one user interface (User that electronic equipment can be shown Interface, UI).For example, the interface or the interface of mobile terminal APP etc. of system software.

The element that operates can show that user can by the display screen of electronic equipment in electronic equipment current interface Element can be operated to be immediately seen this.Certainly, the element that operates can not also be shown in electronic equipment current interface, be used Family can not see that this can operate element by the display screen of electronic equipment, for example, for some lesser electricity of display screen screen Sub- equipment, when showing current interface, since the content in current interface is more, can display portion content, user The page can be dragged by left and right or up and down, and current interface other content is checked with realizing, then, element can be operated at this time Can then see on the display screen of electronic equipment, can also cannot see on the display screen of electronic equipment, only by pair Display interface on electronic equipment display screen is dragged, and could be made other that can operate element and is revealed.

Step S402 encodes the element that operates of current interface on electronic equipment, obtains according to pre-arranged code rule To the mark for operating element.

Here, it is encoded according to pre-arranged code rule, can be and the element that can all operate in current interface is carried out Coding, element can be operated to the part in current interface by, which being also possible to, encodes.

To that can operate after element encodes, each mark for operating element is obtained, it is here, described to operate member The mark of element, which can be this, can operate the number of element.For example, can using number or letter in current interface can Operation element is encoded, then the mark for obtaining that element can be operated is the mark comprising number and letter.It is obtained in this way can The mark of operation element more readily identifies.

For current interface, each mark for operating element is different from the mark that other can operate element, in this way can be with It distinguishes and operates element in current interface.

For different display interfaces, same class, which can operate element, can have identical mark, it is possible to have completely Different marks, alternatively, it is inhomogeneous operate element can have identical mark or it is inhomogeneous operate element tool There is entirely different mark.It is to be understood that the same class can operate element can be it is identical operate element, Be also possible to corresponding same operation processing operates element.

The mark for operating element is shown and is operating the corresponding position of element with described by step S403.

Here, with it is described operate the corresponding position of element can be it is described operate on element or it is described can The side of element is operated, and close to the position that can operate element, in this way, the corresponding display one in each side for operating element Mark, user more can intuitively see the mark that can operate element.

In some embodiments, when can operate the mark of element described in the display, the mark can be amplified and is shown, that , or screen font lesser situation more for screen content, due to mark be displayed magnified, user can be made more It is easy to see the mark that can operate element.

It as shown in Figure 5A, is that the embodiment of the present application shows the optional interface schematic diagram that can operate component identification, In In Fig. 5 A, the interface operation method of the embodiment of the present application is applied to browsing device net page, can at least one in browsing device net page Element 501 is operated, for example, " consulting ", " video ", " picture ", " knowing ", " library ", " discussion bar " and webpage in Fig. 5 A Option news " the leading Online Video media platform of Tencent's video-China ", these are that current interface operates element, It can be operated after element encodes to these, each mark 502 for operating element corresponding one and determining, for example, " consulting " Be identified as SA, " video " be identified as C, " picture " be identified as AC, " knowing " be identified as DC, " library " is identified as FC, " discussion bar " are identified as JC and webpage option news " the leading Online Video media platform of Tencent's video-China " Being identified as SJ ..., these marks are shown in correspondence and can operate the side of element, correspond with each element that operates, when with Family is seen when can operate element, also it can directly be seen that this can operate the mark of element.

It as shown in Figure 5 B, is that the embodiment of the present application shows the optional interface schematic diagram that can operate component identification, In In Fig. 5 B, the interface operation method of the embodiment of the present application is applied to the interface APP, and the embodiment of the present application is with instant messaging APP (example Such as, wechat) for be illustrated, certainly, a kind of any other APP is also all suitable for.When user's suitable for movable terminal operating When wechat APP, in the opening page of wechat, have at least one can operate element 511, for example, query icon, function in Fig. 5 B Can item icon, chatting object etc., these are that current interface operates element, carry out encoding it that can operate element to these Afterwards, each mark 512 for operating element corresponding one and determining, for example, query icon be identified as AA, function items icon be AB, Chatting object be AC (if show multiple chatting objects in current interface, the corresponding mark of each chatting object, and The mark of each chatting object is different) ... these marks are also shown in correspondence and can operate the side of element, with it is each can It operates element to correspond, when user, which sees, to operate element, also it can directly be seen that this can operate the mark of element.

It as shown in Figure 5 C, is that the embodiment of the present application shows the optional interface schematic diagram that can operate component identification, In In Fig. 5 C, the interface operation method of the embodiment of the present application is applied to smaller screen terminal, here, the display screen screen of electronic equipment compared with It is small, and interface content to be shown is more, if all showing current interface content on a display screen, in current interface Text or the display such as picture content can be reduced, be not easy user and check, shown when with the size of normal text or picture If, the display screen of electronic equipment cannot show the full content of current interface.

The embodiment of the present application continues by taking the browser interface in above-mentioned Fig. 5 A as an example, when the interface is shown in smaller screen terminal When, as shown in Figure 5 C, only can display portion content, the content of corresponding other parts cannot then show completely on a display screen. At this point, electronic equipment in addition to in current interface " consulting ", " video ", " picture ", " knowing ", " library ", " discussion bar ", " adopt Purchase ", " map ", " more " and the option news " the leading Online Video media platform of Tencent's video-China " of webpage etc. Can operate except element encoded, can also be shown on electronic equipment display screen it is additional can operate element, in Fig. 5 C Sliding to the left is slided to the right, the slide of upward sliding and slide downward four direction, also, to four slides Also encoded, for example, slide to the left be identified as S1, slide to the right be identified as S2, upward sliding be identified as S3 and to Lower slider is identified as S4.If user wants to check the content on the right side of currently displayed interface, voice can be sent and referred to S1 is enabled, so that interface is slided to the left, with the content on the right side of display interface, in this way, upper and lower, the left and right to current page may be implemented Sliding, to realize in smaller screen terminal with normal text size and normal picture size display interface.

Certainly, in some embodiments, the additional element that can operate can also correspond to amplifying operation, reduction operation etc. times The processing operation for a kind of pair of current interface of anticipating, the embodiment of the present application is without limitation.

Step S404, identification include the phonetic order of the mark that can operate element.

Here, the electronic equipment includes voice recognition unit, and the voice recognition unit may include sound transducer, By being identified to obtain the phonetic order to sound transducer voice collected, the phonetic order user instruction is to working as Any element that operates on front interface is operated.

It include the mark that can operate element in the embodiment of the present application, in the phonetic order, when electronic equipment identifies When to the phonetic order, parse the phonetic order with obtain include in the phonetic order described in can operate the mark of element Know, is operated in this way, can determine that user is intended to which can operate element to.

It for example, include that can operate elements A, B and C in current interface, wherein can operate elements A is identified as 11, can Operation element B is identified as 12, and can operate Elements C is identified as 13, then, when user wants to carry out a little to can operate element B When hitting operation, then can say against electronic equipment " it please click, 12 " or directly say the mark " 12 " that can operate element, this When, electronic equipment obtains the phonetic order and recognizing the voice of user.

Step S405 responds the phonetic order, executes respective operations to the current interface.

Here, electronic equipment responds the phonetic order, after recognizing the phonetic order to described The corresponding element that operates of the included mark for operating element executes corresponding operation processing in phonetic order.

For example, if including that can operate elements A, B and C in current interface, wherein being identified as elements A can be operated 11, can operate element B is identified as 12, and can operate Elements C is identified as 13, when user refers to against the voice that electronic equipment is said It enables as that " please click, when 12 ", then electronic equipment makes a response at this time, executes clicking operation to that can operate element B.

Interface operation method provided by the embodiments of the present application, according to pre-arranged code rule, to current interface on electronic equipment The element that operates encoded, obtain the mark that can operate element；By the mark for operating element show with It is described to operate the corresponding position of element.In this way, can make the method for the embodiment of the present application suitable for any software and system, Any software and system can be encoded, any relevant exploitation was carried out without software, and improved software compatibility Property；Also, display can operate the mark of element directly in current interface, and such user can directly be said by voice mode The operation to that can operate element can be completed in the mark seen, reduces user's operation difficulty and user's learning cost, so that language Sound control process is easy to learn, improves user experience.

In some embodiments, when showing cursor in the current interface, the element that operates includes the light Mark, the mark for operating element includes being used to indicate at least one direction signs of the moving direction of cursor；Such as Fig. 5 D It is shown, it is that the embodiment of the present application shows that the optional interface schematic diagram that can operate component identification can operate member in figure 5d Element includes cursor 541, and the cursor 541 can be mobile to any one direction, for example, moving upwards, downwards, to the left and to the right Dynamic, the mark for operating element is used to indicate the cursor mark mobile to any one direction；For example, with reference to figure 5D, what cursor 541 was moved to the left be identified as L1, move right be identified as R1, move up be identified as U1, move down It is identified as D1；Alternatively, be moved to the left be identified as " left side ", move right be identified as " right side ", move up be identified as "upper", What is moved down is identified as "lower", that is, directly displays the corresponding text of moving direction；It, can be with alternatively, in some embodiments Including the mark moved to the 45 degree of directions in upper left side, the mark moved to the 45 degree of directions in upper right side, 45 degree of lower section directions are moved to the left The mark of dynamic mark, 45 degree of directions movement to the right, the embodiment of the present application is without limitation.

Based on the mark that can operate element shown in Fig. 5 D, accordingly, identification can operate member including described in step S404 The phonetic order of the mark of element, can be realized by following steps:

Step S4041, identification include the phonetic order of the direction signs.

The phonetic order is responded in step S405, and respective operations are executed to the current interface, can pass through following step It is rapid to realize:

Step S4051 responds the phonetic order, and the light in direction corresponding with the direction signs is executed to the cursor Mark moving operation.

Here, user can say the mark in the mobile direction of desired cursor by voice, and then electronic equipment identification is used The mark of moving direction in the voice of family, and the phonetic order of user is responded, cursor moving operation is executed to the cursor.

For example, the desired cursor of user moves right, then user can say " R1 " against electronic equipment, then at this time Electronic equipment determines that user wants cursor and moves right, and therefore, controls the cursor and moves right.

In the embodiment of the present application, the distance that cursor moves every time can be determining length, be also possible to random-length. For example, shown in current interface it is multiple when operating element, and every two can operate the display distance between element it is equal when, The distance that then cursor moves every time can be determining length, that is, the distance moved every time, which is equal to two, to be operated between element Display distance, in this way, cursor can operate the display of element from first when user speech indicating cursor moves right Position is moved to second that this can operate on the right side of element and can operate at the display position of element.

Interface operation method provided by the embodiments of the present application indicates that the cursor on display interface is moved by phonetic order It is dynamic, to not have to the movement that user realizes cursor in current interface by operating bodies such as mouses, the function of voice control is increased, More operation selections are provided for user, improve user experience.

In some embodiments, when the cursor is by when moving to reach target position, user can pass through phonetic order, example The operation for clicking cursor is executed, such as " pressing " or " click " or " selection " to simulate the click of left mouse button.Some embodiments In, user can also assign the combined value that letter or number or both combines to cursor, to replace text-to-speech instruction above, Such as " 11 " or " L1 " equivalence can be given to and press cursor operations, when user says these phonetic orders, execute click The operation of cursor, to simulate the click of left mouse button.

In some embodiments, when showing cursor in the current interface, the element that operates includes the light Mark, the mark for operating element include being used to indicate the menu identification for opening right-click menu；It as shown in fig. 5e, is the application Embodiment shows the optional interface schematic diagram that can operate component identification, and in Fig. 5 E, can operate element includes cursor 541, the cursor 541 is located at a position determined in current interface.On the side of the cursor 541, mouse pattern is shown 551, menu identification M1 is shown on mouse pattern 551, the menu identification M1 is used to indicate opening right-click menu, that is, It says, corresponding operate of the menu identification M1 operates for the right button of mouse.

Accordingly, phonetic order of the identification including the mark that can operate element in step S404, can be by following Step is realized:

Step S4141, identification include the phonetic order of the menu identification.

The phonetic order is responded in step S405, and respective operations are executed to the current interface, can pass through following step It is rapid to realize:

Step S4151 responds the phonetic order, right click operation is executed in the display position of the cursor, to beat Open the right-click menu.

Here, user can say the menu identification M1 for executing right click operation by voice, and then electronic equipment is known Not Chu menu identification M1 in user speech, and respond the phonetic order of user, execute right button in the display position of the cursor Clicking operation, to open the right-click menu.

In some embodiments, when the cursor is by when moving to reach target position, user can pass through phonetic order, example If " right button is pressed " or " clicking by right key " or " right button selection " Lai Zhihang clicks the operation of cursor by right key, to simulate right mouse button It clicks.In some embodiments, user can also assign the combined value that letter or number or both combines to right button cursor, to replace Text-to-speech instruction above, such as " 22 " or " R1 " equivalence can be given to and press right button cursor operations, when user says When these phonetic orders, the operation of right-click cursor is executed, to simulate the click of right mouse button.

It as illustrated in figure 5f, is that the embodiment of the present application electronic equipment responds the interface schematic diagram after the phonetic order, electronics Equipment executes right click operation, opens right-click menu 552, includes at least one option in the right-click menu 552, for example, It may include back option, refresh option, print option and attributes section etc..

Interface operation method provided by the embodiments of the present application passes through the cursor position of phonetic order instruction in the display interface Right click operation is executed, and then shows right-click menu, more option of operation are provided for user, to improve user experience.

In some embodiments, electronic equipment can also receive user's other than it can receive the phonetic order Operation, for example, user operates by performed by operating body first, then, accordingly, the method can also include following step It is rapid:

Step S41 obtains first operation of the operating body on the electronic equipment.

Here, the operating body can be any one in mouse, stylus and user's finger, and the operating body can Realization executes first operation on the electronic equipment.

First operation can for clicking operation, choosing operation, long press operation and drag operation etc., any one can be real Existing mode of operation.

Step S42, when the corresponding position of first operation is corresponding with the position of mark display for operating element When, first operation is executed to the element that operates.

Here, the default model when the corresponding position of first operation in the position of the mark display for operating element Within enclosing, it can think that the corresponding position of first operation is corresponding with the position of mark display for operating element. For example, first time operation is clicking operation, then, the click location of first operation, which can be, described operates element Mark display position, at this point, show user to this can operate element carry out clicking operation, therefore respond the click behaviour Make, first operation is executed to the element that operates to realize.

It in some embodiments, is one of interface operation method provided by the embodiments of the present application optional referring to Fig. 6, Fig. 6 Flow diagram, can also be performed after step S401 based on Fig. 4:

Step S601 receives wake up instruction, controls the electronic equipment based on the wake up instruction received and is in wake-up shape State.

Here, the wake up instruction is used for so that electronic equipment is in wake-up states, when the electronic equipment is in wake-up When state, then the function of voice control interface operation is in the open state, and electronic equipment executes the interface operation method.

In the embodiment of the present application, wake-up states, then corresponding booting can be in after the electronic equipment is switched on Instruction is the wake up instruction；Can also be after electronic equipment be switched on, and the voice application for speech recognition is run Later, electronic equipment is in wake-up states, then corresponding voice application operating instruction is the wake up instruction.

The embodiment of the present application is corresponded in step S402 according to pre-arranged code rule, is encoded to the element that operates, The content of the mark that can operate element is obtained, can be realized by following steps:

Step S602, when the electronic equipment is in the wake-up states, according to the pre-arranged code rule, to described Element can be operated to be encoded, the mark that can operate element is obtained.

Here, when electronic equipment is in the wake-up states, show the voice control interface operation of electronic equipment at this time Function it is in the open state, then electronic equipment can execute the interface operation method, therefore, carry out voice control it Before, it needs first to encode the element that operates in current interface, with each mark for operating element of determination, and then can The mark of operation element is shown on the display screen of electronic equipment, so that user checks, allows user can according to what is seen The mark for operating element carries out corresponding voice-controlled operations process.

In the embodiment of the present application, the voice control function of electronic equipment is waken up, is only called out when electronic equipment is in Wake up state when, just realize the speech control process, in this way, can allow user choose whether according to actual needs using The voice control function sends wake up instruction to electronic equipment, so that the voice control function of electronic equipment when it is desired to be used It opens, when not needing in use, not having to then send wake up instruction to electronic equipment, the voice control function of electronic equipment is closed, It is can reduce in this way when not needing using voice control function, electronic equipment also carries out voice control acquisition phonetic order and made At energy consumption, and provide the selection of different use demands for user, improve user experience.

It in some embodiments, include action type in the wake up instruction；Accordingly, according to pre- described in step S402 If coding rule, the element that operates of current interface on electronic equipment is encoded, obtains the mark that can operate element, It can also be realized by following steps:

Step S4021 will be able to carry out and the action type pair when receiving the wake up instruction in current interface The operation answered operates element, and element can be operated by being determined as target.

Here, the action type can be any one action type, for example, the action type may include clicking Action type, drag operation type and long press operation type etc..

When the action type is clicking operation type, operation corresponding with the action type can be for in interface Element can be operated any one clicking operation type such as clicks, double-clicks and chooses；When the action type is drag operation class When type, operation corresponding with the action type can for the dragging of scroll bar in interface, to showing that text drags in interface It is dynamic to wait any one drag operation type；It is corresponding with the action type when the action type is long press operation type Operation can be to generate other can to operate element to can operate element in interface and carry out long-pressing.

In the embodiment of the present application, member is operated by be able to carry out operation corresponding with the action type in current interface Element, element can be operated by being determined as target, that is to say, that the element that can all operate in current interface screened, it will The element that operates for executing corresponding with action type operation is selected as target and can operate element, for being able to carry out other The element that operates of action type does not elect.

Step S4022 can operate position of the element in the current interface according to the target, successively to each target Element can be operated to be encoded, the mark of element can be operated by obtaining each target.

Here, after determining that target can operate element, element only can be operated to target and encoded, wherein encoded Journey can be with are as follows: can operate position of the element in the current interface according to the target, successively can operate member to each target Element is encoded.For example, position of the element in current interface can be operated according to target, according to fall down from above and/or from Left-to-right sequence successively carries out encoding each element that operates.

The scheme of the embodiment of the present application corresponds to following scene: for current display interface, if user merely desires to execute click Operation, and element can much be operated by showing in current interface, then, in order to enable cataloged procedure is easier to realize, encodes Speed improves, and shows that interface when can operate component identification is more succinct, then can be when being encoded and identifying display, only The corresponding element that operates of clicking operation executed, which is handled, to be want to user.At this point, user is when activating voice control function, And when issuing wake up instruction, the action type can be added in wake up instruction, is called out for example, wake up instruction can be voice It wakes up and instructs, then user can say " please execute clicking operation " against electronic equipment.In this way, electronic equipment is then only to being able to carry out a little The element that operates for hitting operation is encoded.

In some embodiments, voice of the identification including the mark that can operate element in the step S404 shown in Fig. 4 Instruction, can also be realized by following steps:

Step S4241, electronic equipment acquire voice messaging in real time.

Here, the voice collecting unit of electronic equipment is in running order, and the voice messaging around acquisition in real time, needs Illustrate, voice messaging collected can be effective phonetic order, be also possible to invalid voice messaging, for example, working as User and other people when being chatted beside electronic equipment, the conversational speech of user collected is exactly invalid voice messaging, should Invalid voice messaging can not form effective phonetic order.

It in the embodiment of the present application, needs to judge the voice messaging of acquisition, is with determination voice messaging collected No is the phonetic order that user wants that electronic equipment executes operation.

Step S4242 carries out semantic analysis to the voice messaging of acquisition, obtains semantic analysis result.

Semantic analysis is carried out to the voice messaging here it is possible to be electronic equipment, can also by Internet Server into Row semantic analysis.When executing semantic analysis, the voice messaging of acquisition can be input in preset machine learning model, be led to It crosses machine learning model to handle voice messaging, to obtain the result of voice analysis.

In some embodiments, when carrying out semantic analysis to the voice messaging by electronic equipment, when electronic equipment is adopted After collecting the voice messaging, i.e., by the voice messaging by presetting the engineering in semantic analysis software on electronic equipment Model is practised to be handled.

In other embodiments, it when carrying out semantic analysis by Internet Server, then can be realized by following steps:

Voice messaging collected is sent to server by step S4242a, electronic equipment.

Step S4242b, server carry out semantic analysis to the voice messaging, obtain semantic analysis result.

In the embodiment of the present application, is realized by Internet Server and the voice messaging of acquisition is carried out at semantic analysis Reason, can obtain more accurate semantic analysis result.

Step S4243 shows to include any mark for operating element in the voice messaging when the semantic analysis result When knowledge, the voice messaging is determined as include the mark that can operate element phonetic order.

Here, when the semantic analysis result shows to include that can operate the mark of element in the voice messaging, show The voice messaging is effective voice messaging, therefore the voice messaging is determined as effective phonetic order, the voice Instruction, which is used to indicate, operates the corresponding element that operates of the mark for operating element included in the voice messaging.

For example, if including that can operate elements A, B and C in current interface, wherein being identified as elements A can be operated 11, can operate element B is identified as 12, and can operate Elements C is identified as 13.If it is " modern that electronic equipment collects voice messaging Its weather is true ", after carrying out semantic analysis, determine in the voice messaging do not include grasping with any in current interface The corresponding voice of mark for making element, then delete the voice messaging, and acquires next voice messaging；If the voice letter of acquisition Breath is " please click 12 ", then determining the mark for including in the voice messaging and can operating element B after carrying out semantic analysis Corresponding voice thus responds the phonetic order it is thus determined that the voice messaging is an effective phonetic order, to can operate Element B executes clicking operation.

In some embodiments, it can also be performed after step S404 based on Fig. 4:

Step S410, when getting the phonetic order including the mark for operating element, in the correspondence mark Predeterminable area in, show that predetermined cursor reminds label, currently operate element to corresponding with the mark to remind Carry out cursor moving operation.

Here, the predetermined cursor remind label for remind user currently to user speech instruct in mark The corresponding element that operates is operated.The predetermined cursor reminds label to can have any one pattern, for example, described pre- Determine cursor to remind label to be the shapes such as hand-type label or arrow.

The predeterminable area of the correspondence mark can be the range of the certain distance around the mark, in general, The position of the side of the mark close to the mark can be determined as to the predeterminable area, the predetermined cursor reminds label Shown position is as close as possible to the mark.

As shown in fig. 7, being to show that predetermined cursor reminds the interface signal of label in the embodiment of the present application in current interface Figure, on the basis of Fig. 5 A, when the phonetic order of user is to carry out clicking operation to " picture ", then in the language for getting user After sound instruction, in response and reminds, be displayed next to hand-type label 701 in the A C that is identified as of " picture ".

It in some embodiments, is one of interface operation method provided by the embodiments of the present application optional referring to Fig. 8, Fig. 8 Flow diagram, can also be performed after step S404 based on Fig. 4:

Step S801 stores the phonetic order simultaneously when there is operation associated instruction corresponding with the phonetic order Etc. the operation associated instruction to be received.

In the embodiment of the present application, for some phonetic orders, operation associated instruction can be corresponding with, that is to say, that executing When operation, it is to be completed based on phonetic order and operation associated instruction corresponding with the phonetic order, only receives simultaneously Phonetic order and operation associated instruction, are just able to achieve the operation to that can operate element.

Here, it after getting the phonetic order, then needs to judge the phonetic order, determines institute's predicate Whether sound includes corresponding operation associated instruction, that is to say, that for the phonetic order got, need to judge its whether be with Other instructions combine to complete operation together.If it is judged that be it is no, then directly execute the phonetic order, if it is determined that knot Fruit be it is yes, then the operation associated instruction corresponding with the phonetic order to be received such as need.

Step S802 when receiving the operation associated instruction within a preset time, while responding the phonetic order With the operation associated instruction.

Here, the preset time can be what electronic equipment had been pre-set before factory, be also possible to user Customized setting is carried out according to the actual situation.

Step S803 forbids responding the voice and refers to when not receiving the operation associated instruction within a preset time It enables.

Here, when not receiving the operation associated instruction within a preset time, show current phonetic order not Completely, perhaps show that user is not intended to continue to execute the phonetic order or user and currently stops executing the phonetic order, Therefore, forbid responding the phonetic order, to stop executing the phonetic order.

Interface operation method provided by the embodiments of the present application only works as reception for needing the phonetic order of associated response When to phonetic order and operation associated instruction corresponding with phonetic order, just phonetic order is responded, so, it is possible to guarantee The exact operations to interface are realized in accurate response to phonetic order.

In some embodiments, the method also includes following steps:

Step S810 can operate the mark of element when the current interface updates described in deletion.

Here, after electronic equipment responds the phonetic order or electronic equipment refreshes current interface, currently Interface can update, updated display interface due to being changed, for front interface operate element, update May be not present on display interface afterwards, or may also carry out corresponding update, therefore, front interface operate The mark of element is no longer valid, so can operate the mark of element described in deleting.

Step S811 encodes the element that operates of updated display interface.

Here, since current interface is updated, element is operated including new on updated display interface, therefore It needs to recompile the element that operates of updated display interface, and then continues to execute voice operating.

Fig. 9 is an optional flow diagram of interface operation method provided by the embodiments of the present application, as shown in figure 9, It the described method comprises the following steps:

Step S901, when the electronic equipment is in the wake-up states, server determines that electronic equipment is taken in prezone Face operates element.

In the embodiment of the present application, server monitors electronic equipment in real time, when electronic equipment is in wake-up states, The interface operation method is realized by server and electronic equipment jointly.

Step S902, server encode the element that operates according to pre-arranged code rule, obtain described to grasp Make the mark of element.

The mark for operating element is sent to terminal by step S903, server.

Step S904, terminal, which shows the mark for operating element, is operating the corresponding position of element with described.

Step S905, terminal acquisition include the phonetic order of the mark that can operate element.

The phonetic order is sent to server by step S906, terminal.

Step S907, server carry out semantic analysis to the voice messaging of acquisition, obtain semantic analysis result.

Step S908 shows to include any mark for operating element in the voice messaging when the semantic analysis result When, server the voice messaging is determined as include the mark that can operate element phonetic order.

The semantic analysis result is sent to terminal by step S909, server.

Step S910, terminal respond the phonetic order when receiving the semantic analysis result, to realize to described The operation of current interface.

Interface operation method provided by the embodiments of the present application, by the interaction between server and electronic equipment, by servicing Device and electronic equipment realize the method jointly, the mark for being encoded to that can operate element by server, and can operating element Knowledge, which is shown in, operates the corresponding position of element with described.In this way, can be encoded for any software and system, it is not necessarily to Software carried out any relevant exploitation, improved the software compatibility；Also, display can operate element directly in current interface Mark, such user can directly say the mark seen by voice mode can be completed operation to that can operate element, User's operation difficulty and user's learning cost are reduced, so that speech control process is easy to learn, improves user experience.

In the following, will illustrate exemplary application of the embodiment of the present application in an actual application scenarios.

Operation for flat-type computer requires to be operated with operating bodies such as finger or styluses, thus nothing Method discharges both hands.The embodiment of the present application provides a kind of method of position that identification can be clicked automatic on the screen, by acquiring language The operation of sound command control software.

Since the voice control of Siri etc is popular, but it can not solve the old software of history, or not The control of Software Development Kit (Software Development Kit, SDK) integrated software is carried out to Siri.The application Can carry out early warning control for any software or web page, simple interaction, realize by traditional software it is seamless move to language Under the system of sound control.

The key point of the embodiment of the present application is: proposing the method that a kind of pair of windows traditional software carries out coding displaying； It is proposed that a kind of pair of mobile terminal software carries out the method that interface element coding is shown；It proposes to show that software can operate when equipment wakes up Mark after element coding, the mark autocoding that can operate element generate, intervene before not needing anything or integrate any SDK；When proposing software responses click action, it can carry out clicking while showing click effect at interface display coordinate position； The interaction logic for proposing a kind of voice control software action is accurately controlled soft by starting session, action order, position command Part movement.

Interface operation method provided by the embodiments of the present application has reformed the operation shape for needing mouse-keyboard in the related technology Formula.The embodiment of the present application shows product using effect for having screen sound equipment, it should be noted that the method for the embodiment of the present application Actually cover any electronic equipment comprising audio sensor and screen.

Figure 10 is the embodiment of the present application interface operation method application scenarios schematic diagram, and as shown in Figure 10, user 1001 is main With there is screen sound equipment 1002 to interact, there is screen sound equipment 1002 to include at least audio collection sensor 1021 and visualization screen 1022, audio collection sensor 1021 can acquire sound instruction, and visualization screen 1022 can be shown coding and action Effect.When realizing interface operation, as shown in figure 11, mainly comprise the steps that

Step S1101, user, which issues phonetic order wake-up, screen sound equipment, makes have screen sound equipment to carry out continuing reception to voice.

For example, user can be against there is screen sound equipment to say " ding-dong, ding-dong ", at this point, there is the screen of screen sound equipment to keep being always on.

Step S1102 starts target software.

For example, user can star mapping software.User is against there is screen sound equipment to say " starting ' mapping software ' ", at this point, having Shield sound equipment and starts corresponding mapping software application.

Step S1103, user send operation activation instruction (wake up instruction in corresponding any of the above-described embodiment).

Here, the operational order may include any one action type, for example, the action type is to click behaviour Make.So, having screen sound equipment to show the software on the screen, each can interact the element that operates of click, and show and use word Mother encode after mark.

Step S1104, user say specific mark, and the electronic equipment element that operates corresponding to the mark executes correspondence Movement.

Step S1105 shows hand-type label in screen taps position, indicates in the position during execution movement Click action occurs.

So far, one-off interaction is completed.

Step S1106, user can continue cycling through the movement for executing step S1103 to step S1105.

It should be noted that movement performed in the embodiment of the present application includes the conventional tactiles such as click, long-pressing and sliding Shield the manipulable mode of gesture.

The interface operation method of the embodiment of the present application can be applied not only in Windows system, also can be applied to pacify In the mobile systems such as tall and erect system (Android) and iOS system.

Figure 12 is the structural schematic diagram of interface operating device provided by the embodiments of the present application, as shown in figure 12, the interface Operating device 1200 includes: speech analysis device 1201, software starter 1202, web starter 1203 and clicks controller 1204.

Wherein, the speech analysis device 1201, for receiving voice messaging, and can be by voice messaging escape at text Or instruction, and the content after escape is sent to click controller.When the speech analysis device is to the parsing energy of voice messaging When power is limited, voice messaging can also be uploaded into Internet Server by network, pass through the mould with machine learning ability Type carries out parsing identification to voice messaging, and recognition result is sent to click controller.

The software starter 1202, for carrying out software starting to traditional native applications, wherein the native applications Refer to the executable software for having graphical interfaces in windows system, referring in mobile device has the APP of graphical interfaces soft Part.The effect of the software starter 1202 is the movement of dispatcher software starting pull-up, and after starting software, is supervised in real time The variation of control software interface element (operating element in corresponding above-described embodiment), is encoded for interface element, and When receiving phonetic order, the number of each interface element is shown on interface.

The web starter 1203, for starting to web page.The web starter 1203 realizes browser Function, pull-up browser carry out web page displaying.The effect of the web starter 1203 and software starter 1202 Act on it is identical, and for the movement of dispatcher software starting pull-up, and after starting software, real-time monitoring software Interface Element The variation of element is encoded for interface element, and when receiving phonetic order, each interface element is shown on interface Number.

The click controller 1204, for showing hand-type at the coordinate of interface element when there is interactive action generation Label, to show click effect.Meanwhile mouse click event is really sent, mouse is simulated by the instruction of equipment in screen interface Punctuate is hit or moving operation, realizes the interaction of software.

Based on above embodiments, the embodiment of the present application provides a kind of coding method of interface element, the embodiment of the present application again Method realize user and issue action command, the number of the automatic display interface element of screen interface is (in corresponding above-described embodiment The mark of element can be operated), after issuing coded command, the mouse simulated automatically to its position is clicked or mouse is mobile. In an encoding process, it using from coding rule, encodes without pre-edit, is encoded according only to the spatial order of traversal, and And show the number obtained after coding.Wherein, as shown in figure 13, cataloged procedure the following steps are included:

Step S1301 is mentioned from software interface automatically by starter (can be software starter or web starter) Take the interface element of visual interface and the coordinate of each interface element.

It should be noted that not needing to encode for not available interface element, for example, not can be carried out for some There is no need to be encoded for the interface element of clicking operation or drag operation.

Step S1302 is encoded to each interface element respectively according to sequence of positions of the interface element in current interface.

For example, can be encoded since AA, add up one by one in an encoding process.The range of character can from A to Z, In this way, number default, which amounts to quantity, can achieve 23*23=529.

Step S1303 is more than 529 situations for same interface median surface element, starts tri-bit encoding.

For example, being encoded using ADC number to interface element, in this way, number amount total is up to 12167.

Figure 14 is an optional flow diagram of interface operation method provided by the embodiments of the present application, such as Figure 14 institute Show, the described method comprises the following steps:

Step S1401, for windows software, for the first time in use, registering APP first.

Step S1402, user say APP name by voice, to start corresponding A PP.

Step S1403 persistently judges whether to receive effective wake up instruction after APP starting.

If receiving effective wake up instruction, S1404 is thened follow the steps；Otherwise, terminate process.

Step S1404 traverses all interface elements in APP software current interface.

Step S1405 is encoded for each interface element, obtains the number of each interface element.

Step S1406 shows the number in the corresponding response position of the interface element.

At this point, the Show Button encodes on interface, and waits and instructing in next step.

Step S1407, the phonetic order of electronic equipment continuous collecting user.

Step S1408 parses phonetic order collected, obtains parsing result.

Step S1409 responds the parsing result, executes the corresponding movement of the phonetic order.

Figure 15 is an optional flow diagram of interface operation method provided by the embodiments of the present application, such as Figure 15 institute Show, during primary complete individually interactive voice, the described method comprises the following steps:

Step S1501, audio sensor receive voice messaging.

Step S1502 carries out semantic analysis processing to the voice messaging, the voice messaging is parsed into text information Or command information.

Here, perhaps the text information or command information are sent out after command information obtaining the text information Give click controller.

Step S1503 clicks controller and judges whether that there are also subsequent instructions.

Here, if it is judged that be it is no, then execute S1504；If it is judged that be it is yes, then execute S1506.

Step S1504, the corresponding behavior operation of simulation, triggers screen operator directly at the coordinate of screen reference numeral.

For example, the screen operator can be clicking operation or moving operation.

Step S1505, cartoon display screen curtain operates at the top layer respective coordinates of interface.

For example, can be by showing hand-type diagram in click location, to show screen operator.

Step S1506 clicks controller and keeps in text information or command information, and return step S1501 etc. is waiting Receive subsequent instructions.

Step S1507, if it exceeds there are no subsequent instructions to report for preset time, then current sessions terminate.

In some embodiments, it after interface is refreshed, needs again to encode interface element, as shown in figure 16, It is an optional flow diagram of interface operation method provided by the embodiments of the present application, the described method comprises the following steps:

Step S1601, after interface is refreshed, software starter monitors the interface information of Current software.

Step S1602 parses the interface element in the visual range at the interface after refreshing, to the interface visual model after refreshing Interface element in enclosing is retrieved.

Step S1603 encodes each interface element in the interface after refreshing.

Here, the coding of element is ranked up according to the discovery sequence of element, the same interface element on interface, In The different stages might have different numbers.

Step S1604 judges whether to receive effective wake up instruction.

If it is judged that be it is yes, then follow the steps S1605；If it is judged that be it is no, then terminate process.

Step S1605, when system receives wake up instruction, each interface element on the interface after the refreshing Corresponding number is shown at coordinate.

Step S1606 receives phonetic order, to instruction execution response action described in phonetic order.

For example, can instruct with voice responsive, the movements such as mouse is clicked or mouse is mobile are executed with simulation.

Interface operation method provided by the embodiments of the present application can realize voice control for any traditional software；And Direct voice control simulation is clicked, so that user's learning cost substantially reduces；It can be grasped in the interface real simulation mouse of software Make, it is not invasive to software；Dynamic coding is easy to use, and encodes and encoded using 2 or 3, facilitates speech recognition And parsing, improve interactive accuracy and fluency.

By the method for the embodiment of the present application, the voice operating transformation of most of traditional software may be implemented, due to Position command end is readily identified, can be improved the accuracy of control.And in interactive process, mouse hand is shown in the figure layer of interface Gesture can be improved the use feeling of user.

In other embodiments, cataloged procedure can also do a variety of transformation, for example, replacing letter to be encoded with number. In addition, can also increase custom coding on the basis of space compression schemes, such space can carry more information, with More complicated interaction logic may be implemented in this.

In other embodiments, it can also identify that specific click location and mouse are moved by the movement of video capture user Dynamic position.For example, when the movement of user be finger refer to upwards when, show user be intended to be located at screen above interface element into Row clicking operation, therefore, action video and parsing of the electronic equipment by shooting user obtain parsing result, are tied according to parsing Fruit carries out clicking operation to the interface element being located above screen.

Continue with explanation interface operating device 355 provided by the embodiments of the present application is embodied as the exemplary of software module Structure, in some embodiments, as shown in figure 3, the software module being stored in the interface operating device 354 of memory 340 can To include:

Coding module 3541, for according to pre-arranged code rule, on electronic equipment current interface operate element into Row coding obtains the mark that can operate element；

Display module 3542 is operating the corresponding position of element with described for showing the mark for operating element It sets；

Identification module 3543 includes the phonetic order of the mark that can operate element for identification；

Respond module 3544 executes respective operations to the current interface for responding the phonetic order.

In some embodiments, when showing cursor in the current interface, the element that operates includes the light Mark, the mark for operating element includes being used to indicate at least one direction signs of the moving direction of cursor；Accordingly, The identification module is also used to identify the phonetic order including the direction signs；

The respond module is also used to: being responded the phonetic order, is executed to the cursor corresponding with the direction signs Direction cursor moving operation.

In some embodiments, when showing cursor in the current interface, the element that operates includes the light Mark, the mark for operating element include being used to indicate the menu identification for opening right-click menu；

Accordingly, the identification module is also used to: identification includes the phonetic order of the menu identification；

The respond module is also used to: being responded the phonetic order, is executed and click by right key in the display position of the cursor Operation, to open the right-click menu.

In some embodiments, described device further include:

Module is obtained, for obtaining first operation of the operating body on the electronic equipment；

Processing module, for the position when the corresponding position of first operation and the mark display for operating element First operation is executed to the element that operates to when corresponding to.

In some embodiments, described device further include:

Receiving module is in based on the wake up instruction control electronic equipment received and is called out for receiving wake up instruction The state of waking up；

Accordingly, the coding module is also used to when the electronic equipment is in the wake-up states, according to described pre- If coding rule, the element that operates is encoded, obtains the mark that can operate element.

It in some embodiments, include action type in the wake up instruction；

Accordingly, the coding module is also used to when receiving the wake up instruction, will be able to carry out in current interface Operation corresponding with the action type operates element, and element can be operated by being determined as target；It can be operated according to the target Position of the element in the current interface successively can operate element to each target and encode, and obtaining each target can grasp Make the mark of element.

In some embodiments, the acquisition module is also used to carry out semantic analysis to the voice messaging of acquisition, obtains language Justice analysis result；It, will when the semantic analysis result shows in the voice messaging to include any mark for operating element The voice messaging be determined as include the mark that can operate element phonetic order.

In some embodiments, described device further include:

Remind label display module, for when getting the phonetic order including the mark for operating element, In In the predeterminable area of the corresponding mark, show that predetermined cursor reminds label, it is current to corresponding with the mark to remind Operate element carry out cursor moving operation.

In some embodiments, described device further include:

Memory module, for storing the voice and referring to when there is operation associated instruction corresponding with the phonetic order It enables and waits the operation associated instruction to be received；

The respond module is also used to when receiving the operation associated instruction within a preset time, while described in response Phonetic order and the operation associated instruction；When not receiving the operation associated instruction within a preset time, forbid responding The phonetic order.

In some embodiments, described device further include:

Removing module, for the mark of element can be operated described in deletion when the current interface updates；

The coding module is also used to encode the element that operates of updated display interface.

It should be noted that the description of the embodiment of the present application device, is similar, tool with the description of above method embodiment There is the similar beneficial effect of same embodiment of the method, therefore does not repeat them here.For undisclosed technical detail in present apparatus embodiment, It please refers to the description of the application embodiment of the method and understands.

The embodiment of the present application provides a kind of storage medium for being stored with executable instruction, wherein it is stored with executable instruction, When executable instruction is executed by processor, processor will be caused to execute method provided by the embodiments of the present application, for example, such as Fig. 4 The method shown.

In some embodiments, storage medium can be ferroelectric memory (FRAM, Ferromagnetic Random Access Memory), read-only memory (ROM, Read Only Memory), programmable read only memory (PROM, Programmable Read Only Memory), Erasable Programmable Read Only Memory EPROM (EPROM, Erasable Programmable Read Only Memory), band Electrically Erasable Programmable Read-Only Memory (EEPROM, Electrically Erasable Programmable Read Only Memory), flash memory, magnetic surface storage, CD or the read-only storage of CD The memories such as device (CD-ROM, Compact Disk-Read Only Memory)；Be also possible to include one of above-mentioned memory or The various equipment of any combination.

In some embodiments, executable instruction can use program, software, software module, the form of script or code, By any form of programming language (including compiling or interpretative code, or declaratively or process programming language) write, and its It can be disposed by arbitrary form, including be deployed as independent program or be deployed as module, component, subroutine or be suitble to Calculate other units used in environment.

As an example, executable instruction can with but not necessarily correspond to the file in file system, can be stored in A part of the file of other programs or data is saved, for example, being stored in hypertext markup language (HTML, Hyper Text Markup Language) in one or more scripts in document, it is stored in the single file for being exclusively used in discussed program In, alternatively, being stored in multiple coordinated files (for example, the file for storing one or more modules, subprogram or code section).

As an example, executable instruction can be deployed as executing in a calculating equipment, or it is being located at one place Multiple calculating equipment on execute, or, be distributed in multiple places and by multiple calculating equipment of interconnection of telecommunication network Upper execution.

In conclusion interface operation method provided by the embodiments of the present application, device, equipment and storage medium, including it is following The utility model has the advantages that

1) any software and system can be encoded suitable for any software and system, without soft Part carried out any relevant exploitation, improved the software compatibility.

2) display can operate the mark of element directly in current interface, and such user can directly be said by voice mode The operation to that can operate element can be completed in the mark seen out, reduces user's operation difficulty and user's learning cost, so that Speech control process is easy to learn, improves user experience.

3) only the element that operates of software current interface is encoded, coding is deleted after interface operation, to software Do not have it is invasive, can be in the interface real simulation mouse action of software.

4) dynamic coding is simply easily realized, coding using smaller digit letter or number realize, facilitate speech recognition and Parsing, improves interactive accuracy and fluency.

The above, only embodiments herein are not intended to limit the protection scope of the application.It is all in this Shen Made any modifications, equivalent replacements, and improvements etc. within spirit and scope please, be all contained in the application protection scope it It is interior.

Claims (12)

1. a kind of interface operation method characterized by comprising

According to pre-arranged code rule, the element that operates of current interface on electronic equipment is encoded, obtains described to operate The mark of element；

The mark for operating element is shown and is operating the corresponding position of element with described；

Identification includes the phonetic order of the mark that can operate element；

The phonetic order is responded, respective operations are executed to the current interface.

2. the method according to claim 1, wherein when showing cursor in the current interface, it is described can Operating element includes the cursor, and the mark for operating element includes being used to indicate at least the one of the moving direction of cursor A direction signs；

Accordingly, the identification includes the phonetic order of the mark that can operate element, comprising: identification includes the direction sign The phonetic order of knowledge；

The response phonetic order, executes respective operations to the current interface, comprising:

The phonetic order is responded, the cursor moving operation in direction corresponding with the direction signs is executed to the cursor.

3. the method according to claim 1, wherein when showing cursor in the current interface, it is described can Operating element includes the cursor, and the mark for operating element includes being used to indicate the menu identification for opening right-click menu；

Accordingly, the identification includes the phonetic order of the mark that can operate element, comprising: identification includes the menu mark The phonetic order of knowledge；

The response phonetic order, executes respective operations to the current interface, comprising:

The phonetic order is responded, right click operation is executed in the display position of the cursor, to open the right-click menu.

4. the method according to claim 1, wherein the method also includes:

Wake up instruction is received, the electronic equipment is controlled based on the wake up instruction received and is in wake-up states；

Accordingly, described that the element that operates is encoded according to pre-arranged code rule, it obtains described to operate element Mark, comprising:

When the electronic equipment is in the wake-up states, according to pre-arranged code rule, to it is described operate element into Row coding obtains the mark that can operate element.

5. according to the method described in claim 4, it is characterized in that, including action type in the wake up instruction；

Accordingly, described that the element that operates is encoded according to pre-arranged code rule, it obtains described to operate element Mark, comprising:

When receiving the wake up instruction, grasping for operation corresponding with the action type will be able to carry out in current interface Make element, element can be operated by being determined as target；

Position of the element in the current interface can be operated according to the target, element successively can be operated to each target and carried out Coding, the mark of element can be operated by obtaining each target.

6. the method according to claim 1, wherein the identification includes the language of the mark that can operate element Sound instruction, comprising:

Semantic analysis is carried out to the voice messaging of acquisition, obtains semantic analysis result；

When the semantic analysis result shows in the voice messaging to include any mark for operating element, by the voice Information be determined as include the mark that can operate element phonetic order.

7. method according to any one of claims 1 to 6, which is characterized in that the method also includes:

When getting the phonetic order including the mark for operating element, in the predeterminable area of the correspondence mark, It shows that predetermined cursor reminds label, currently the element that operates corresponding with the mark is operated with reminding.

8. method according to any one of claims 1 to 6, which is characterized in that the method also includes:

When there is operation associated instruction corresponding with the phonetic order, storing the phonetic order and waiting the pass to be received Join operational order；

When receiving the operation associated instruction within a preset time, while responding the phonetic order and described operation associated Instruction；

When not receiving the operation associated instruction within a preset time, forbid responding the phonetic order.

9. method according to any one of claims 1 to 6, which is characterized in that the method also includes:

When the current interface updates, the mark of element can be operated described in deletion；

The element that operates of updated display interface is encoded.

10. a kind of interface operating device characterized by comprising

Coding module, for encoding, obtaining to the element that operates of current interface on electronic equipment according to pre-arranged code rule To the mark for operating element；

Display module is operating the corresponding position of element with described for showing the mark for operating element；

Identification module includes the phonetic order of the mark that can operate element for identification；

Respond module executes respective operations to the current interface for responding the phonetic order.

11. a kind of interface operation equipment characterized by comprising

Memory, for storing executable instruction；Processor, when for executing the executable instruction stored in the memory, Realize the described in any item methods of claim 1 to 9.

12. a kind of storage medium, which is characterized in that being stored with executable instruction, when for causing processor to execute, realizing right It is required that 1 to 9 described in any item methods.