AU2020264367A1 - Contextual voice commands - Google Patents

Abstract

b ABSTRACT Among other things, techniques and systems are disclosed for implementing contextual voice commands. On a device, a data item in a first context is displayed. On the device, a physical input selecting the displayed data item in the first context is received. On the device, a voice input that relates the selected data item to an operation in a second context is received. The operation is performed on the selected data item in the second context.

Description

100323231b

CONTEXTUAL VOICE COMMANDS CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. Patent Application Ser. No. 12/479,477, filed June 5, 2009.

[0001a] This application is related to International Application Number PCT/US2010/037378 (International Publication Number WO 2010/141802 filed on 4 June 2010, the contents of which are incorporated herein by reference in their entirety.

TECHNICAL FIELD

[0002] This application relates to speech recognition.

BACKGROUND

[0003] Speech recognition can be used to convert spoken words to machine readable input, such as key presses, using the binary code for a string of character codes. Typical speech recognition applications include voice dialing, call routing, domotic appliance control, simple data entry and speech-to-text processing. In these applications, speech can be used as a part of a user interface to create pre-defined speech or voice commands.

SUMMARY

[0004] A combination of physical and voice inputs can be used to implement contextual voice commands that control device operations across different contexts (e.g., applications) on a device, such as a smart phone. In addition, by using contextual voice commands, a user can execute desired operations faster than by navigating through a set of nested menu items. Also, contextual voice commands can be used to teach the device to accurately predict the intent of the user from a single voice command. Further, contextual voice commands can be used to vary the manner in which the user provides the voice input based on the context of the device being used.

BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figures 1a and 1b show front and bottom views of a device for performing

contextual voice commands in an embodiment.

[0006] Figures 2a and 2b illustrate activation of contextual voice commands for a

data processing device in an embodiment.

[0007] Figures 3a and 3b show front views of a data processing device that illustrate

usage of contextual voice commands in an embodiment.

[0008] Figure 4 shows contextual voice commands implemented in another data

processing device in an embodiment.

[0009] Figure 5 is a block diagram illustrating a contextual voice command module

interfacing with other applications, an input unit and an output unit in an embodiment.

[0010] Figure 6 is a process flow diagram of an example process for implementing

contextual voice commands in an embodiment.

[0011] Figure 7 is a process flow diagram showing an example process for

implementing a confirmation model in an embodiment.

[0012] Figure 8a is a flow diagram showing an example process for implementing a

best assumption or learning model in an embodiment.

[0013] Figure 8b shows a data processing device accessing a knowledge base to

identify the best assumption or most likely choice of what the user intended in an

embodiment.

[0014] Figure 9 is a block diagram of example architecture of a data processing

device in an embodiment.

[0015] Like reference symbols and designations in the various drawings indicate like

elements.

Detailed Description

[0016] Figures 1a and 1b show front and bottom views of a device 102 for

performing contextual voice commands to operate the device. The data processing

device 102 can be any device, such as a mobile device, with the ability to receive voice

and physical inputs and perform operations in one or more applications or contexts

based on the received voice and physical inputs. The data processing device 102

includes communication technology (e.g., wireless technology) for sharing information

with other devices. The data processing device 102 can include a variety of integrated

user interface units or can be coupled to user interface units through one or more

communication ports or data links of the device. Some examples of the user interface

units include but are not limited to a voice input unit, such as a microphone 106. Some

examples of the user interface units include but are not limited to physical input units,

such a keyboard, a mouse, a track ball, a rotary dial or wheel, a touchpad, or a touch

screen. Some examples of the user interface units include but are not limited to motion

sensors, such as an accelerometer, magnetometer, or a gyroscope. Any of these user

interface units can be implemented as an external unit that communicates with the data

processing device 102 to provide user input using a wired or wireless communication

technology. Examples of wired communication technology include Universal Serial Bus

(USB) interface, FireWire interface, etc. Examples of wireless communication

technology include Bluetooth, Wi-Fi, WiMax, infrared, etc. Through these user interface

units, the data processing device 102 can receive physical or voice inputs from the user.

[0017] The data processing device 102 includes output units to present visual and

audio information to a user. Some examples of the output units include but are not

limited to a display unit 108 and a speaker 104. Figures la and 1b show the data

processing device 102 as a mobile device for illustrative purposes, but the data

processing device 102 can include other types of devices such as a mobile phone a

personal digital assistant, a portable computer, and a desktop computer.

[0018] The disclosed voice commands are contextual in that inputs, in addition to

voice, that indicate different levels or types of context for the commands are considered.

The contextual inputs considered can include the particular activity being performed at a

particular time, a particular portion of the activity selected, or voice commands in such

context. For example, contextual voice commands can be implemented in the context

of the activity that a user is performing on a data processing device, such as using a

particular application. For example, Figure 1a shows data items, such as icons

displayed on the display unit 108 of the data processing device 102. Each icon

represents a corresponding application available to be performed on the data

processing device 102. Some examples of the available icons and associated

applications include but are not limited to: a phone application icon 110a, an email

application icon 11Ob, a Web browser application icon 11Oc, a music player application

icon 11Od, a media player application icon 11Oe, a music download application icon

11Of, an image processing application icon 11Og, a geopositioning (e.g., GPS, Wi-Fi

based positioning, cellular triangulation) application icon 11Oh, a contacts application

icon 11Oi, a short message service (SMS) application icon 11Oj, a video game

application icon 110k, and a text processing application 1101 icon. Also, the available

applications can be presented to the user using audible data items, such as recorded sounds associated with the available applications. The recorded sounds can be user recognizable speech, such as the names of the applications or simple sounds, such as a beep, a tone, a chime, etc.

[0019] Thus, contextual voice commands can be more precise than conventional

voice commands that merely control the device as a whole. For example, different

context specific voice commands can be implemented based on user activity or

application in use. In addition, the scope of the context can extend to specific portions

of user activity or an application, such as particular areas of a user interface. In such

manner, the contextual voice commands can selectively provide voice navigation of a

user interface for a selected portion of the user activity or application.

[0020] Each of these applications can be in communication with a contextual voice

command controller or module that controls operation of the applications based on

received user input. The contextual voice command module can control the operation

of the applications by activating a contextual voice command mode on the data

processing device. To control the applications, the contextual voice command controller

or module can be implemented as a plug-in for the other applications. In other

implementations, the data processing device can include an Application Programming

Interface (API) which allows a contextual voice command module to access libraries

and utilities provided by an operating system or other native application. The contextual

voice command module can communicate with an input unit (e.g., voice and/or physical

input units) to receive user input used to control the rest of the applications.

[0021] An audio input unit can include user interface units that receive audio inputs,

such as a microphone 106. As described above, the audio input unit can be either a

built-in unit or an external unit. The user can speak into the microphone 106 to provide

voice inputs that can be translated into contextual voice commands using speech

recognition technology (e.g., technologies based on Hidden Markov Models, dynamic

time warping, etc.) Speech recognition software or technology can be used to

recognize voice commands in speech, such as commercially available products or open

source products such as Open Mind Speech, VoxForge, Julius, etc. among others.

[0022] A physical input unit (e.g., a touch screen, mouse, touchpad, etc.) can be

used to receive physical inputs, such as positional information with respect to a visual

screen displayed on an output unit (e.g., an output unit 108) of the data processing

device 102. Some physical input units, (e.g., a button) can be used to receive non

positional information, such as user selection or confirmation. For example, the user

can use a mouse to execute various point-and-click functions on the displayed screen.

Also, a user can use a touch screen to perform the same functions on the displayed

screen. Some user interface units, such as the keyboard, can provide textual inputs in

addition to physical inputs.

[0023] Figures 2a and 2b illustrate activation of a contextual voice command mode

on a data processing device 102. Figure 2a shows a front view of a data processing

device 102 before the contextual voice command mode is activated. The user can

activate the contextual voice command mode and place the data processing device 102

in a listening mode by interacting with a hardware button or virtual user interface

element, such as the contextual voice command icon 11Om. In addition, the contextual

voice command mode can be implemented to present to the user a complete voice control service that includes contextual and non-contextual options. Thus, the term

"contextual voice command mode" is applied in this specification to include but not

limited to contextual voice commands.

[0024] Figure 2b shows a front view of a data processing device 102 with the

contextual voice command mode activated. In the example shown in Figure 2b, the

user touches and selects the contextual voice command icon 11Om using his finger 103,

for example. User selection of the data item or element (e.g., contextual voice

command icon 110m) displayed on the display unit 108 can be communicated to the

user using a visual indication, such as a bolded border, a colored glow, a highlight,

different color, etc. Also, an audio indication can be used in addition to, or in place of,

the visual indication. For example, an audible, "contextual voice command active" or

"I'm listening" can be played through a speaker for the user to hear. In some

implementations, the audio indication presented to the user can include prompting tones

played instead of, or in addition to, the recognizable speech.

[0025] Once the contextual voice command mode is active, the data processing

device 102 can generate additional visual and/or audio indications to present a choice

of contextual voice commands to the user. For example, a portion of the display 108

can be used to present a visual indication (e.g., a status bar 105) to the user. In the

example shown in Figure 2b, the status bar 105 is used to display various text that

indicate to the user that the device is listening for contextual voice commands from the

user and that the user can select an element (e.g., an application) to receive a list of

available contextual voice command for the selected element. The device can provide

the choice of contextual voice commands and/or instructions to the user using audible

and visual indications.

[0026] Figures 3a and 3b show front views of a data processing device 102 to

visualize contextual voice commands in use. From the choice of data items or elements

(e.g., icons displayed), the user can select an icon to activate an associated application.

The contextual voice command mode is already active at this point using the process

illustrated in Figures 2a and 2b above. However, in some implementations, the

contextual voice command mode can be activated after selecting and initiating one of

the available applications. In some implementations, the data processing device can

exit the contextual voice command mode in response to a user request to switch from

one application to another application. Also, the user can change the focus of

interaction from a particular application to a home or main screen (e.g., by pressing a

user interface button associated with the main screen) to exit the contextual voice

command mode. In another example, the contextual voice command mode can be

active while the user's focus is on the main screen. In this example, the data

processing device can exit or de-activate the contextual voice command mode when the

user changes his focus from the main screen to an application of interest.

[0027] User selection of any of these icons loads and initiates the corresponding

application on the data processing device 102 to be used by the user. For example, the

user can physically touch the email icon 11Ob through a physical input unit, such as a

touch sensor, to select and initiate the email application. When the contextual voice

command mode is active, the user can also make a selection using an audio input unit,

such as a microphone, with or without a positional input.

[0028] Similar to the user selection of the contextual voice command icon, user

selection of one of the available icons can be communicated to the user using a visual

indication, such as a bolded border, a colored glow, a highlight, different color, etc.

Also, an audio indication can be used in addition to, or in place of, the visual indication.

For example, an audible, "Email Application" can be played through a speaker for the

user to hear.

[0029] The application corresponding to the user selected icon is displayed on a

display unit 108. For example, Figure 3a shows an email displayed on the display unit

108 in response to the user selection of the email icon 11Ob. When the contextual voice

command mode is active, a list of available functions for the selected application can be

displayed using another visual indication, such as a status bar 105. The status bar 105

can display the list of available functions using any number of visual techniques. For

example, the list can be displayed as a scrolling string of text. The list can be divided

into groups of N numbers, with N being the number of the functions that can be safely

displayed at once. Each group of functions can be displayed in a rotating fashion to

display all functions once at the end of each rotation. Also, while the status bar 105 is

shown to be displayed at the top of the display unit 108, the status bar 105 can be

located anywhere on the display unit 108. In some implementations, the available voice

commands or options can be displayed around a particular touch point or user-interface

point on the screen. For example, when the user physically touches an interface

element or a data item (e.g., an icon representing an application), the available voice

command options for that interface element can be shown near the interface element

and/or the user's finger. In addition to displaying available functions on the status bar

105, available functions can be provided as audio to be heard by the user. In the example shown in Figure 3a, the data processing device presents a visual indication that the device is listening for voice commands and the user can select a data item or an element displayed on the screen to obtain voice commands available for the user selected element. Also, the user can request and obtain help using contextual voice commands. For example, when the user requests help with operations available for a data item of interest, the device can present the available operations to the user. The help option is described further below.

[0030] On any user interface of any application, the contextual voice command mode

can be activated to enable voice commands tailored for specific activities associated

with a user selected element. In such manner, the voice commands are specifically

based on the context resulting from physical input associated with any portion or

element of the screen.

[0031] Figure 3b shows a front view of a data processing device to illustrate cross

contextual voice commands in use. The email displayed on the display unit 108

includes a test image 107 attached to the email. The user can select (e.g., using his

finger 103 through a touch screen) the test image to bring up a choice of contextual

voice commands associated with the selected test image. Each contextual voice

command is associated with a corresponding operation that can be performed on the

selected test image 107. The corresponding operations can include those for the

present context (e.g., email application) or for another context (e.g., image processing

application). In response to the user selection of the test image 107, the data

processing device 102 uses a status bar 105 to display the choice of available

contextual voice commands for the selected test image 107.

[0032] Figure 4 shows contextual voice commands used in another data processing

device. The contextual voice command mode can be implemented in various data

processing devices. For example, the data processing device capable of operating the

contextual voice command mode can include a personal computer 400. Similar to the

mobile device examples shown in Figures 1a-3b, the user can select an element (e.g.,

an email icon 110b) from the display unit 108 to initiate a corresponding application.

The example shown in Figure 4 shows the user selecting the email icon 11Ob using a

mouse pointer 402. In response to this physical input, the corresponding email

application is loaded and initiated. The initiated email application is displayed on an

active screen 404 that indicates the active application. The active screen 404 includes

an email 406 and an image 107 attached to the email. In response to the user selection

of the email 406, the personal computer 400 can display a choice of available

operations associated with the user selected element (e.g., the email 406) using a

status bar 105. For example, Figure 4 shows an optional status bar 105 that displays

the choice of operations that includes "compose, reply, forward, read, save, copy, paste,

search, file, edit, view, tools, action, help, etc." However, such visual or audible

indication of the list of available commands need not be presented to the user prior to

the user issuing a contextual voice command. For example, the voice command

module can process the verbal input received from the user and deduce the intent of the

user without presenting the list of commands to the user.

[0033] Figure 5 shows a block diagram illustrating a contextual voice command

module interfacing with other applications, an input unit and an output unit. A contextual

voice command controller or module 51Om can communicate with and control other

applications available on a data processing device. Consistent with the examples shown in Figures 1a, 1b, 2a, 2b, 3a, 3b and 4, the available applications can include but are not limited to: a phone application 510a, an email application 510b, a Web browser

51Oc, a music player application 51Od, a media player application 150e, a music

download application 510f, an image processing application 510g, a geopositioning

application 510h, a contacts application 510i, an SMS application 510j, a video game

application 510k, and a text processing application 5101. These applications can be

loaded and initiated in response to the user selection of the associated icon as shown in

Figures 2a, 2b, 3a, 3b and 4.

[0034] The contextual voice command module 51Om receives user input through an

input unit 520. The input unit can include an audio input unit 522, such as a

microphone, and a physical input unit 524, such as a touch screen. However, other

input units can be implemented as described above with respect to Figures 1a and 1b.

In response to the user input received, the contextual voice command module

understands the context of the input (e.g., selected application or a selected element in

the selected application) and associated voice command. Based on the received user

input, the contextual voice command module 51Om controls one or more of the

applications and their associated operations or functions.

[0035] Figure 6 is a flow diagram of an example process 600 for implementing

contextual voice commands. The device (e.g., data processing device 102) enters a

voice control or listening mode (610). To enter the listening mode, the user can provide

a predetermined user input through an input unit. For example, the user can press and

hold a physical or virtual button or simply speak a predetermined voice command that

triggers the contextual voice command option. For example, a predetermined trigger

phrase, such as "device listen" can be used to enter the device into the listening mode.

Also, the contextual voice command option can be active as a default, and the user can

deactivate the option by providing a predetermined triggering input. For example, the

device can exit the listening mode based on a voice input, such as "exit listening mode"

or a physical input, such as a touch or a button press. Also, the device can enter and

exit the listening mode using a combination of voice and physical inputs. For example,

the user can touch a button to initiate entry into or exit from the listening mode and a

follow-up voice command, such as "confirm exit or entry" to finalize the entry into or exit

from the listening mode.

[0036] When entering the listening mode, the device can optionally present a visual

and/or audible indication that the device is in the listening mode (620). An example

audible indication can be an audio recording played to the user that indicates: "I'm

listening and these are words you can say to act on the element." Another example of

an audible indication can be a simple sound alert, such as a beep, a ding, a chime or a

combination of these. The visual and/or audible indication alerts the user that the

device is in listening mode and that the user can provide the desired voice input that

represents a corresponding contextual voice command.

[0037] When receiving and processing a contextual voice command, a device can

operate on a specific element in the context, which may be an indicated element, a

currently active element, or an implied element, referred to herein as the element of

interest. This element of interest may be referred to within the vocabulary of the voice

command processing system by any appropriate term such as "this" or "item" or "current

item" or "selection" among many others. In addition, the voice command system may on receiving a keyword, ignore or use additional words as guidance. Thus "email this" or

"email selected image" or "email this photo" can be treated similarly because of the use

of the keyword "this" or "selected" for the current item of interest.

[0038] The device receives a user input that identifies the desired element of interest

that provides the context for the contextual voice commands (630). The user can

specify and select an element of interest by using any one of the physical input units,

such as a touch screen, a mouse, a keyboard, etc. Also, the user can specify and

select an element of interest by using an audio input unit to speak the predetermined

trigger phrase, such as "the text on third line" or "the image at the bottom" that provide

specific direction to what is displayed on the screen. The device need not enter the

listening mode before the user makes his selection of the element of interest. When not

already in the listening mode, the user can input the predetermined triggering input to

enter the device into the listening mode after making the selection of the element of

interest.

[0039] Additional visual and/or audible indication can be optionally provided to

include a list of normal and contextual voice commands available for the selected

element of interest (640). For example, a status bar (e.g., status bar 105 in Figures 2b,

3a, 3b and 4) can be used to provide a visual indication of the available contextual voice

commands. The list of available commands can include non-contextual commands,

such as "call person," "read email," "next meeting," etc., and contextual commands,

such as "send this to...," "bookmark this," "remind me of this in 2 hours," "delete this,"

"search Web for this," etc. The contextual voice commands are recognized to be in the

context of "this." The term, "this" or "item" or "current item" or "selection" or any similar

identifier for the element of interest can relate to the entire displayed screen, the selected active screen, or any element on any screen, or any physical input, such as hardware buttons. The element of interest can be a data item displayed on a screen if the user touched or named the element. The element of interest can be an application specific data item (e.g., "clear list"). Also, the element of interest can be a unit presented on the screen (e.g., "this mail message").

[0040] An element of interest can be identified by implication, that is without explicit

user indication of the element of interest. For example, if the device is currently

displaying an edit screen for a single contact in an address book, no further user input

may be needed for the system to process "delete current item" or "delete this" or any

similar phrase referring to the item of interest.

[0041] The device receives a voice command from the user through an input unit

(650). In response to the user selection of the element of interest and the received

voice command, the device acts on the received contextual voice command (660).

Acting on the received contextual voice command can include performing a set of

functions, such as identifying and calling an appropriate application that can act on the

user selected element of interest. Also, the device may need to provide additional

visual and/or audible indications, such as a list of the functions available for the selected

element of interest. The contextual voice command module (see Figure 2a, 2b, 3a and

3b) understands the context of the element of interest in the contextual voice command

based on the user input received through the input device and identifies the

corresponding functions available for the element or data item of interest.

[0042] The functions or operations available for the selected element or data item of

interest can include the functions associated with the active application or context. For

an email element, such as an image 107 attached to an email (see Figures 3a and 3b),

email application functions or operations are available. The image 382 can be

forwarded to another email address, for example. An example command can be "email

this image to email@email.com."

[0043] In addition, the visual and audible lists for the user selected element can

include functions or operations available for any other application or context that can act

on the selected element. This allows for context crossover. After selecting an element,

the user can provide a contextual voice command in context of the selected element,

such as "copy this" or "modify selection" or "delete item" to indicate that the contextual

voice command applies to the selected element.

[0044] For example, the image 107 can be modified using an image processing

application by speaking the predetermined trigger phrase, "modify this image." In

response to the contextual voice command, "modify this," the contextual voice

command module searches for a list of available applications that can modify the

selected image. When more than one application is available, the status bar 105 can

be used to display a list of the available applications. Also, the available applications

can be played as an audible list. In response to the visual or audible list of applications,

the user can speak the name of the desired application to activate that application to

modify the selected image.

[0045] In such manner, contextual voice commands can be used to act on the user

selected element across different contexts. In the example described above, the image

105 displayed in the email context can be used in the image processing context. In

another example, a business name in the contacts context can be used in the Web

browser context to search the Web for the business name. Using contextual voice

commands makes implementing context crossover easier and faster because the user

can use simple and intuitive voice commands (e.g., "Google* this business name")

instead of a long list of context sensitive menus, buttons, etc.

[0046] For example, using touch and/or voice input, the user can make a selection

on any screen, such as selecting a block of text in a word processing application

context. To perform context crossover for the selected block of text, the user can simply

speak the predetermined trigger phrase, such as "email this to email@email.com" or

"move this to Web browser" to act on the selected block of text in another context. In

contrast to such contextual voice command, a touch user interface requires the user to

copy and save the selected block of text, open the email application, compose a new

email, find or type the email address, paste the saved block of text into the email, and

press the send button.

[0047] Also, the user can issue a command such as "email this to John" without

leaving the present context or application. For example, in a traditional user interface,

the user would need to select the "email this photo" option, and then enter the email

address of the person to receive the email in some e-mail user interface, etc. The

contextual voice command mode allows the user to issue a command and have the

action performed in the background without the user leaving the context of the currently used application. Also, many more operations are at the user's disposal without requiring space on-screen to fit more buttons or add long menus for finding such options.

[0048] In some implementations, the user can provide an input to select a data item

of interest first. For example, the user can touch or interact with a data item displayed

or otherwise cause the data processing device to display a data item of interest on the

screen (e.g., select a photo in a photo album to display the selected photo in full

screen). Then, the user can instruct the data processing device to enter the contextual

voice command mode to interact with the selected data item.

[0049] In providing user interaction using audible indication, different user interaction

models, such as a confirmation model or a best assumption model can be used.

[0050] Figure 7 is a process flow diagram showing an example process 700 for

implementing a confirmation model. The confirmation model provides audible

confirmations in the form of a question in response to each voice input received from

the user. The device receives user input indicating the desired element of interest and

a contextual voice command related to the context of the element of interest (710). In

response to the received contextual voice command, the contextual voice command

module identifies the available choices (720) and requests confirmation from the user

using audible and/or visual output (730). The device receives another user input in

response to the confirmation request (740). The contextual voice command module

processes the other received user input to determine whether additional confirmation is

needed (e.g., determining if the contextual voice command is finalized) (750). When

additional confirmation is needed, the contextual voice command module identifies the

available choices (720) and requests confirmation from the user using audible and/or visual output (730). When no additional confirmation is needed (e.g., the contextual voice command is finalized), the device acts on the received contextual voice command

(760).

[0051] For example, when the user speaks the predetermined trigger phrase, such

as "Call John," the device provides the confirmation response, "Which John?" and then

provides an audible and/or visual list of all available persons with the name John. The

user can select one of the names by speaking the predetermined trigger phrase, such

as "This one" or "This John" when using a positional input device to touch the visual

indication of the desired John in the list. When strictly relying on voice input, the user

can speak the full name of the desired John. In response to the user selection, the

device provides the next appropriate confirmation, such as "Ok, which number?" and

provides audible and/or visual list of all phone numbers available for the selected John.

The user can select from the available phone numbers.

[0052] Figure 8a is a flow diagram showing an example process 800 for

implementing a best assumption or learning model. The best assumption or learning

model uses usage or preference history of the user to make the best assumption.

Examples of learning models can include but are not limited to: machine learning

models, such as support vector machines (SVM), an inductive inference model, concept

learning, decision tree learning, Bayesian learning, and others. A machine learning

model can be used to develop a process to teach the data processing device to improve

its performance based on accumulated data received through an input unit or stored in

a database. Such machine learning models can be used to automatically produce a

desired result based on rules and patterns designed from the accumulated data. The

device receives user input indicating the desired element of interest and contextual voice command (810). The contextual voice command module checks a knowledge base to identify the best assumption or most likely choice of what the user intended

(820).

[0053] Figure 8b shows a data processing device accessing a knowledge base to

identify the best assumption or most likely choice of what the user intended. In the

example shown in Figure 8b, the contextual voice command mode 510m is active. The

user can touch the corresponding contextual voice command icon 11Om using his finger

103 through a touch screen, for example. Also, the user selects one of the applications

to load and initiate the selected application. In the example shown in Figure 8b, the

phone application 510a is selected in response to the user touching the corresponding

phone icon 110a. In the context of the physical input, the user can speak a voice input

802, such as "call John." The physical and voice inputs are received through the

respective input units 524 and 522, and the contextual voice command module 510m

accesses a knowledge base 806 to identify the best assumption. The knowledge base

806 can include a local knowledge base 812 stored within the data processing device

102 and an external knowledge base 814 located external to the data processing device

102. For example, the data processing device 102 can access the external knowledge

base 814 stored on a remote server through a network connectivity 804, such as the

Internet.

[0054] The knowledge base 806 can include a usage history 816 related to the

requested contextual voice command. For example, for the received contextual voice

command, "call John," the knowledge base can include a list of persons named John

and the number of times each John was called. In the example usage history 816

shown in Figure 8b, John Doe was called the most times (e.g., 31 times). Thus, the contextual voice command module 510m identifies John Doe as the best assumption of the user's intent. Based on this identification, the contextual voice command module

510m outputs through an output unit 540 (e.g., the status bar 105 of the display unit

108) a message asking the user whether John Doe should be called. In addition, the

contextual voice command module can present an audible indication of the same

message. In response to this visual and/or audible indication, the user can confirm or

correct the assumption.

[0055] In some implementations, the knowledge base can include other usage

information, and the best assumption can be predicted based on the other usage

information. For example, the usage information can include an identity of the John

called most recently, an identity of the John called at the same time on previous days,

an identity of the John tagged as a favorite, an identity of the John that lives at a certain

location, etc.

[0056] Referring back to Figure 8a, the contextual voice command mode conveys to

the user the identified best assumption using audible and/or visual output (830). This

allows the user to make any correction to the identified assumption. The device waits to

receive user input either confirming or correcting the identified best assumption (840).

However, a user prompt is not needed. A wait time can be preset or modified to provide

an interval of time for the user to cancel or correct the assumed command using

utterances, such as "no! John Smith!" or using a physical press of a hardware button to

cancel the assumed command. In the absence of a user prompt within the interval time,

the device can perform the operation indicated by the best assumption. The contextual

voice command module determines whether the user input confirms or corrects the

identified best assumption (850). When detecting that the user input is one that confirms the identified best assumption, the device acts on the identified best assumption (860). As described above, a valid confirmation from the user can include a non-response from the user during the time interval.

[0057] When detecting that the user input is one that corrects the identified best

assumption, the contextual voice command controller determines whether a different

assumption or choice should be made (870). When the contextual voice command

controller detects that the user has corrected the initially identified best assumption, the

contextual voice command controller makes a different assumption based on the user

input (880). However, when the contextual voice command controller detects that the

user has merely indicated that the identified best assumption is incorrect, the contextual

voice command returns to the knowledge base to make the next best assumption (890).

The device requests confirmation of the identified next best assumption (830) and the

process 800 continues until the user intent is correctly identified.

[0058] For example, when the user speaks the predetermined trigger phrase, such

as "Call John," the device checks a knowledge base and identifies that the user always

(or most of the time, etc.) calls John Doe on his mobile phone. Thus, the device dials

the mobile phone number of John Doe. However, the device provides an opportunity

for the user to change the command. The device can provide a visual or audible

confirmation, such as "Would you like to Call John Doe's mobile." To this, the user can

respond by saying "yes" or do nothing for a predetermine duration after receiving a

confirmation to dial the number. To change the voice command, the user can speak the

predetermined trigger phrase, "No, I meant John Smith." To this response, the device

checks the user history and identifies that the user always (or most of the time, etc.)

calls John Smith at home. So the device dials that number. By this process, the device builds a knowledge base based on user history so as to reduce the number of ask-and confirms. Based on this knowledge base, the device can make the best assumption. If the assumption is wrong, the user is asked to correct the assumption.

[0059] To build the knowledge base, the user can teach the device to remember

certain commands. For example, with the contextual voice command option active, the

user can rotate an object by 45 degrees in an image processing application. Initially,

the act of rotating the image can be performed using a positional input device. Then,

the user can select this rotated object and identify it by saying "an image in 45 degree

angle" to teach the device. Next time the user wants to rotate the object by 45 degrees,

the user can simply say "an object rotated in 45 degrees."

[0060] In addition, macros can be implemented to teach the data processing device.

For example, macros can be generated by combining multiple operations into one

contextual voice command. The following describes a process for generating an

example macro:

> Data processing device enters contextual voice command mode.

> Contextual voice command module receives a contextual voice command that

indicates macro generation, such as "learn this."

> Contextual voice command module receives user selection of a data item of interest

and contextual voice commands for performing multiple operations, such as "use in a

note," "remind me of note in 2 hours," "send note as email to contact group," etc.

> Contextual voice command module receives a contextual voice command that

indicates end of macro generation, such as "done learning."

> Contextual voice command module prompts the user to name the generated macro.

> Contextual voice command module receives user input indicating a name for the

generated macro, such as "Perform Operations."

> Contextual voice command module confirms association of the received name with

the generated macro.

> End macro generation.

[0061] Thus, a macro can be used to associate a sequence of operations with a

descriptive name (e.g., "Perform Operations") and teach the data processing device to

perform the sequence of operations in response to user utterance of the name. The

next time the user desires to perform the same sequence of operations, the user can

instruct the data processing device to "Perform Operations" on a selected data item of

interest. This learned contextual voice command can be processed by the data

processing device to perform the entire sequence of operations on the selected data

item of interest.

[0062] The sequence learning using programmable macros can be implemented as

a separate system that can operate without voice commands. When implemented as a

separate system external to the contextual voice command module, the contextual

voice command module can be implemented as a layer that controls the separate

system.

[0063] To expedite the response time, a filter can be implemented to automatically

filter out commands that do not make sense or are spoken out of context. For example,

in response to the voice command, "Call," the filter can eliminate from the visual and

audible indications, the email addresses, home address, etc. In other examples, the

contextual voice command controller can be configured to limit a mapping command to

street addresses and limit a bookmarking command to URL addresses.

[0064] In addition to providing a list of available choices, the visual and audible

indications can be used to obtain help at any time. The help available to the user

includes helpful suggestions to assist the user in a particular situation. For example, the

user may be stuck in a particular application and does not know how to get out of the

situation. The user can request help by speaking the predetermined trigger phrase,

such as "Help me" or "Help, I'm stuck," for example, to obtain the necessary instruction.

The contextual voice controller can output a visual and/or audible message to help the

user, such as "You are trying to e-mail this image." Also, the help available to the user

includes possible options for acting on the selected element. For example, to find out

what the user can do with the selected element, the user can simply ask by speaking

the trigger term, such as "What can I do with this?"

[0065] During the listening mode, the visual and/or audible indication (e.g., prompts)

provided by the device can change in response to each voice input that is received from

the user. For example, when the device receives the voice input, "Call John," and the

device detects that there are multiple Johns in the phone book, the device plays an

audio prompt asking the user to identify: "which John?" Also, the visual indication can

change to display the contact information for everyone named John listed in the user's

phone book. The contact information displayed can include the name(s) of

corresponding phone number(s) (including mobile, home and work phones) for each

John. If the user speaks a voice command to select one of the available Johns, such as

"John Smith" and John Smith is associated with multiple numbers in the user's phone

book, the device can play an audio prompt asking the user to identify: "which number?"

and display the names of the multiple numbers (e.g., mobile, work, home) on the

screen.

[0066] In some implementations, contextual voice commands can be used to receive

information or feedback from the device about the selected element of interest. For

example, in a media album application, the user can say "Find more about fourth track"

to obtain information about that track. Because the user is already in the media album

context, the contextual voice command controller knows the context of the command.

In non-contextual voice commands, the user would have to specifically identify the

name of the artist and the song by saying "find more about the artist X and his song Y,"

for example. Other types of feedback can be received from the device. For example,

the user can speak a predetermined trigger phrase, such as "How long is this" to

receive an answer from the device. Also, the user can ask the device to provide other

audible feedback by speaking predetermined trigger phrases, such as "read this," "write

this," "display this," etc.

[0067] As described above, the context of the contextual voice command can be

associated with the particular application in which the user selected element resides. In

some implementations, the context of the voice commands is also associated with the

environment in which the device is used. Examples of such environmental context

includes automobile context, navigation context, meeting context, etc.

[0068] In the navigation context, the orientation or position of the phone can be used

to customize the contextual voice commands for providing direction to the user. The

direction can be provided using GPS and a compass. For example, the user can speak

the predetermined trigger phrase, such as "which way should I walk?" to receive

appropriate direction from the device. The contextual voice command controller can

use the GPS and compass based position of the device to provide proper audible

direction to the user, such as "turn left," "walk straight," etc. In addition, the user can use the contextual voice commands to obtain directional information from the device.

For example, the user can speak the predetermined trigger term, such as "what is this

device pointing at" or "what am I looking at" or "What is to the left of me?" or "What is to

the right of me?" or "what is ahead of me" or "what is ahead" or "what is behind me" or

"which way to destination" or "how far away is destination," etc. In these examples, the

orientation of the device can be a contextual element. These types of triggering

phrases can be predetermined to cause the device to trigger the appropriate navigation

related applications.

[0069] For certain situations, such as the automobile context, in which the user is

unable to provide input through a positional input device, an audible feedback

mechanism can be customized for voice intensive interaction. For example, the

contextual voice command controller can simplify the audible response sent from the

device based on the context, such as driving context. For example, instead of the

contextual voice command controller providing audible outputs, such as "We have

multiple names, which one do you want?" to provide a list of choices, the audible

response from the device can be narrowed to simple Yes-No scenarios. For example,

the device can ask, "Do you want this? Yes or No." Such simple Yes-No response is

easier for the user to use when driving, for example.

[0070] The device can be made aware of such context using various mechanisms.

For example, the device can detect that the user is using a Bluetooth device to provide

the voice input. Also, the device can detect that the device is plugged into an

input/output jack of an automobile stereo. Also, the user can press, touch, or press

and-hold a physical or virtual button to indicate such context. In addition, the device can

detect that the device is moving using GPS and/or a compass. For example, when located inside a moving vehicle, the location of the device can change with respect to time. This movement or change in the location of the device can be detected and used to trigger the different modes of feedback.

[0071] The voice of the audible output generated by the contextual voice command

controller can be changed by installing different voices. Also, different voices can be

used for different context. Moreover, the techniques, apparatus and systems described

in this specification can be extended to any spoken language, including specialized

command languages. For example, the voice commands (contextual and non

contextual) can be implemented to allow the device to recognize English, French,

German, Korean, Chinese (Mandarin and Cantonese), Japanese, Italian, Spanish,

Farsi, etc. Also, any of these spoken languages can be used to generate specialized

command languages, such as a set of commands designed to elicit efficient response

from the device.

[0072] Contextual voice commands are not limited to current functionalities, but

rather can be adapted to control future functionalities. For example, any additional

functionalities of the device can be controlled under a contextual voice command

module. This is possible by implementing a contextual voice command module as a

plug-in to the new functionalities. Also, third party developers can be provided with an

option (e.g., using a software development kit (SDK)) to indicate to the system that

additional non-contextual and contextual commands are available.

[0073] Figure 9 is a block diagram of example architecture 900 of a data processing

device. The data processing devices 102 and 400 can include a memory interface 902,

one or more data processors, image processors and/or central processing units 904,

and a peripherals interface 906. The memory interface 902, the one or more processors 904 and/or the peripherals interface 906 can be separate components or can be integrated in one or more integrated circuits. Various components in the data processing devices 102 and 400 can be coupled together by one or more communication buses or signal lines.

[0074] Sensors, devices, and subsystems can be coupled to the peripherals

interface 906 to facilitate multiple functionalities. For example, a motion sensor 910, a

light sensor 912, and a proximity sensor 914 can be coupled to the peripherals interface

906 to facilitate the orientation, lighting, and proximity functions. A location processor

915 (e.g., GPS receiver) can be connected to the peripherals interface 906 to provide

geopositioning. A magnetic compass integrated circuit 916 can also be connected to

the peripherals interface 906 to provide orientation (e.g., to determine the direction of

due North).

[0075] A camera subsystem 920 and an optical sensor 922, e.g., a charged coupled

device (CCD) or a complementary metal-oxide semiconductor (CMOS) optical sensor,

can be utilized to facilitate camera functions, such as recording photographs and video

clips.

[0076] Communication functions can be facilitated through one or more wireless

communication subsystems 924, which can include radio frequency receivers and

transmitters and/or optical (e.g., infrared) receivers and transmitters. The specific

design and implementation of the communication subsystem 924 can depend on the

communication network(s) over which the data processing devices 102 and 400 is

intended to operate. For example, data processing devices 102 and 400 may include

communication subsystems 924 designed to operate over a GSM network, a GPRS

network, an EDGE network, a Wi-Fi or WiMax network, and a BluetoothTM network. In particular, the wireless communication subsystems 924 may include hosting protocols such that the data processing devices 102 and 400 may be configured as a base station for other wireless devices.

[0077] An audio subsystem 926 can be coupled to a speaker 928 and a microphone

930 to facilitate voice-enabled functions, such as voice recognition, voice replication,

digital recording, and telephony functions.

[0078] The I/O subsystem 940 can include a touch screen controller 942 and/or

other input controller(s) 944 as described with respect to Figures 1a and 1b. For

example, the I/O subsystem 940 can include a microphone (internal and/or external), a

speaker and a voice command recognition engine. The I/O subsystem 940 can receive

voice commands and present audio outputs over full duplex communication. For

example, transport technologies other than regular cellular voice communications, such

as voice over IP, can be implemented.

[0079] Also, voice commands can be processed using a two-pass process. The on

device voice command module can process the received voice commands to perform a

generalized recognition. Audio data of the received voice commands can be sent to a

server to provide a more detailed and accurate processing. The server may be better

equipped (e.g., using a faster and more powerful processor) to perform voice command

recognition than a mobile device. To reduce bandwidth requirements and latency

issues, the audio data may not be sent to the server in its entirety. For example, the on

device voice command module can process the voice commands to identify strings of

numbers, but may not be able to identify the exact voice commands. Thus, the on

device voice command module may determine that the voice commands or utterance

contain "some numbers." A larger surrounding segment of the audio data can be sent to the server, and the server can asynchronously return a much better idea of what was actually said in the voice commands. By using the server in such manner, the benefits of server processing can be obtained while reducing or minimizing the costs involved with the server processing.

[0080] The touch-screen controller 942 can be coupled to a touch screen 946. The

touch screen 946 and touch screen controller 942 can, for example, detect contact and

movement or break thereof using any of a plurality of touch sensitivity technologies,

including but not limited to capacitive, resistive, infrared, and surface acoustic wave

technologies, as well as other proximity sensor arrays or other elements for determining

one or more points of contact with the touch screen 946.

[0081] The other input controller(s) 944 can be coupled to other input/control devices

948, such as one or more buttons, rocker switches, thumb-wheel, infrared port, USB

port, and/or a pointer device such as a stylus. The one or more buttons (not shown)

can include an up/down button for volume control of the speaker 928 and/or the

microphone 930.

[0082] In one implementation, a pressing of the button for a first duration may

disengage a lock of the touch screen 946; and a pressing of the button for a second

duration that is longer than the first duration may turn power to the data processing

devices 102 and 400 on or off. The user may be able to customize a functionality of

one or more of the buttons. The touch screen 946 can, for example, also be used to

implement virtual or soft buttons and/or a keyboard.

[0083] In some implementations, the data processing devices 102 and 400 can

present recorded audio and/or video files, such as MP3, AAC, and MPEG files. In some

implementations, the data processing devices 102 and 400 can include the functionality

of an MP3 player, such as an iPod TouchTM.

[0084] The memory interface 902 can be coupled to memory 950. The memory 950

can include high-speed random access memory and/or non-volatile memory, such as

one or more magnetic disk storage devices, one or more optical storage devices, and/or

flash memory (e.g., NAND, NOR). The memory 950 can store an operating system

952, such as Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, or an embedded

operating system such as VxWorks. The operating system 952 may include instructions

for handling basic system services and for performing hardware dependent tasks. In

some implementations, the operating system 952 can be a kernel (e.g., UNIX kernel).

[0085] The memory 950 may also store communication instructions 954 to facilitate

communicating with one or more additional devices, one or more computers and/or one

or more servers. The memory 950 may include graphical user interface instructions 956

to facilitate graphic user interface processing; sensor processing instructions 958 to

facilitate sensor-related processing and functions; phone instructions 960 to facilitate

phone-related processes and functions; electronic messaging instructions 962 to

facilitate electronic-messaging related processes and functions; web browsing

instructions 964 to facilitate web browsing-related processes and functions; media

processing instructions 966 to facilitate media processing-related processes and

functions; GPS/Navigation instructions 968 to facilitate GPS and navigation-related

processes and instructions; camera instructions 970 to facilitate camera-related

processes and functions; and voice command instructions 972 to facilitate operation of the data processing device 102 using contextual voice commands, as described in reference to Figures 1a, 1b, 2a, 2b, 3a and 3b. In some implementations, the GUI instructions 956 and/or the media processing instructions 966 implement the features and operations described in reference to Figures 1-8.

[0086] The memory 950 may also store other software instructions (not shown), such

as web video instructions to facilitate web video-related processes and functions; and/or

web shopping instructions to facilitate web shopping-related processes and functions.

In some implementations, the media processing instructions 966 are divided into audio

processing instructions and video processing instructions to facilitate audio processing

related processes and functions and video processing-related processes and functions,

respectively. An activation record and International Mobile Equipment Identity (IMEI)

974 or similar hardware identifier can also be stored in memory 950.

[0087] Each of the above identified instructions and applications can correspond to a

set of instructions for performing one or more functions described above. These

instructions need not be implemented as separate software programs, procedures, or

modules. The memory 950 can include additional instructions or fewer instructions.

Furthermore, various functions of the data processing devices 102 and 400 may be

implemented in hardware and/or in software, including in one or more signal processing

and/or application specific integrated circuits.

[0088] The techniques for implementing the contextual voice commands as

described in Figures 1-9 may be implemented using one or more computer programs

comprising computer executable code stored on a tangible computer readable medium

and executing on the data processing device 102. The computer readable medium may

include a hard disk drive, a flash memory device, a random access memory device such as DRAM and SDRAM, removable storage medium such as CD-ROM and DVD-ROM, a tape, a floppy disk, a Compact Flash memory card, a secure digital (SD) memory card, or some other storage device. In some implementations, the computer executable code may include multiple portions or modules, with each portion designed to perform a specific function described in connection with Figures 6-8. In some implementations, the techniques may be implemented using hardware such as a microprocessor, a microcontroller, an embedded microcontroller with internal memory, or an erasable, programmable read only memory (EPROM) encoding computer executable instructions for performing the techniques described in connection with Figures 6-8. In other implementations, the techniques may be implemented using a combination of software and hardware.

[0089] Processors suitable for the execution of a computer program include, by way

of example, both general and special purpose microprocessors, and any one or more

processors of any kind of digital computer, including graphics processors, such as a

GPU. Generally, the processor will receive instructions and data from a read only

memory or a random access memory or both. The elements of a computer are a

processor for executing instructions and one or more memory devices for storing

instructions and data. Generally, a computer will also include, or be operatively coupled

to receive data from or transfer data to, or both, one or more mass storage devices for

storing data, e.g., magnetic, magneto optical disks, or optical disks. Information carriers

suitable for embodying computer program instructions and data include all forms of non

volatile memory, including by way of example semiconductor memory devices, e.g.,

EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0090] To provide for interaction with a user, the systems apparatus and techniques

described here can be implemented on a data processing device having a display

device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for

displaying information to the user and a positional input device, such as a keyboard and

a pointing device (e.g., a mouse or a trackball) by which the user can provide input to

the computer. Other kinds of devices can be used to provide for interaction with a user

as well; for example, feedback provided to the user can be any form of sensory

feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from

the user can be received in any form, including acoustic, speech, or tactile input.

[0091] While this specification contains many specifics, these should not be

construed as limitations on the scope of any invention or of what may be claimed, but

rather as descriptions of features that may be specific to particular embodiments of

particular inventions. Certain features that are described in this specification in the

context of separate embodiments can also be implemented in combination in a single

embodiment. Conversely, various features that are described in the context of a single

embodiment can also be implemented in multiple embodiments separately or in any

suitable subcombination. Moreover, although features may be described above as

acting in certain combinations and even initially claimed as such, one or more features

from a claimed combination can in some cases be excised from the combination, and

the claimed combination may be directed to a subcombination or variation of a

subcombination.

[0092] Similarly, while operations are depicted in the drawings in a particular order,

this should not be understood as requiring that such operations be performed in the

particular order shown or in sequential order, or that all illustrated operations be

performed, to achieve desirable results. In certain circumstances, multitasking and

parallel processing may be advantageous. Moreover, the separation of various system

components in the embodiments described above should not be understood as

requiring such separation in all embodiments, and it should be understood that the

described program components and systems can generally be integrated together in a

single software product or packaged into multiple software products.

[0093] Only a few implementations and examples are described and other

implementations, enhancements and variations can be made based on what is

described and illustrated in this application.

Claims (13)

100323231b CLAIMS What is claimed is:

1. A method, comprising: at an electronic device with one or more processors and memory: receiving a user input; receiving a voice input; determining, from the voice input, a request for help related to an operation associated with the user input, wherein the operation can be performed using the electronic device; and in response to receiving the user input, providing an output relating to the operation.

2. The method of claim 1, further comprising: determining an operation a user is currently performing, wherein the operation associated with the user input is the operation the user is currently performing.

3. The method of any of claims 1-2, wherein the user input is a selection of a data item.

4. The method of claim 3, wherein the operation associated with the user input is an operation that can be performed on the data item.

5. The method of any of claims 1-4, wherein the user input is in an application context.

6. The method of claim 5, wherein the at least one operation associated with the user input is an operation that can be performed in the application context.

7. The method of any of claims 1-6, wherein the output includes instructions for performing the operation relating to the user input using the electronic device.

8. The method of any of claims 1-7, wherein the operation associated with the user input is a first operation associated with the user input, and wherein the output

100323231b

includes a list including at least the first operation associated with the user input and a second operation associated with the user input.

9. The method of any of claims 1-8, wherein the output includes an audio output.

10. The method of any of claims 1-9, wherein the output includes a displayed output.

11. The method of any of claims 1-10, wherein the voice input includes a predetermined trigger phrase.

12. An electronic device, comprising: one or more processors; a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the methods of any one of claims 1-11.

13. A computer-readable storage medium storing instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the methods of any one of claims 1-11.