KR101617665B1 - Automatically adapting user interfaces for hands-free interaction - Google Patents

Abstract

User interfaces for systems such as virtual assistants are automatically adapted to hands-free use. The hands-free context is detected via automatic or manual means, and the system coordinates the various steps of the complex interactive system to modify the user experience to reflect particular limitations of such context. Thus, the system of the present invention allows a single implementation of a composite system, such as a virtual assistant, to dynamically provide user interface elements and to enable hands-free use without compromising the user experience of the same system for hands- Change interface behavior.

Description

[0001] AUTOMATICALLY ADAPTING USER INTERFACES FOR HANDS-FREE INTERACTION FOR HANDS-FREE INTERACTION [0002]

The present invention relates to multi-mode user interfaces, and more particularly to user interfaces including both voice-based and visual modalities.

Many existing operating systems and devices use voice input as a form that allows the user to control the operation. One example is voice command systems, which map specific verbal commands to actions to initiate dialing of a phone number, for example by saying a person's name. Other examples are Interactive Voice Response (IVR) systems, which allow people to access static information via telephones, such as automated telephone service desks.

Many voice commands and IVR systems are relatively narrow in scope and can only handle a predefined set of voice commands. In addition, their output often draws from a fixed set of responses.

An intelligent automated assistant, also referred to herein as a virtual assistant, can provide an improved interface between a human and a computer, including the processing of natural language input. Such an assistant, which may be implemented as described in related U. S. Patent Application No. 12 / 987,982, entitled "Intelligent Automation Assistant" filed January 10, 2011, the entire disclosure of which is incorporated herein by reference Allowing users to interact with devices or systems using natural language, in voice and / or text forms. Such an assistant interprets the user inputs, manipulates the user's intent with the tasks and parameters for these tasks, executes the services to support these tasks, and generates user-understandable output.

Virtual assistants can use generic speech and natural language that understands the technology to recognize a larger range of inputs, enabling the creation of dialogs with users. Some virtual assistants can generate output in a combination of modes, including verbal responses and written text, and can also provide a graphical user interface (GUI) that allows direct manipulation of on-screen elements. However, the user may not always be in a situation where he can use such visual output or direct manipulation interfaces. For example, the user may be driving, operating the machine, have a visual impairment, or simply be uncomfortable or unfamiliar with the visual interface.

Any situation in which a user has limited ability to read or interact with a device through a contact (keyboard, mouse, touch screen, pointing device, etc.) or has no such capability is referred to herein as a " hands-free context ". For example, as noted above, in situations where the user tries to operate the device during operation, the user can listen to the audible output and respond using their voice, but due to safety reasons, Or tapping menus or not entering text.

Hands-free contexts provide special challenges to developers of complex systems such as virtual assistants. Users require full access to the features of the devices whether they are in a hands-free context or not. However, failure to account for any special limitations inherent in hands-free operation can lead to situations that limit both the utility and usefulness of the device or system and may even cause the user to be distracted from the main task, The safety can be impaired.

According to various embodiments of the present invention, a user interface for a system, such as a virtual assistant, is automatically adapted to hands-free use. The hands-free context is detected via automatic or manual means, and the system coordinates the various steps of the complex interactive system to modify the user experience to reflect particular limitations of such context. Thus, the system of the present invention allows a single implementation of a virtual assistant or other complex system to dynamically provide user interface elements and enable hands-free operation without damaging the user experience of the same system for hands- Allows the user interface behavior to change to allow use.

For example, in various embodiments, the system of the present invention provides mechanisms for manipulating the operation of a virtual assistant to provide output in a manner that allows users to complete tasks of users without having to read the details on the screen. . Moreover, in various embodiments, the virtual assistant can provide mechanisms for receiving voice input as an alternative to performing read, tapping, clicking, typing, or other functions that are often accomplished using a graphical user interface.

In various embodiments, the system of the present invention allows for special requirements and limitations associated with the hands-free context, while providing the same (or approximate) basic functionality as the functionality of a conventional graphical user interface. More generally, the system of the present invention facilitates operation in a hands-free context, while allowing core functions to remain substantially the same. In some embodiments, systems configured in accordance with the techniques of the present invention allow users to freely select between a hands-free mode and a conventional ("hands-on") mode in some instances within a single session. For example, the same interface can be made adaptable to both the office environment and the moving vehicle, where the system makes the necessary changes to the user interface behavior dynamically as the environment changes.

According to various embodiments of the present invention, any of a number of mechanisms may be implemented to adapt the operation of the virtual assistant to a hands-free context. In various embodiments, the virtual assistant is described in U.S. Patent Application No. 12 / 987,982, entitled " Intelligent Automation Assistant, " filed January 10, 2011, the entire disclosure of which is incorporated herein by reference It is the same intelligent automation assistant. Such an assistant interacts with the user in an integrated, conversational manner using natural language dialogs, and operates external services when appropriate to acquire information or perform various operations.

According to various embodiments of the present invention, the virtual assistant is configured to detect a hands-free context in performing various different types of operations, functions, and / or features and to adjust the operation of the virtual assistant accordingly, and Designed and / or operable to combine multiple features, operations, and applications of the electronic device in which the virtual assistant is installed. In some embodiments, the virtual assistant of the present invention receives input (s), provides an output, participates in a dialog with a user, and / or performs (or initiates) operations based on the identified intent, It is possible to detect the hands-free context and adjust the operation of the virtual assistant accordingly.

Operations may be performed, for example, by activating and / or interfacing with any applications or services that may be available on the electronic device, as well as services available over an electronic network such as the Internet. In various embodiments, such activation of external services may be performed through application programming interfaces (APIs) or by any other suitable mechanism (s). In this manner, the virtual assistant implemented in accordance with various embodiments of the present invention provides a hands-free environment for many different applications and functions of the electronic device, and for services that may be available over the Internet can do. The use of such virtual assistants, as described in the aforementioned related applications, allows a user to determine what functions are available on the device and on web-accessible services, how the user interfaces with those services to get what he wants, And it can lessen the burden of learning how to interpret the received output from such services; Rather, the assistant of the present invention can act as a mediator between the user and such a variety of services.

In addition, in various embodiments, the virtual assistant of the present invention provides a dialog interface that allows the user to look more intuitively and less painfully than conventional graphical user interfaces. The user may participate in an interaction dialog with the assistant using any of a number of available input and output mechanisms, depending in part on whether the hands-free or hands-on context is active. Examples of such input and output mechanisms include, without limitation, speech, graphical user interfaces (buttons and links), text entries, and the like. The system may be implemented using any of a number of different platforms, such as device APIs, web, email, etc., or any combination thereof. Requests for additional input may be provided to the user in the context of a conversation provided in an audible and / or visual manner. Short-term and long-term memory can be involved so that user input can be interpreted as an appropriate context when considering history and profile information for users as well as previous events and communications within a given session.

In various embodiments, the virtual assistant of the present invention may control various features and operations of the electronic device. For example, a virtual assistant may be a service that interfaces with functions and applications on a device via APIs or by other means to perform functions and operations that may otherwise be initiated using a conventional user interface on the device Can be called. Such functions and operations may include, for example, setting an alarm, placing a call, sending a text message or an email message, adding calendar events, and the like. Such functions and operations may be performed as add-on functions in the context of dialog dialogs between the user and the assistant. Such functions and operations may be specified by the user in the context of such a dialog, or they may be performed automatically based on the context of the dialog. Those skilled in the art will recognize that the assistant can thereby be used as a mechanism for initiating and controlling various operations on an electronic device. By collecting contextual evidence that contributes to the inferences about the user's current situation, and thus by regulating the operation of the user interface, the system of the present invention allows the hands-free operation of the virtual assistant to provide such a mechanism for controlling the device And can provide mechanisms to enable the implementation.

According to some embodiments, the method includes detecting at a processor whether a hands-free context is active. The method also includes, at the output device, prompting the user for input. The method also includes receiving, at the input device, user input. The method also includes, at the processor, interpreting the received user input to derive a representation of the user's intent; Identifying at least one parameter for at least one task and a task based at least in part on an expression of the derived user intent; Executing at least one task using at least one parameter to derive a result; And generating a dialog response based on the derived result. The method also includes, at the output device, outputting the generated dialog response. Responsive to detecting that the device is in a hands-free context, prompting the user for input, receiving user input, interpreting the received user input, determining at least one Identifying a parameter, and generating a dialog response are performed in a manner that complies with the constraints associated with the hands-free context.

According to some embodiments, an electronic device includes one or more processors, a memory, and one or more programs; One or more programs are stored in memory and configured to be executed by one or more processors, and one or more programs include instructions for performing operations of any of the methods described above. According to some embodiments, a computer-readable storage medium has stored therein instructions that, when executed by an electronic device, cause the device to perform operations of any of the methods described above. According to some embodiments, the electronic device includes means for performing operations of any of the methods described above. According to some embodiments, an information processing apparatus for use in an electronic device includes means for performing operations of any of the methods described above.

According to some embodiments, the electronic device includes a processing unit configured to detect whether a hands-free context is active. The electronic device also includes an output unit coupled to the processing unit and configured to prompt the user for input, and an input unit coupled to the processing unit and configured to receive the user input. The processing unit interprets the received user input to derive a representation of the user's intent and identifies at least one parameter for the at least one task and the task based at least in part on the derived representation of the user's intent, To execute at least one task using at least one parameter, to generate a dialog response based on the derived result, and to cause the output device to output the generated dialog response. Responsive to detecting that the device is in a hands-free context, prompting the user for input, receiving user input, interpreting the received user input, determining at least one Identifying a parameter, and generating a dialog response are performed in a manner that complies with the constraints associated with the hands-free context.

The accompanying drawings illustrate several embodiments of the invention and, together with the description, serve to explain the principles of the invention in accordance with the embodiments. Those skilled in the art will recognize that the specific embodiments illustrated in the figures are illustrative only and are not intended to limit the scope of the invention.
≪ 1 >
1 is a screen shot illustrating an example of a hands-on interface for reading a text message according to the prior art;
2,
2 is a screen shot illustrating an example of an interface for responding to a text message;
3A and 3B,
Figures 3a and 3b are a series of screen shots illustrating an example in which a voice dictation interface is used to respond to a text message.
<Fig. 4>
4 is a screen shot illustrating an example of an interface for receiving a text message, according to one embodiment.
5A to 5D,
5A-5D are a series of screen shots illustrating an example of the operation of a multimode virtual assistant in accordance with an embodiment of the present invention in which a user receives and responds to a text message in a hands-free context.
6A to 6C,
6A-6C are a series of screen shots illustrating an example of the operation of a multimode virtual assistant in accordance with an embodiment of the present invention in which a user calibrates a text message in a hands-free context.
7,
7 is a flow diagram illustrating a method for dynamic detection of a hands-free context and a method of operation of a virtual assistant to support adaptation thereof, in accordance with an embodiment;
8,
8 is a block diagram showing an example of a virtual assistant system according to an embodiment;
9,
9 is a block diagram illustrating a computing device suitable for implementing at least a portion of a virtual assistant in accordance with at least one embodiment.
<Fig. 10>
10 is a block diagram illustrating an architecture for implementing at least a portion of a virtual assistant on a standalone computing system, in accordance with at least one embodiment.
11)
11 is a block diagram illustrating an architecture for implementing at least a portion of a virtual assistant on a distributed computing network, in accordance with at least one embodiment.
12,
12 is a block diagram illustrating a system architecture illustrating several different types of clients and modes of operation.
13,
13 is a block diagram illustrating a client and server communicating with one another to implement the invention in accordance with one embodiment.
<Fig. 14>
14 is a functional block diagram of an electronic device according to some embodiments.

According to various embodiments of the present invention, a hands-free context is detected in association with the operations of the virtual assistant and the user interface of the virtual assistant is thereby adjusted so that the user can interact meaningfully with the assistant in a hands- Let's do it.

For purposes of explanation, the term "virtual assistant" is equivalent to the term "intelligent automation assistant ", and both refer to any information handling system that performs one or more of the following functions:

Ability to interpret human language input in voice and / or text form;

The ability to manipulate a representation of a user's intent in a form that can be executed, such as a representation of a task with steps and / or parameters;

The ability to execute task expressions by activating programs, methods, services, APIs, etc.; And

Ability to generate output responses for users in language and / or graphical form.

An example of such a virtual assistant is described in related US patent application Ser. No. 12 / 987,982, filed January 10, 2011, entitled "Intelligent Automation Assistant", the entire disclosure of which is incorporated herein by reference.

Various techniques will now be described in detail with reference to exemplary embodiments as illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects and / or features described or referenced herein. However, it will be apparent to one skilled in the art that one or more aspects and / or features described or referenced herein may be practiced without some or all of these specific details. In other instances, well-known process steps and / or structures are not described in detail so as not to obscure some of the features and / or aspects described or referenced herein.

One or more different inventions may be described in the present application. Also, for one or more of the inventions (s) described herein, a number of embodiments may be described in this patent application and are provided for illustrative purposes only. The described embodiments are not intended to be limiting in any sense. One or more of the present invention (s) may be broadly applicable to a number of embodiments, as will be readily apparent from the disclosure. These embodiments are described in sufficient detail to enable those skilled in the art to practice one or more of the invention (s), and that other embodiments may be utilized, and that structural, logical, It will be appreciated that software, electrical and other changes may be made. Accordingly, those skilled in the art will recognize that one or more of the inventions may be practiced with various modifications and variations. One or more of the specific features of the invention (s), by way of illustration, may be described with reference to one or more specific drawings or embodiments in which one or more specific embodiments of the invention (s) are shown and form a part hereof have. It is to be understood, however, that such features are not limited to use in one or more specific embodiments or drawings to which they are referenced for illustration. The present disclosure is not intended to be a list of one or more of the features of the invention (s) that must be present in all embodiments in the literal description of any one or more of the embodiments of the invention (s).

The titles of the sections provided in such patent application and the title of the present patent application are for convenience only and are not to be taken as limiting the present disclosure in any way.

Devices that communicate with each other need not be in continuous communication with each other unless they are otherwise clearly specified. In addition, devices that communicate with each other can communicate directly or indirectly through one or more intermediaries.

The description of an embodiment having several components communicating with each other does not imply that all such components are required. Conversely, various optional components are described to illustrate one or more of the many different possible embodiments of the invention (s).

Also, while process steps, method steps, algorithms, and the like may be described in a sequential order, such processes, methods, and algorithms may be configured to operate in any suitable order. In other words, any sequence or sequence of steps that may be described in this patent application, by itself and naturally, does not indicate the requirement that the steps be performed in that order. Also, some steps may be performed concurrently (for example, because one step is described after another step), although the steps may be described or implied as occurring non-concurrently. In addition, an illustration of a process by way of illustration in the drawings does not imply that the illustrated process excludes other variations and modifications thereto, and that any of the illustrated processes or steps thereof may be implemented by one or more of the inventions , And does not imply that the illustrated process is preferred.

When a single device or article is described, it will be readily apparent that more than one device / article (whether or not they interact) can be used instead of a single device / article. Similarly, although more than one device or article is described (whether they interact or not), it will be readily apparent that a single device / article may be used in place of more than one device or article.

The functions and / or features of the device may alternatively be implemented by one or more other devices that are not explicitly described as having those functions / features. Accordingly, one or more of the other embodiments of the present invention need not include the device itself.

The techniques and mechanisms described or referenced herein will sometimes be described in singular form for clarity. It should be noted, however, that a particular embodiment encompasses a number of repetitions of the technique or numerous instantiations of the mechanism, unless otherwise noted.

Although described within the context of a technique for implementing an intelligent automation assistant known as a virtual assistant, the various aspects and techniques described herein may also be used in conjunction with computerized interactions with software and / Can be developed and / or applied in various ways.

Other aspects of virtual assistant technology (e.g., utilized and / or provided by and / or implemented by one or more virtual assistant system embodiments described herein) may be found in the entire disclosure herein, Are disclosed in one or more of the following incorporated by reference:

U.S. Patent Application No. 12 / 987,982, entitled "Intelligent Automation Assistant, " filed January 10, 2011;

U.S. Provisional Patent Application 61 / 295,774, filed on January 18, 2010, entitled "Intelligent Automation Assistant ";

U.S. Patent Application No. 13 / 250,854, entitled "Use Context Information to Facilitate Processing of Instructions in a Virtual Assistant, " filed September 30, 2011;

U.S. Patent Application No. 11 / 518,292, filed September 8, 2006, entitled " Method and Apparatus for Constructing Intelligent Automation Assistant ";

U.S. Provisional Patent Application 61 / 186,414, filed June 12, 2009, entitled " System and Method for Semantic Auto-Completion ".

Hardware architecture

Generally, the virtual assistant techniques disclosed herein may be implemented in hardware, or a combination of software and hardware. For example, they may be implemented in an operating system kernel, in a separate user process, in a library package coupled with network applications, on a specially configured machine, and / or on a network interface card. In certain embodiments, the techniques described herein may be implemented in software, such as an operating system, or in an application running on an operating system.

The software / hardware hybrid implementation (s) of at least some of the embodiments of virtual assistant embodiment (s) disclosed herein may be implemented on a programmable machine selectively activated or reconfigured by a computer program stored in memory. Such network devices may have multiple network interfaces that may be configured or designed to use different types of network communication protocols. A generic architecture for some of these machines may appear from the descriptions disclosed herein. According to certain embodiments, at least some of the features and / or functions of the various virtual assistant embodiments disclosed herein may be implemented in a computer system, a computer, a network server or server system, a mobile computing device Such as a personal digital assistant, a mobile phone, a smartphone, a laptop, a tablet computer, etc., a consumer electronic device, a music player, or any other suitable electronic device, router, switch, Lt; RTI ID = 0.0 &gt; general-purpose &lt; / RTI &gt; network host machines. In at least some embodiments, at least some of the features and / or functions of the various virtual assistant embodiments disclosed herein may be implemented in one or more virtualized computing environments (e.g., network computing clouds, etc.).

Referring now to FIG. 9, a block diagram depicting a computing device 60 suitable for implementing at least a portion of the virtual assistant features and / or functions disclosed herein is shown. The computing device 60 may be, for example, an end user computer system, a network server or server system, a mobile computing device (e.g., a personal digital assistant, a mobile phone, a smartphone, a laptop, A music player, or any other suitable electronic device, or any combination or portion thereof. The computing device 60 may be configured to communicate with other computing devices, such as clients and / or servers, via a communication network, such as the Internet, using known protocols for such communication, whether wireless or wired .

In one embodiment, the computing device 60 includes a central processing unit (CPU) 62, interfaces 68, and a bus 67 (e.g., a peripheral component interconnect (PCI) bus). When operating under the control of appropriate software or firmware, the CPU 62 may be responsible for implementing particular functions associated with the functions of the specifically configured computing device or machine. For example, in at least one embodiment, a user's personal digital assistant (PDA) or smart phone is a virtual assistant system using a CPU 62, memories 61 and 65, and interface 68 (s) Lt; / RTI &gt; In at least one embodiment, the CPU 62 may provide different types of virtual assistant functions and / or functions under the control of software modules / components, which may include, for example, an operating system and any suitable application software, drivers, Or perform one or more of the following operations.

CPU 62 may include one or more processors 63 (s), such as, for example, microprocessors of the Motorola or Intel family, or processors from microprocessors of the MIPS family. In some embodiments, processor 63 (s) may include hardware (e.g., application-specific integrated circuit (ASIC), electrically erasable programmable read-only memory (EEPROM) ), A field-programmable gate array (FPGA), etc.). In certain embodiments, the memory 61 (non-volatile RAM and / or ROM) also forms part of the CPU 62. However, there are many different ways in which memory can be coupled to the system. The memory block 61 may be used for various purposes such as, for example, caching and / or storing of data, programming of instructions, and the like.

As used herein, the term "processor" is not limited to only those integrated circuits referred to in the art as a processor, but may be broadly referred to as a microcontroller, microcomputer, programmable logic controller, And any other programmable circuitry.

In one embodiment, the interfaces 68 are provided as interface cards (sometimes referred to as "line cards"). In general, they control the transmission and reception of data packets over the computing network and sometimes support other peripheral devices used with the computing device 60. Among the interfaces that may be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, and the like. In addition, it is also possible to use, for example, general-purpose serial bus (USB), serial, Ethernet, FireWire, PCI, parallel, radio frequency (RF), Bluetooth ™ (for example near-field magnetics) near-field communication, 802.11 (WiFi), Frame Relay, TCP / IP, ISDN, Fast Ethernet interfaces, Gigabit Ethernet interfaces, asynchronous transfer mode (ATM) interfaces, various types of interfaces may be provided, such as serial interface (HSSI) interfaces, point of sale (POS) interfaces, fiber data distributed interface (FDDI) In general, such interfaces 68 may include ports suitable for communication with an appropriate medium. In some cases, these may also include independent processors and may include volatile and / or non-volatile memory (e.g., RAM) in some cases.

The system illustrated in FIG. 9 illustrates one particular architecture for computing device 60 to implement the techniques of the present invention described herein, but at least some of the features and techniques described herein It's never the only device architecture that can be. For example, architectures with one or more processors 63 may be used, and such processors 63 may reside in a single device or be distributed among multiple devices. In one embodiment, the single processor 63 handles communications as well as routing operations. In various embodiments, different types of virtual assistant features and / or functions may be implemented on a client device (e.g., personal digital assistant or smartphone-powered client software) and server system (s) A server system described in detail).

Regardless of the network device configuration, the inventive system may include one or more memories configured to store data, data for general-purpose network operations, program instructions and / or other information relating to the functionality of the virtual assistant technologies described herein, Memory modules (e.g., memory block 65) may be employed. The program instructions may, for example, control the operation of the operating system and / or one or more applications. The memory or memories may also be configured to store data structures, keyword classification information, advertising information, user click and exposure information, and / or other specific non-program information described herein.

Because such information and program instructions may be employed to implement the systems / methods described herein, at least some network device embodiments may include, for example, program instructions, And non-volatile machine-readable storage media that can be configured or designed to store status information, and the like. Examples of such non-volatile machine-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; Optical media such as CD-ROM disks; Magneto-optical media such as floptical disks; And hardware devices that are specifically configured to store and perform program instructions such as ROM, flash memory, memristor memory, RAM, and the like. Examples of program instructions include both machine code as generated by the compiler and files containing high-level code that can be executed by the computer using the interpreter.

In one embodiment, the system of the present invention is implemented on a standalone computing system. Referring now to FIG. 10, a block diagram depicting an architecture for implementing at least a portion of a virtual assistant on a standalone computing system, in accordance with at least one embodiment, is shown. The computing device 60 includes a processor 63 (s) that runs software to implement the multi-mode virtual assistant 1002. The processor 63 (s) The input device 1206 may be any type suitable for receiving user input, including, for example, a keyboard, a touch screen, a mouse, a touchpad, a trackball, a five-way switch, a joystick, and / have. The device 60 may also include a speech input device 1211, such as, for example, a microphone. Output device 1207 may be a screen, a speaker, a printer, and / or any combination thereof. The memory 1210 may be a RAM having the structure and architecture as known in the art, for use by the processor 63 (s) while running the software. The storage device 1208 may be any magnetic, optical and / or electrical storage device for storage of data in digital form; Examples include flash memory, magnetic hard drives, CD-ROMs, and / or the like.

In another embodiment, the system of the present invention is implemented on a distributed computing network, such as having multiple clients and / or servers. Referring now to FIG. 11, a block diagram depicting an architecture for implementing at least a portion of a virtual assistant on a distributed computing network, in accordance with at least one embodiment, is shown.

In the arrangement shown in Figure 11, a number of clients 1304 are provided; Each client 1304 may run software to implement the client-side portions of the present invention. In addition, multiple servers 1340 can be provided to handle requests received from clients 1304. Clients 1304 and servers 1340 may communicate with each other via an electronic network 1361, such as the Internet. The network 1361 may be implemented using any known network protocols including, for example, wired and / or wireless protocols.

In addition, in one embodiment, the servers 1340 may call external services 1360 as needed to obtain additional information or refer to stored data relating to previous interactions with particular users. Communications with external services 1360 may occur via network 1361, for example. In various embodiments, external services 1360 include web-enabled services and / or functions associated with or installed on the hardware device itself. For example, in an embodiment where the assistant 1002 is implemented on a smartphone or other electronic device, the assistant 1002 may be stored in a calendar application ("app"), contacts, and / Information can be obtained.

In various embodiments, the assistant 1002 can control many features and operations of the electronic device in which the assistant is installed. For example, the assistant 1002 may interface with functions and applications on the device via APIs or by other means to perform functions and operations that may be otherwise initiated using the conventional user interface on the devices Lt; RTI ID = 0.0 &gt; 1360 &lt; / RTI &gt; Such functions and operations may include, for example, setting an alarm, placing a call, sending a text message or an email message, adding calendar events, and the like. Such features and operations may be performed as add-on functions in the context of the dialogue dialog between the user and the assistant 1002. [ Such functions and operations may be specified by the user in the context of such a dialog, or they may be performed automatically based on the context of the dialog. Those skilled in the art will appreciate that the assistant 1002 can thereby be used as a control mechanism for initiating and controlling various operations on an electronic device, which can be used as an alternative to conventional mechanisms such as buttons or graphical user interfaces Will recognize.

For example, the user may provide an input to the assistant 1002, such as "I have to wake up tomorrow morning at 8 am ". Once the assistant 1002 has determined the intent of the user using the techniques described herein, the assistant 1002 may invoke external services 1360 to interface with the alarm clock function or application on the device. The assistant 1002 sets an alarm on behalf of the user. In this manner, the user can use the assistant 1002 as an alternative to conventional mechanisms for setting alarms or performing other functions on the device. If the user's requests are ambiguous or require additional clarification, the assistant 1002 may use various techniques described herein, including active derivation, paraphrasing, suggestions, etc., which may be adapted to the hands-free context The correct services 1360 are invoked and the intended operation is taken. In one embodiment, the assistant 1002 may request additional contextual information from any suitable source before invoking the service 1360 to prompt and / or perform a function for verification. In one embodiment, the user may selectively disable the ability of the assistant 1002 to call specific services 1360, or may disable all such service-calls, if desired.

The system of the present invention may be implemented with any of a number of different types of clients 1304 and modes of operation. Referring now to FIG. 12, a block diagram depicting a system architecture illustrating several different types of clients 1304 and modes of operation is shown. Those skilled in the art will appreciate that the various types of clients 1304 and modes of operation illustrated in FIG. 12 are merely exemplary and that the system of the present invention may be implemented using clients 1304 and / or modes of operation other than those depicted &Lt; / RTI &gt; In addition, the system may include any or all of such clients 1304 and / or modes of operation, alone or in combination. Illustrative examples include:

Computer devices having input / output devices and / or sensors 1402. The client component may be located on any such computer device (1402). At least one embodiment may be implemented using a web browser 1304A or other software application to enable communication with the servers 1340 via the network 1361. [ The input and output channels may be of any type, including, for example, visual and / or auditory channels. For example, in one embodiment, the system of the present invention utilizes voice-based communication methods, which allow an embodiment of an assistant for the visually impaired to have an equivalent of a web browser driven by speech and using speech for output. .

Mobile devices with I / O and sensors 1406, wherein the client may be implemented as an application on mobile device 1304B. This includes, but is not limited to, mobile telephones, smart phones, personal digital assistants, tablet devices, networked game consoles, and the like.

Consumer devices having I / O and sensors 1410, wherein the client may be implemented as an application embedded on device 1304C.

Automobiles and other vehicles with dashboard interfaces and sensors 1414 that can be implemented as a system application 1304D with a client embedded. This includes, but is not limited to, car navigation systems, voice control systems, in-car entertainment systems, and the like.

Networked computing devices such as routers 1418 or any other device that resides on or interfaces with the network, where the client may be implemented as a device-resident application 1304E.

An email client 1424 in which an embodiment of an assistant is connected via email form server 1426. The e-mail form server 1426 acts as a communication bridge, for example, taking input from a user as email messages sent to an assistant and sending output from the assistant as responses, to users.

Instant messaging clients 1428, in which embodiments of an assistant are connected through a messaging style server 1430. Messaging style server 1430 acts as a communication bridge to take input from the user as an assisted-forwarded message and to send the output from the assistant as messages in response to the user.

Voice phone 1432 to which embodiments of the assistant are connected via a Voice over Internet Protocol (VoIP) form server 1434. The VoIP form server 1434 acts as a communication bridge to take input from the user as the voice spoken to the assistant and transmit the output from the assistant to the user as a synthesized speech in response, for example.

For messaging platforms including, but not limited to, e-mail, instant messaging, discussion forums, group chat sessions, live help or customer support sessions, assistant 1002 may act as a participant in conversations. Assistant 1002 can monitor conversations and respond to individuals or groups using one or more of the techniques and methods described herein for one-to-one interactions.

In various embodiments, the functionality for implementing the techniques of the present invention may be distributed among a number of client and / or server components. For example, various software modules may be implemented to perform various functions in connection with the present invention, and such modules may be variously implemented to operate on a server and / or client components. Additional details for such arrangements are provided in related US patent application Ser. No. 12 / 987,982, entitled " Intelligent Automation Assistant, " filed January 10, 2011, the entire disclosure of which is incorporated herein by reference. do.

In the example of Figure 13 the input and output processing functions are such that the client portion 2794a of the input derivation and the client portion 2792a of the output processing are located in the client 1304 and the server portion 2794b and the output Is distributed among the client 1304 and the server 1340, with the server portion 2792b of the processing being located in the server 1340. [ The following components are located at server 1340:

· Completed vocabulary (2758b);

A completion library 2760b of language pattern recognizers;

A master version 2752b of short term personal memory;

· Master version of long-term personal memory (2754b).

In one embodiment, client 1304 locally maintains subsets and / or portions of these components to improve responsiveness and reduce dependence on network communications. These subsets and / or portions may be maintained and updated in accordance with well known cache management techniques. Such subsets and / or portions include, for example, the following:

A subset of the vocabulary 2758a;

A subset 2760a of libraries of language pattern recognizers;

Cache 2752a of short term personal memory;

Cache 2754a of long term personal memory.

Additional components may be implemented as part of the server 1340, including, for example, the following:

A language interpreter 2770;

A dialog flow processor 2780;

An output processor 2790;

Domain entity databases 2772;

· Task flow models (2786);

A combination of services 2782;

· Service capability models (2788).

Server 1340 obtains additional information by interfacing with external services 1360 as required.

Referring now to FIG. 14, a functional block diagram of an electronic device 2000 in accordance with some embodiments is shown. The functional blocks of the device may be implemented by hardware, software, or a combination of hardware and software to implement the principles of the present invention. It is understood by those skilled in the art that the functional blocks described in FIG. 14 may be combined or separated into sub-blocks to implement the principles of the invention as described above. Thus, the description herein may support any possible combination or separation or further definition of the functional blocks described herein.

As shown in FIG. 14, the electronic device 2000 includes a processing unit 2006. In some embodiments, the processing unit 2006 includes a context detection unit 2008, a user input interpretation unit 2010, a task and parameter identification unit 2012, a task execution unit 2014, and a dialog response generation unit 2016 ). The electronic device 2000 also includes an output unit 2002 coupled to the processing unit and configured to prompt the user for input. The electronic device also includes an input unit 2003 coupled to the processing unit and configured to receive user input. In some embodiments, the electronic device 2000 may also include one or more sensors and / or sensors of the device 2000, such as an environmental condition sensor, a peripheral device, an onboard system of the vehicle, a position sensor (e.g., GPS sensor) Or a communication unit 2004 configured to receive data from sensors / devices external to the electronic device 2000.

The processing unit detects whether the hands-free context is active (e.g., using the context detection unit 2008); Interpret the received user input to derive a representation of the user's intention (e.g., using the user input interpretation unit 2010); Identify at least one parameter for at least one task and a task (e.g., using task and parameter identification unit 2012) based at least in part on an expression of the derived user intent; Execute at least one task using at least one parameter to derive a result (e.g., using task execution unit 2014); Generate a dialog response based on the derived result (e.g., using the dialog response generating unit 2016); (E.g., using output unit 2002) to cause the output device to output the generated dialog response. In response to detecting that the device is in a hands-free context (e.g., by context detection unit 2008), prompting the user for input, receiving user input, interpreting the received user input , Identifying at least one parameter for at least one task and a task, and generating a dialog response (e.g., input unit 2003, output unit 2002, and / or processing unit 2006)) in accordance with the constraints associated with the hands-free context.

In some embodiments, in response to detecting that the device is in a hands-free context, if at least two interaction modes are available for user interaction with the computing device, prompting the user for input, At least one of the steps of receiving an input, interpreting a received user input, identifying at least one task and at least one parameter for the task, and generating a dialog response (e.g., 2006) using a first interaction mode adapted for hands-free operation; In response to detecting that the device is not in a hands-free context, prompting the user for input, receiving user input, interpreting the received user input, determining at least one task and at least At least one of the steps of identifying a parameter and generating a dialog response may be performed in a second interaction mode that is not adapted to the hands-free operation (e.g., using one or more units of the processing unit 2006) .

In some embodiments, the processing unit 2006 includes: a user's ability to view the visual output provided by the computing device; The user's ability to interact with the graphical user interface provided by the computing device; The ability of a user to use physical components of a computing device; The ability of a user to perform touch input on a computing device; The ability of the user to activate the switch on the computing device; (E.g., using the context detection unit 2008) to detect whether the hands-free context is active or not by detecting conditions indicative of a restriction in at least one selected from the group consisting of the ability of the user to use the keyboard on the computing device .

In some embodiments, the output unit 2002 is responsive to detecting that the device is not in a hands-free context, by prompting the user through a first output mode that is not adapted to the hands-free context, - prompting the user for input by prompting the user via a second output mode adapted to the hands-free context, in response to detecting that the user is in the free context. In some embodiments, the first output mode is a visual output mode. In some embodiments, the second output mode is an audible output mode.

In some embodiments, prompting the user through the visual output mode (e.g., using the output unit 2002) may include displaying a prompt on the display screen (e.g., using the output unit 2002) ) Prompting the user through the audible output mode involves outputting a spoken prompt.

In some embodiments, the processing unit 2006 is responsive to detecting that the device is in a hands-free context, using the vocabulary associated with the hands-free operation (e.g., using the user interpretation unit 2010) And is configured to interpret user input.

In some embodiments, in response to detecting that the device is in a hands-free context, the processing unit 2006 may identify at least one task flow associated with the hands-free operation (e.g., using the task execution unit 2014) Step &lt; / RTI &gt; In some embodiments, performing at least one task flow identification step associated with a hands-free operation may be performed using a speech-based interface (e.g., using output unit 2002 and / or input unit 2003) Lt; RTI ID = 0.0 &gt; review &lt; / RTI &gt;

In some embodiments, the processing unit 2006 is also configured to perform at least one task flow step using the audible output (e.g., using the task execution unit 2014 and / or the output unit 2002) do. In some embodiments, the processing unit 2006 may perform at least one task flow step (e.g., using the task execution unit 2014) selected from a limited group of available task flow steps suitable for the hands-free context And is further configured to perform at least one task flow identification step.

In some embodiments, in response to detecting that the device is in a hands-free context, the processing unit 2006 may be configured to generate a dialog response in a speech-based output mode (e.g., using the dialog response generation unit 2016) .

In some embodiments, the processing unit 2006 is configured to generate a dialog response in a speech-based output mode by translating at least a portion of the user input in a voice form (e.g., using the dialog response generation unit 2016) . In some embodiments, the processing unit 2006 may generate a speech by using a plurality of voices to distinguish user input that is diverted from another voice output (e.g., using the dialog response generation unit 2016) Based output mode. In some embodiments, the processing unit 2006 generates a dialog response in a speech-based output mode by combining a dialog template with at least one item of personal data (e.g., using the dialog response generation unit 2016) .

In some embodiments, the processing unit 2006 may be configured to receive a user input that specifies a hands-free context (e.g., using the input unit 2003); Receiving data indicative of environmental conditions associated with the hands-free context from at least one sensor (e.g., using communication unit 2004); Detecting a connection of peripheral devices associated with the hands-free context (e.g., using communication unit 2004); Detecting disconnection of peripheral devices not associated with the hands-free context (e.g., using communication unit 2004); Detecting communication with the onboard system of the vehicle (e.g., using communication unit 2004); (E.g., using the communication unit 2004) to detect the current position; And to detect whether the hands-free context is active by performing at least one process selected from the group consisting of detecting the current speed (e.g., using communication unit 2004).

In some embodiments, the output unit 2002 is configured to prompt the user via a dialog interface; The input unit 2003 is configured to receive user input via a dialog interface. In some embodiments, the input unit 2003 is configured to receive speech input; The processing unit 2006 is configured to convert the speech input to a textual representation (e.g., using the user input interpretation unit 2010).

Conceptual architecture

Referring now to FIG. 8, a simplified block diagram of a particular exemplary embodiment of a multimode virtual assistant 1002 is shown. Different embodiments of the multi-mode virtual assistant 1002 generally include a variety of different types of operations, functions, and / or features relating to virtual assistant technology, as described in more detail in related US patent applications referenced above. May be configured, designed, and / or operable to provide, Also, as described in greater detail herein, many of the various operations, functions, and / or features of the multimode virtual assistant 1002 disclosed herein may be implemented using different entities / RTI &gt; may provide or provide different types of advantages and / or benefits to the user. The embodiment shown in FIG. 8 may be implemented using any of the hardware architectures described above, or using different types of hardware architectures.

For example, in accordance with different embodiments, the multi-mode virtual assistant 1002 may include a variety of different types of operations, functions, and / or features, such as, for example, one or more Designed and / or operable to provide:

Automate applications of data and services available over the Internet to discover, find, select, purchase, reserve or order products and services. In addition to automating processes that use these data and services, the multimode virtual assistant 1002 can also immediately enable the combined use of multiple sources of data and services. For example, the multimode virtual assistant combines information about products from a number of review sites, checks prices and availability from multiple distributors, checks their location and time constraints, They can help you find a personalized solution to their problem.

· Things to do (including, but not limited to, movies, events, performances, exhibitions, shows and attractions); (Including, but not limited to, travel destinations, hotels and other places to stay, landmarks, and other sites of interest); To discover, investigate, and provide information about food and drink places (e.g., restaurants and bars), times and places for meeting other things, and any other source of entertainment or social interaction that can be found on the Internet, And automates the use of data and services available over the Internet to select, reserve, and otherwise learn from them.

· Navigation (including location-based navigation); Navigation (maps and directions); Searching the database (e.g., looking for businesses or people by name or other attributes); Obtaining weather conditions and forecasts, checking the price of market items or the status of a financial transaction; Monitoring the status of traffic or flights; Accessing and updating calendars and schedules; Managing reminders, alerts, tasks and projects; Communicating via email or other messaging platforms; And a graphical user interface, including locally or remotely operating devices (e.g., dialing phones, controlling light and temperature, controlling home security devices, playing music or video, etc.) Enables the operation of applications and services, which are otherwise provided by dedicated applications having interfaces, via natural language dialogs. In one embodiment, the multimode virtual assistant 1002 can be used to launch, operate, and control many of the functions and applications available on the device.

· Any other kind of recommendation that benefits from activities, products, services, sources of entertainment, personal recommendations for time management, or interactive dialogs in natural language and from automated access to data and services. Service.

According to different embodiments, at least a portion of the various types of functions, operations, operations, and / or other features provided by the multimode virtual assistant 1002 may be implemented in one or more client systems (s) System (s), and / or combinations thereof.

According to different embodiments, at least some of the various types of functions, operations, operations, and / or other features provided by the multimode virtual assistant 1002 may be implemented as user input The context information can be used when analyzing and manipulating the information.

For example, in at least one embodiment, the multimode virtual assistant 1002 may be operable to utilize and / or generate various different types of data and / or other types of information when performing particular tasks and / It can be possible. This may include, for example, input data / information and / or output data / information. For example, in at least one embodiment, the multimode virtual assistant 1002 may be coupled to one or more different types of sources, such as, for example, one or more local and / or remote memories, devices, and / May be operable to access, process, and / or otherwise use the information. Additionally, in at least one embodiment, the multimode virtual assistant 1002 may include one or more different types of output data / data streams, which may be stored in, for example, one or more local and / or remote devices and / May be operable to generate information.

Examples of different types of input data / information that may be accessed and / or utilized by the multimode virtual assistant 1002 may include, but are not limited to, one or more of the following (or a combination thereof)

Voice input: mobile devices such as mobile phones and tablets, computers with microphones, Bluetooth headsets, from automobile voice control systems, telephone systems, recordings on response services, on unified messaging services Consumer applications with voice inputs such as audio voice mail, clock radios, telephone offices, home entertainment control systems, and game consoles.

Keyboards on computers or mobile devices, keypads on remote controls or other consumer electronic devices, email messages sent to an assistant, instant messages sent to an assistant or similar short messages, Text received from players in environments, and text streamed from message feeds.

Location information from sensors or location - based systems. Examples include GPS (Global Positioning System) and A-GPS (Assisted GPS) on mobile phones. In one embodiment, the location information is combined with explicit user input. In one embodiment, the system of the present invention can detect when a user is at home based on known address information and current positioning. In this way, as opposed to being outside the home, there is a certain amount of information about the types of services and behaviors that should be called on behalf of the user, depending on the type of information that the user may be interested in, Inferences can be made.

Time information from clocks on client devices. This may include, for example, time from telephones or other client devices indicating local time and time zones. In addition, time can be used in the context of user requests, for example, to interpret phrases such as "within an hour" and "tonight ".

Other sensor data from embedded systems, such as compass, accelerometer, gyroscope, and / or movement speed data, as well as mobile or handheld devices or auto-motion control systems. It may also include device location data from remote controls to devices and game consoles.

Clicking and menu selection and other events from the GUI on any device with a graphical user interface (GUI). Additional examples include touches to the touch screen.

Other data-driven triggers, such as events from sensors and alarm clocks, calendar alerts, pricing triggers, location triggers, push notifications from servers to devices, etc.

The input to the embodiments described herein also includes the context of the user interaction history, including the dialog and request history.

Many different types of output data / information can be generated by the multimode virtual assistant 1002, as described in the above-mentioned related U.S. patent applications. These may include, but are not limited to, one or more of the following (or combinations thereof):

Text output sent directly to the output device and / or to the user interface of the device;

Text and graphics sent to the user via email;

Text and graphics sent to the user through the messaging service;

Speech output that may include one or more of the following (or combinations thereof):

synthesized speech;

o sampled speech;

o Recorded messages;

Graphic layout of information with pictures, rich text, videos, sounds, and hyperlinks (e.g., content rendered in a web browser);

An actuator output for controlling physical operations on the device, such as to cause the device to turn on or off, to make a sound, to change color, to vibrate, to control light;

It is also possible to use: - calling a mapping application, voice dialing a phone, sending an email or an instant message, playing media, creating entries in calendars, task managers, and note applications, &Lt; / RTI &gt; on the device;

For devices attached or controlled by a given device, such as operating a remote camera, controlling a wheelchair, playing music on remote speakers, playing videos on remote displays, etc. Actuator output to control physical operations.

It should be appreciated that the multimode virtual assistant 1002 of FIG. 8 is merely exemplary from a wide range of virtual assistant system embodiments that may be implemented. Other embodiments (not shown) of the virtual assistant system may include additional, fewer and / or different components / features than those illustrated in, for example, the exemplary virtual assistant system embodiment of FIG. 8 .

The multimode virtual assistant 1002 may comprise a plurality of different types of components, devices, modules, processes, and / or components that may be implemented and / or implemented through use of, for example, hardware and / Systems, and the like. 8, the assistant 1002 includes one or more (or combinations thereof) of the following types of systems, components, devices, processors, etc. can do:

One or more active ontologies 1050;

Active input induction component (s) 2794 (which may include a client portion 2794a and a server portion 2794b);

Short term personal memory component (s) 2752 (may include master version 2752b and cache 2752a);

Long term personal memory component (s) 2754 (may include master version 2754b and cache 2754a);

Domain models component (s) 2756;

Lexical component (s) 2758 (which may include the completed vocabulary 2758b and subset 2758a);

Language pattern recognizer (s) component (s) 2760 (may include full library 2760b and subset 2760a);

Language interpreter component (s) 2770;

Domain entity database (s) 2772;

Dialogue flow processor component (s) 2780;

Service combination component (s) 2782;

Services components (s) 2784;

Task flow model component (s) 2786;

Dialogue flow model component (s) 2787;

Service model component (s) 2788;

Output processor component (s) 2790.

In some client / server-based embodiments, some or all of these components may be distributed between the client 1304 and the server 1340. Such components are further described in the related U.S. patent applications referred to above.

In one embodiment, virtual assistant 1002 receives user input 2704 via any suitable input form including, for example, touch screen input, keyboard input, voice input, and / or any combination thereof. In one embodiment, the assistant 1002 is also referred to as "facilitating processing of instructions in the virtual assistant ", filed September 30, 2011, the entire disclosure of which is incorporated herein by reference. An application context, a personal acoustic context, and / or other forms of information, as described in U.S. Patent Application No. 13 / 250,854, entitled &quot; Using Context Information to Facilitate Processing of Commands in a Virtual Assistant &Lt; RTI ID = 0.0 &gt; 1000 &lt; / RTI &gt; Context information 1000 also includes a hands-free context that, if applicable, can be used to adapt the user interface in accordance with the techniques described herein.

When processing user input 2704 and context information 1000 in accordance with the techniques described herein, virtual assistant 1002 generates an output 2708 for presentation to a user. Output 2708 may be generated according to any suitable output format, which may be notified by hands-free context as well as other factors if appropriate. Examples of output forms include visual output as provided on the screen, audible output (which may include audio output and / or beeps and other sounds), haptic output (e.g., vibration), and / Or any combination thereof.

Additional details regarding the operation of the various components depicted in FIG. 8 may be found in related U.S. patent applications to "Intelligent Automation Assistant" filed January 10, 2011, the entire disclosure of which is incorporated herein by reference. 12 / 987,982.

Hands - free In the context  Adaptation of user interfaces

For illustrative purposes, the present invention is described herein by way of example. However, those skilled in the art will recognize that the particular input and output mechanisms depicted in the examples are intended to illustrate only one possible interaction between the user and the assistant 1002, and are not intended to limit the scope of the invention as claimed. something to do. Moreover, in alternative embodiments, the present invention may be implemented in a device without necessarily involving a multimode virtual assistant 1002; Rather, the functionality of the invention may be embodied directly in an operating system or application running on any suitable device, without departing from the essential characteristics of the present invention as defined in the claims.

Referring now to FIG. 1, there is shown a screen shot illustrating an example of a conventional hands-on interface 169 for reading a text message, in accordance with the prior art. A graphical user interface (GUI), such as that shown in Figure 1, generally refers to the ability of the user to read fine details such as the message text shown in the bubble 171, to type in the text field 172, ) By tapping. In many devices, such operations require viewing and touching the screen, and thus it is impractical to perform in certain contexts referred to herein as hands-free contexts.

Referring now to FIG. 2, a screen shot illustrating an example of an interface 170 for responding to a text message 171 is shown. A virtual keyboard 270 is provided in response to user tapping in the text field 172 to cause the text to be entered into the text field 172 by tapping areas of the screen corresponding to the keys. The user taps the transmit button 173 when a text message is entered. If the user wishes to enter text by speaking, the user taps the speech button 271, which invokes the voice dictation interface to receive the speech input and convert it to text. Thus, the button 271 provides a mechanism that allows the user to indicate that the user is in a hands-free context.

3A and 3B, a series of screen shots illustrating an example of an interface 175 in which a voice dictation interface is used to respond to a text message 171 is shown. The screen 370 is provided after the user has tapped the speech button 271, for example. The microphone icon 372 indicates that the device is ready to accept the voice input. The user inputs speech, which is received via speech input device 1211, which may be a microphone or similar device. The user taps the completion button 371 to indicate that the user has completed inputting the voice input.

The speech input is converted to text using any well-known speech-to-text algorithm or system. The speech-to-text function may reside on the device 60 or on the server. In one embodiment, the speech-to-text function is implemented using, for example, a Nuance Recognizer available from Nuance Communications, Inc., Burlington, Mass., USA.

As shown in FIG. 3B, the results of the transformation may be shown in field 172. FIG. The keyboard 270 may be provided to allow the user to edit the text generated in the field 172. When the user is satisfied with the entered text, the user taps the transmit button 173 to allow the text message to be transmitted.

In the example described with reference to Figures 2, 3a, and 3b, several actions require the user to view / view the display screen or provide a touch input. Such operations include the following:

Reading the text message 171 on the display screen;

- Touching the button 271 to enter the speech input mode;

- touching the Done button 371 to indicate that the speech input is complete;

Viewing the converted text generated from the user &apos; s speech input;

• Touching the transmit button 173 to transmit a message.

In one embodiment of the present invention, the mechanisms for accepting and processing speech input are configured in a manner that allows the user to interact with the display screen and / or reduce the need to use a touch interface when in the hands- ). Thus, the system of the present invention can thereby provide an improved user interface for interaction in a hands-free context.

Referring now to FIG. 4 and FIGS. 5A-5D, a series of screen shots is illustrated illustrating an example of an interface for receiving and responding to a text message, in accordance with one embodiment in which a hands-free context is recognized; Thus, in this example, according to the techniques of the present invention, the need for the user to interact with the screen is reduced.

In FIG. 4, screen 470 depicts a text message 471 that is received while device 60 is in the locked mode. The user may activate the slider 472 to respond to or otherwise interact with the message 471 in accordance with known techniques. However, in this example, the device 60 may be invisible and / or out of reach, or the user may interact with the device 60 if, for example, the user is driving or participates in some other activity It will not be possible. As described herein, the multimode virtual assistant 1002 provides functionality for receiving and responding to a text message 471 in such a hands-free context.

In one embodiment, the virtual assistant 1002 installed on the device 60 automatically detects the hands-free context. Such detection may occur by any means by which a user may interact with the screen of the device 60 or determine a scenario or situation in which it may be difficult or impossible to properly operate the GUI.

For example and without limitation, the judgment of the hands-free context can be made alone or in any combination based on any of the following:

Data from sensors (eg, compasses, accelerometers, gyroscopes, speedometers, ambient light sensors, Bluetooth connectivity detectors, clocks, WiFi signal detectors, microphones, etc.);

For example, via GPS to determine that the device 60 is in a predetermined geographic location;

Data from the clock (e.g., the hands-free context may be defined to be active at certain times of day and / or on certain days of the week);

Predefined parameters (e.g., the user or administrator may specify that the hands-free context is active when any condition or combination of conditions is detected);

A connection of Bluetooth or other wireless I / O devices (e.g., when a connection with a Bluetooth-enabled interface of a mobile vehicle is detected);

Any other information that may indicate that the user is in or is driving a moving vehicle;

The presence or absence of attached peripherals including headphones, headsets, those connected by adapter cables, etc .;

Determining whether the user is not in contact with or in proximity to the device 60;

(E.g., a motion gesture in which a user holds a device in the ear, or a push on a Bluetooth device, or on an attached audio device) that is used to trigger an interaction with the assistant 1002 Pushing the button of the button);

Detection of specific words in a continuous stream of words (e.g., the assistant 1002 is configured to be called to hear commands and when the user speaks some command, such as a name or "computer! And a particular command may indicate whether the hands-free context is active).

In other embodiments, the user can manually indicate whether the hands-free context is active or inactive and / or to activate / deactivate certain days of the week and / or certain days of the week, - Free context can be scheduled.

In one embodiment, upon receiving the text message 470 while in the hands-free context, the multimode virtual assistant 1002 causes the device 60 to output an audio indication, such as a beep or tone, Display the reception of a text message. As described above, the user may activate the slider 472 to respond to or otherwise interact with the message 471 in accordance with known techniques (e.g., if the hands-free mode is detected incorrectly, or If the user chooses to stop the drive or otherwise allows the user to make himself available for hands-on interaction with the device 60). Alternatively, the user may participate in the voice dialog with the assistant 1002 to enable interaction with the assistant 1002 in a hands-free manner.

In one embodiment, the user initiates a voice dialogue by any suitable mechanism appropriate for the hands-free context. For example, in an environment where a user drives a Bluetooth-equipped vehicle and the device 60 communicates with the vehicle, an easily-accessed button (e.g., a button mounted on the steering wheel of the vehicle) becomes available . The pressing of the button initiates a voice dialog with the assistant 1002 and allows the user to communicate with the assistant 1002 via a Bluetooth connection and through a microphone and / or speaker installed in the vehicle. Alternatively, a user may select some other distinction that signals to the assistant 1002 that he or she wants to initiate a voice dialog on the device 60 itself, on a headset, or on any other peripheral device, The voice dialogue can be started. As another example, the user may speak an instruction understood by the assistant 1002 and initiating a voice dialogue. Those skilled in the art will appreciate that many other techniques may be provided to allow the user to easily initiate voice dialogs with the assistant 1002. [ Preferably, the mechanism used to initiate the voice dialogue does not require eye-hand synchro- nization on the user side, allowing the user to focus on the main task, such as driving, and / or in Figures 2, 3a, May be performed by an individual having adverse conditions that interfere with, inhibit, constrain, or limit the ability of the user to interact with the GUI as depicted.

Once the voice dialogue is started, the assistant 1002 hears voice input. In one embodiment, the assistant 1002 announces receipt of the speech input by some output mechanism that is easily detected by the user while in the hands-free context. An example is a visual output on a vehicle dashboard that is easily seen by the user, and / or some other mechanism, during audio chimes or tones, and / or even during driving. Voice input is processed using known speech recognition techniques. The assistant 1002 then performs the action (s) indicated by the voice input. In one embodiment, the assistant 1002 provides audio output, which may be output via speakers (either on the device 60 or installed in a vehicle), headphones, etc., to continue the audio dialog with the user . For example, the assistant 1002 can read content such as text messages, e-mail messages, and the like, and can provide options to the user in a voice form.

For example, if the user says "read my new message ", the assistant 1002 can cause the device 60 to issue an acknowledgment tone. Assistant 1002 can then output voice output such as "You have a new message from Tom Devon". The assistant says, "Hey, let's go to the game." Voice output may be generated by the assistant 1002 using any known technique for converting text to speech. In one embodiment, the text-to-speech function is implemented using, for example, a Nuance Vocalizer available from Nuance Communications, Inc., Burlington, Mass., USA.

Referring now to FIG. 5A, an example of a screenshot 570 showing the output that can be viewed on the screen of the device 60 during language interchange between the user and the assistant 1002 is shown. In some hands-free situations, the user can, for example, see the screen but not easily touch the screen if the output on the screen of the device 60 is duplicated on the display screen of the vehicle's navigation system. The visual echoing of a voice conversation, as depicted in FIGS. 5A-5D, may help the user confirm that the voice input of the user is properly and accurately understood by the assistant 1002, Lt; RTI ID = 0.0 &gt; voice responses. &Lt; / RTI &gt; However, such visual reflections are optional, and the present invention may be implemented without any visual display on the screen of the device 60 or elsewhere. Thus, the user can interact with the assistant 1002 only by voice input and output, or by a combination of visual and speech inputs and / or outputs.

In this example, the assistant 1002 indicates and speaks the prompt 571. [ In response to the user input, the assistant 1002 repeats the user input 572 on the display and / or in the form of a voice. The assistant then introduces an incoming text message (573) and reads it. In one embodiment, the text message may also be displayed on the screen.

As shown in FIG. 5B, after reading the incoming message for the user, the assistant 1002 then tells the user that the user can "reply to the message or read the message again" (574). Once again, such an output, in one embodiment, is provided in a voice form (i.e., verbally). In this way, the system of the present invention does not require the user to view text fields, buttons, and / or links and does not require interaction with on-screen objects or direct manipulation by touch, And informs the user of available operations in a manner that is well-adapted to the hands-free context. As depicted in FIG. 5B, in one embodiment, the speech output is echoed on the screen (574), but such an indication of the speech output is not required. In one embodiment, echo messages displayed on the screen are automatically scrolled upward according to well-known mechanisms.

In this example, the user says, "Yes, answer what is there at 6 o'clock." As depicted in FIG. 5B, in one embodiment, the user's voice input is echoed to allow the user to check 575 that the voice input is properly understood. In addition, in one embodiment, the assistant 1002 repeats the user's speech input in an audible form, allowing the user to verify his / her understanding of the command even though the user can not see the screen. Thus, the system of the present invention does not require the user to interact with the device 60 or view the screen in a manner that is either not feasible or not well suited to the current operating environment, both in the hands-free context, , Configure the response, and provide a mechanism to ensure that the command and configured response are properly understood.

In one embodiment, the assistant 1002 provides additional confirmation of the user's configured text message by reading the message back. In this example, the assistant 1002 verbally says, "Your response to Tom Devon is:" Yeah, I'll be there at 6. ". In one embodiment, For example, the string "Your response to Tom Devon is here" can be spoken in one voice like a male voice, while the string "Yes, at 6 o'clock Can be said to be a different voice, such as a female voice. Alternatively, the same voice may be used with different rhyme to convey the quotation marks.

In one embodiment, the assistant 1002 provides a visual echo of voice exchange, as depicted in Figures 5B and 5C. Figures 5B and 5C show a message 576 that echoes the voice output of the assistant 1002 of "Your response to Tom Devon is here ". FIG. 5C shows a summary 577 of the composed text message including the recipient and content of the message. In Figure 5c, previous messages are scrolled upwards outside the screen, but can be seen by scrolling downward according to known mechanisms. The transmission button 578 transmits the message, and the cancel button 579 cancels the message. In one embodiment, the user can also send or cancel the message by speaking a keyword such as "send" or "cancel ". Alternatively, the assistant 1002 may generate a voice prompt such as "Are you ready to transmit ?, and once again, the display 570 with the buttons 578 and 579 will be shown while the voice prompt is being output . The user can then display what the user wants by touching the buttons 578, 579 or by answering the voice prompt. Prompts can be generated in a format that allows an "yes " or" no "response, thereby obviating the need for the user to use any special vocabulary to inform the user's intent.

In one embodiment, the assistant 1002 can verify the user's voice command to transmit the message, for example, by generating a voice output such as "yes, your message will be sent &quot;. As shown in FIG. 5D, this voice output may be echoed on screen 570 (580), along with a summary 581 of text messages to be transmitted.

The above-described voice exchange combined with selective visual echo illustrates an example of how the assistant 1002 provides redundant outputs in a multimode interface. In this way, the assistant 1002 can support a range of contexts including eyes-free, hands-free, and complete hands-on.

This example also illustrates the mechanisms by which the displayed and spoken output can be made to differ from each other to reflect their different contexts. This example also illustrates the ways in which alternative mechanisms for responding become available. For example, after the assistant speaks "Are you ready to send a message?" And displays the screen 570 shown in FIG. 5C, the user may speak the word "TRANSMIT" or "YES" ). &Lt; / RTI &gt; Any of these actions will be interpreted in the same way by the assistant 1002, causing the text message to be transmitted. Thus, the system of the present invention provides a high degree of flexibility for the interaction of the user with the assistant 1002.

6A-6C, there is shown a series of screen shots illustrating an example of the operation of a multimode virtual assistant 1002 in accordance with an embodiment of the present invention, wherein the user may, for example, correct To add more content, the text message 577 is calibrated in a hands-free context. In a visual interface with direct manipulation as described above in connection with FIGS. 3A and 3B, the user can edit the contents of the text field 172 and edit it on the virtual keyboard 270 to correct the text message 577 You can type. Because such operations may not be feasible in a hands-free context, the multimode virtual assistant 1002 provides a mechanism for such editing of the text message 577 to occur through voice input and output at the conversation interface .

In one embodiment, once the text message 577 is configured (e.g., based on the user's voice input), the multi-mode virtual assistant 1002 notifies the user that the message is ready to be sent, Generates verbal output asking the user whether it should be sent. If the user indicates through the verbal or direct manipulation input that the user is not ready to send a message, the multimode virtual assistant 1002 may send the available options to the user, such as sending, canceling, reviewing, Generate voice output to inform. For example, the assistant 1002 may say, "Yes, I have not sent a message yet. To continue, you can send, cancel, review or change the message."

6A, in one embodiment, the multi-mode virtual assistant 1002 displays a message 770 and echoes the voice output by visually announcing to the user the options available for the text message 577 . In one embodiment, the text message 577 may be edited by the user by tapping within the field 773, together with the buttons 578 and 579 for transmitting or canceling the text message 577, Is displayed in the editable field 773 so as to indicate that it is possible to do so. In one embodiment, tapping within the editable field 773 invokes a virtual keyboard (similar to that depicted in Figure 3B) to allow direct editing.

The user can also interact with the assistant 1002 by providing voice input. Thus, in response to the voice message of the assistant 1002 providing options for interacting with the text message 577, the user can say "Change Message ". Assistant 1002 responds with a verbal message that recognizes the spoken text and prompts the user to say the corrected message. For example, the assistant 1002 can say "Yes, what do you want the message to say ?, " and then start listening to the user's response. FIG. 6B depicts an example of a screen 570 that may be viewed in association with such a voice prompt. Once again, the user's voice text is visually echoed (771) with the prompt 772 of the assistant 1002.

In one embodiment, once the user is prompted in this manner, the exact content of the user's subsequent speech input is interpreted as the content for the text message, bypassing the normal natural language interpretation of user commands. The user's voice input may be activated by a user pressing a button to indicate when a pause of sufficient length at the input is detected, or when detecting a specific word indicating that the input is complete, or to indicate that the user has finished speaking a text message It is presumed to have been completed when it detects that it has pressed or activated some other command. In one embodiment, the assistant 1002 then repeats the input text message again in the form of a voice, and may optionally echo the text message as shown in FIG. 6C. Assistant 1002 provides a voice prompt such as "Ready to Send a Message ?, " which can also be echoed (770) on the screen as shown in Fig. 6C. The user can then respond with "cancel", "transfer", "yes" or "no", any of which is correctly interpreted by the assistant 1002. Alternatively, the user may press a button 578 or 579 on the screen to trigger the desired action.

By providing a mechanism for modifying the text message 577 in this manner, the system of the present invention can, in one embodiment, be integrated with a hands-on approach so that the user can freely select the mode of interaction at each stage , Providing a suitable flow path for the hands-free context. Moreover, in one embodiment, the assistant 1002 adapts the natural language processing mechanism of the assistant to the specific steps in the overall flow, for example, as described above, in some situations, When prompted to speak a message, you can enter a mode to bypass the normal natural language interpretation of user commands.

Way

In one embodiment, the multimode virtual assistant 1002 detects the hands-free context and adapts one or more steps of the operation of the assistant to modify the user experience for the hands-free operation. As described above, the detection of the hands-free context can be applied in various ways to influence the operation of the multimode virtual assistant 1002. [ 7, a flow diagram depicting a method 10 of operation of a virtual assistant 1002 supporting dynamic detection of a hands-free context and adaptation to a hands-free context is shown, according to one embodiment have. The method 10 may be implemented in association with one or more embodiments of the multimode virtual assistant 1002. As depicted in FIG. 7, the hands-free context may be used in various processing steps in a multimode virtual assistant 1002, according to one embodiment.

In at least one embodiment, the method 10 may perform and / or implement various types of functions, operations, operations, and / or other features, e.g., one or more of the following Lt; RTI ID = 0.0 &gt;

Performs an interface control flow loop of the conversation interface between the user and the multimode virtual assistant 1002. At least one iteration of method (10) may serve as a round-trip in the conversation. The conversation interface is an interface through which the user and the assistant 1002 communicate by talking back and forth.

Provides execution control flow to the multimode virtual assistant 1002. That is, the procedure controls the collection of inputs, processing of inputs, generation of outputs, and presentation of output to the user.

Controls communications between the components of the multimode virtual assistant 1002. That is, where the output of one component is fed to another component, and where the full input from an action in the environment and environment can occur.

In at least some embodiments, portions of the method 10 may also be implemented in other devices and / or systems of a computer network.

According to certain embodiments, a plurality of instances or threads of the method 10 may be accessed through the use of one or more processors 63 and / or other combinations of hardware and / or hardware and software May be implemented and / or initiated simultaneously. In at least one embodiment, one or more of the methods 10 may be implemented in one or more clients 1304 (s), one or more servers 1340 (s), and / or combinations thereof .

For example, in at least some embodiments, various aspects, features, and / or functions of method 10 may be implemented in software components, network services, databases and / or the like, May be implemented, implemented, and / or initiated by a processor.

According to different embodiments, one or more different threads or instances of the method 10 may be used to trigger the initiation of at least one instance of the method 10 (e. G., The minimum threshold criteria) May be initiated in response to detection of one or more conditions or events that satisfy different types of criteria. Examples of various types of conditions or events that may trigger the initiation and / or implementation of one or more different threads or instances of a method may include, but are not limited to, one or more of the following:

User session with an instance of a multimode virtual assistant 1002, such as, but not limited to, one or more of the following:

a mobile device application that implements an embodiment of a starting mobile device application, e.g., a multimode virtual assistant 1002;

o an application that implements an embodiment of a starting computer application, e.g., a multimode virtual assistant 1002;

a dedicated button on the pressed mobile device, such as a "speech input button";

o a button on a peripheral device attached to a computer or mobile device, such as a headset, a telephone handset or any other device having a base station, a GPS navigation system, a consumer device, a remote control, or any other device having a button that can be associated with an invocation of an assistant;

a web session initiated from a web site that implements a multimode virtual assistant 1002 from a web browser;

o Interaction initiated from a web site that implements a multi-mode virtual assistant 1002, for example from within an existing web browser session, where a multimode virtual assistant 1002 service is requested;

o an email message sent to the forms server 1426 that mediates communication with the embodiment of the multimode virtual assistant 1002;

o A text message is sent to the form server 1426 that mediates communication with the embodiment of the multimode virtual assistant 1002;

o A telephone call is made to the form server 1434 that mediates communication with the embodiment of the multimode virtual assistant 1002;

o An event, such as an alert or notification, is sent to the application providing an embodiment of the multimode virtual assistant 1002.

When a device providing a multimode virtual assistant (1002) is turned on / turned on or started.

According to different embodiments, one or more different threads or instances of the method 10 may be initiated and / or implemented manually, automatically, statically, dynamically, concurrently, and / or in combinations thereof . Additionally, it should be appreciated that different instances and / or embodiments of the method 10 may be implemented in one or more different time intervals (e.g., during a particular time interval, at regular periodic intervals, at irregular periodic intervals, ).

In at least one embodiment, a given instance of the method 10 may be used to detect various different types of data and / or actions when performing certain tasks and / or operations, including detection of a hands-free context as described herein. Or other types of information. The data may also include any other type of input data / information and / or output data / information. For example, in at least one embodiment, at least one instance of the method 10 may access, process, and / or otherwise use information from one or more different types of sources, such as, for example, one or more databases . In at least one embodiment, at least a portion of the database information may be accessed through communication with one or more local and / or remote memory devices. Additionally, at least one instance of the method 10 may generate one or more different types of output data / information that may be stored, for example, in local memory and / or remote memory devices.

In at least one embodiment, the initial configuration of a given instance of the method 10 may be performed using one or more different types of initialization parameters. In at least one embodiment, at least a portion of the initialization parameters may be accessed through communication with one or more local and / or remote memory devices. In at least one embodiment, at least a portion of the initialization parameters provided to the instance of method 10 may correspond to / derived from / from the input data / information.

In the particular example of FIG. 7, a single user is assumed to access an instance of the multimode virtual assistant 1002 over the network from a client application with speech input capabilities. In one embodiment, the assistant 1002 is installed on a device 60 such as a mobile computing device, a personal digital assistant, a mobile phone, a smartphone, a laptop, a tablet computer, a consumer electronic device, a music player, The assistant 1002 is associated with a user interface that allows users to interact with the assistant 1002 through direct input and / or output of the graphical user interface as well as voice input and output (e.g., via a touch screen) do.

The device 60 has a current state 11 that can be analyzed 20 to detect whether the device is in a hands-free context. The hands-free context may be detected 20 based on state 11, using any applicable detection mechanism or combination of mechanisms, whether automatic or manual. Examples are described above.

When the hands-free context is detected (20), the information is provided in the name of the invention, which was filed on September 30, 2011, the entire disclosure of which is incorporated herein by reference, To the other context information 1000 that can be used to notify various processes to the assistant, as described in related US patent application Ser. No. 13 / 250,854, entitled " Use of context information to facilitate processing ".

A speech input is derived and interpreted (100). Induction may include providing prompts in any suitable mode. Thus, depending on whether a hands-free context is detected, in various embodiments, the assistant 1002 may provide one or more of several input modes. These may include, for example, the following:

Active typing-an interface for typing input that can trigger an input induction procedure;

An interface for speech input that can trigger an active speech input induction procedure.

An interface for selecting inputs from a menu that can trigger an active GUI-based input induction.

For example, if a hands-free context is detected, the speech input may be derived by a tone or other audible prompt, and the user's speech may be interpreted as text. However, those skilled in the art will recognize that other input modes may be provided.

The output of step 100 may be a set of candidate interpretations of the text of the input speech. Candidate interpretations of this set are processed (200) by a language interpreter 2770 (also referred to as a natural language processor or NLP), which analyzes the text input to determine a set of possible meanings Generate theoretical interpretations.

At step 300, these representations of the user's intent are conveyed to the dialog flow processor 2780, which implements an embodiment of the dialogue and flow analysis procedure to manipulate the user's intent as the task steps. Dialogue flow processor 2780 determines which interpretation of intent is most likely, maps these interpretations to the task model parameters and instances of domain models, and determines the next flow step in the task flow. If appropriate, one or more task flow step (s) adapted to the hands-free operation is selected (310). For example, as described above, the task flow step (s) for modifying a text message may differ when a hands-free context is detected.

In step 400, the identified flow step (s) is executed. In one embodiment, the invocation of the flow step (s) is performed by a service combination component 2782 that invokes a set of services instead of the user's request. In one embodiment, these services provide some data to a common result.

At step 500, a dialog response is generated. In one embodiment, the dialog response generator 500 is affected by the state of the hands-free context. Thus, when a hands-free context is detected, different and / or additional dialog units may be selected for presentation using the audio channel (510). For example, additional prompts such as "Are you ready to transmit? &Quot; may be verbally spoken and may not necessarily be displayed on the screen. In one embodiment, the detection of the hands-free context may affect the prompting for the additional input 520, for example to confirm the input.

At step 700, a multimode output (including, in one embodiment, verbal and visual content) is provided to the user and the user can then selectively respond visibly using the speech input.

If the user has finished (790) after seeing / viewing or listening to the response, the method ends. If the user does not finish, another iteration of the loop is initiated by returning to step 100.

Context information 1000, including the detected hands-free context, as described herein, can be used by various components of the system to affect the various stages of the method 10. For example, as depicted in FIG. 7, a context 1000 including a hands-free context may be used in steps 100, 200, 300, 310, 500, 510, and / However, those skilled in the art will appreciate that the use of context information 1000, including the hands-free context, is not limited to these specific steps, and that the system can also use context information in other parts without deviating from the essential characteristics of the present invention &Lt; / RTI &gt; Further description of the use of context 1000 in various steps of the operation of the assistant 1002 may be found in U.S. Patent Application Serial No. 10/1992, filed September 30, 2011, the entire disclosure of which is incorporated herein by reference, Related US Patent Application No. 13 / 250,854, entitled " Use of Context Information to Facilitate Processing of Instructions in a Virtual Assistant ", and related U.S. Patent Application, entitled "Context Voice Instructions, &quot; filed June 5, 12 / 479,477.

In addition, those skilled in the art will appreciate that different embodiments of method 10 may include additional features and / or operations in addition to those illustrated in the particular embodiment depicted in Figure 7 and / At least a portion of the operations and / or features of the method 10 as described above may be omitted.

The adaptation of steps 100, 200, 300, 310, 500, 510, and / or 520 for the hands-free context is described in more detail below.

Adaptation of input induction and interpretation (100) to a hands-free context

The derivation and interpretation (100) of the speech input may be adapted to the hands-free context, in any of several ways, individually or in any combination. As described above, in one embodiment, if a hands-free context is detected, the speech input may be derived by a tone and / or other audible prompt, and the user's speech is interpreted as text. In general, the multimode virtual assistant 1002 includes a number of possible mechanisms for audio input (e.g., Bluetooth-connected microphones or other attached peripherals) and a plurality of possible mechanisms for calling the assistant 1002 (E. G., By pressing a button on a peripheral device or using a motion gesture in close proximity to the device 60). Information about how the assistant 1002 is invoked and / or what mechanism is used for audio input can be used to indicate whether the hands-free context is active and can be used to modify the hands-free experience . More specifically, such information may be used to direct step 100 to use a particular audio path for input and output.

In addition, when the hands-free context is detected, the manner in which the audio input devices are used can be changed. For example, in the hands-on mode, the interface may require the user to press a button or make a physical gesture to cause the assistant 1002 to begin listening to the speech input. In contrast, in the hands-free mode, the interface may continue to prompt for input after all instances of output by the assistant 1002, or may allow continuous speech in both directions (assistant 1002) Allowing the user to interrupt the assistant 1002 while still speaking.

Adaptation of Natural Language Processing (200) to Hands-Free Context

The natural language processing (NLP) 200 can be adapted to the hands-free context, for example, by adding support for specific voice responses that are well-suited to hands-free operation in particular. Such a response may include, for example, "yes "," read message ", and "change it. In one embodiment, support for such responses may be provided in addition to support for voice commands available in a hands-on situation. Thus, for example, in one embodiment, the user would be able to operate the graphical user interface by saying commands appearing on the screen (e.g., when a button labeled "Transmit" appears on the screen, Support for understanding the word "transmission" and its semantic equivalents can be provided). In the hands-free context, additional commands may be recognized to account for the fact that the user may not be able to view the screen.

The detection of the hands-free context may also change the interpretation of the words by the assistant 1002. [ For example, in the hands-free context, the assistant 1002 sends the command "Keep quiet! And to turn off all audio output in order to recognize its semantic variants and in response to such comments. In the non-hands-free context, such an instruction may be ignored as not suitable.

The adaptation of the task flow 300 to the hands-free context

Step 300, which includes identifying task (s) associated with a user's intention, parameter (s) for task (s) and / or task flow steps 300 for execution, , Can be adapted to the hands-free context individually or in combination.

In one embodiment, one or more additional task flow step (s) adapted for hands-free operation is selected for operation (310). Examples include orally verifying and identifying content. In addition, in the hands-free context, the assistant 1002 can read lists of results that would otherwise be provided on the display screen. Oral commands may be provided to interact with the individual items in the list. For example, a number of incoming text messages may be provided to the user, and if a hands-free context is detected, the identified task flow steps may include separately reading each text message aloud and providing the user with a voice command And then pausing after each message.

In one embodiment, the task flows may be modified for a hands-free context. For example, a task flow for taking notes in a note application may typically involve prompting for the content and immediately adding the content to the note. Such an operation may be appropriate in a hands-on environment where content is immediately visible in a visual interface and immediately available for direct manipulation correction. However, when a hands-free context is detected, the task flow may be modified to, for example, review the content verbally and allow modification of the content before the content is added to the note. This allows the user to identify speech dictation errors before the speech dictation errors are stored in a permanent document.

In one embodiment, the hands-free context may also be used to limit tasks that are allowed at a given time. For example, a predetermined policy may be implemented to not allow videos to be played back when the user's device is in a hands-free context, or when in a particular hands-free context, such as driving a vehicle.

In one embodiment, the assistant 1002 may only make available the entire available domains of discourse and / or tasks applicable to the hands-free context. Examples include accessibility modes such as those designed for people with limited vision or limited use of hands. These accessibility modes include, for example, instructions implemented as hands-free alternatives for operating any GUI on a given application platform to recognize commands such as "press a button" or " . Other tasks that may only be applicable in hands-free modes include tasks related to the hands-free experience itself such as " using my car's Bluetooth kit "or" slowing down [text- .

Adaptation of Dialog Generation (500) for Hands-Free Context

In various embodiments, any of a number of techniques may be used to modify the dialog generation 500 to adapt to the hands-free context.

On the hands-on interface, the interpretation of the assistant's 1002 of the user's input may be echoed upon recording, but such feedback may not be visible to the user when in the hands-free context. Thus, in one embodiment, when a hands-free context is detected, the assistant 1002 uses a text-to-speech (TTS) technique to parse the user's input. Such term may be optional; For example, prior to sending a text message, the assistant 1002 may speak a text message so that the user can view the contents of the text message, even though the user can not see the display screen.

The determination of when to interpret a user's speech and which portions of speech to translate may be driven by task- and / or flow-specific dialogs. For example, in response to a user's voice command, such as "read my new message &quot;, in one embodiment, the assistant 1002 does not parse the command because the command is understood. ) Is clear from the response. However, in other situations, such as when the user's input is not recognized at step 100 or not at step 200, the assistant 1002 may use the voice input of the user to inform the user why the input is not understood Can try to paraphrase. For example, the assistant 1002 can say "I did not understand the reel my newt massage &quot; Please try again."

In one embodiment, the verbal paraphrase of the information may combine the dialogue templates with the personal data on the device. For example, when reading a text message, in one embodiment, the assistant 1002 reads "You have a new message from $ person, say $ message." We use a speech output template with variables of the form. The variables in the template can be replaced with user data and then switched to speech by a process running on the device 60. [ In one embodiment where the present invention is implemented in a client / server environment, such a technique may still allow personalization of the output, since personal data may remain on the device 60 and be filled upon receipt of an output template from the server Allowing users to protect their privacy.

In one embodiment, when a hands-free context is detected, different and / or additional dialog units, particularly adapted to the hands-free contexts, may be selected for presentation using the audio channel (510). The code or rules for determining which dialogue units to select may be sensitive to the details of the hands-free context. In this way, a generic dialog generation component can be adapted and extended to support a variety of hands-free variations without necessarily creating a separate user experience for the different hands-free situations.

In one embodiment, the same mechanism for generating text and GUI output units can annotate with tailored text for the audio (spoken word) output form. E.g:

In one embodiment, the dialog generation component can be adapted to the hands-free context by reading all of the recorded dialog responses of the dialog generation component using the TTS.

In one embodiment, the dialog generation component is adapted to the hands-free context by literally reading some of the recorded dialog response of the dialog generation component, via TTS, and using TTS variants for other dialog responses .

In one embodiment, such annotation supports a flexible alternate template mechanism for separating user data from dialog generation.

In one embodiment, the graphical user interface elements may annotate with text indicating how they should be verbally translated via the TTS.

In one embodiment, the TTS texts can be adjusted to use voice, speaking rate, pitch, pauses, and / or other parameters to convey what is otherwise conveyed in punctuation or visual rendering. have. For example, the voices used to repeat the user's words may be voices different from those used for other dialog units, or may use a different rhyme. As another example, the voice and / or rhyme may differ depending on whether the content or commands are spoken. As another example, pauses can be inserted between sections of text with different meanings, for ease of understanding. For example, when you parse a message and ask for confirmation, the pause can be inserted between the paraphrase of the content "Your message is read ..." and the prompt "Are you ready to transmit?".

In one embodiment, the non-hands-free contexts may be enhanced using similar mechanisms using TTS as described above for the hands-free contexts. For example, a dialog may generate verbal-only prompts in addition to recorded text and GUI elements. For example, in some situations, the assistant 1002 may orally say "send it?" To augment the on-screen display of the send button. In one embodiment, the TTS output used for both hands-free and non-hands-free contexts can be tailored for each case. For example, the assistant 1002 may use longer pauses when in a hands-free context.

In one embodiment, the detection of the hands-free context may also be used to determine when and whether to automatically prompt the user for a response. For example, when an interaction between an assistant 1002 and a user occurs substantially concurrently so that one party talks while the other party listens, the design selection is made by the assistant 1002 after the assistant 1002 has said, And whether or not the speech input should be automatically started to be heard. The details of the hands-free context can be used to implement various policies for this auto-start-listening attribute of the dialog. Examples include, without limitation, the following:

· Always automatic - Start - Listen;

· Auto-start-listen only when in hands-free context;

Automatic-start-listening only for certain task flow steps and dialog states;

Automatic-start-listening only for certain task flow steps and dialog states in a hands-free context.

In other embodiments, the detection of a hands-free context may also affect selection for other parameters of the dialogue, for example:

The length of lists of options to provide to the user;

Whether to read lists;

· Whether to ask questions with single or multiple value answers;

· Whether to prompt for data that can only be provided using the direct manipulation interface;

Thus, in various embodiments, the hands-free context is a system-side parameter that, once detected, can be used to adapt various processing stages of a complex system, such as a multimode virtual assistant 1002. The various methods described herein provide ways to adapt the general procedures of the assistant 1002 to the hands-free contexts to support a series of user experiences from the same basic system.

Various mechanisms for collecting, communicating, expressing, and accessing contexts are known in the art, entitled "Virtual Assistants, " filed on September 30, 2011, the entire disclosure of which is incorporated herein by reference. Use of context information to facilitate processing "in copending U. S. Patent Application Serial No. &lt; RTI ID = 0.0 &gt; 13 / 250,854. &Lt; / RTI &gt; Those skilled in the art will also recognize that such techniques are applicable to hands-free contexts.

Use cases

The following use cases are provided as examples of the operation of the assistant 1002 in the hands-free context. Those skilled in the art will recognize that the use cases are illustrative and are provided for illustrative purposes only.

Phone use cases

In one embodiment, when in the hands-free context, the assistant 1002 allows the user to call someone if the user can specify who to call without tapping or otherwise touching the device. Examples include calling a contact name, dialing a telephone number (numbers quoted by the user), and the like. Ambiguities can be solved by additional voice prompts. Examples are shown below.

Example 1: Calling an Unambiguous Contact

· Enter your voice: "Call Adam Smith"

· Assistant (1002) voice output: "I am calling Adam Smith's mobile."

· I get a phone call.

A similar interaction will occur for any of the following use cases:

· Call the contact by name ("Call Adam Smith")

· Call the contact by name, non-default phone number ("Call Adam Smith Mobile")

· Call the number ("800 555 1212")

· Call your contact by relationship nickname ("Call Mom")

· Call your contact by location nickname ("call home")

· Phone through FaceTime ("Face Time to Adam Smith")

· Call back from the context ("Call him again")

Example 2: Cancel a phone call

· Enter your voice: "Call Adam Smith"

· Assistant (1002) voice output: "I am calling Adam Smith's mobile."

· The user is notified that he is calling the wrong person; The user presses the Home button once to cancel the call, or says "Stop"

· The call is canceled.

Example 3: Call by ambiguous name

· Enter your voice: "Call Adam"

· Voice output of assistant (1002): "<Voice change> We found many matches for Adam"

Visual output of the assistant (1002): "<Voice change> I found many matches for Adam"

Voice output of the assistant 1002: Read names

Visual output of the assistant 1002:

Disambiguation Menu

Adam Chayer House

Adam Sandler House

Adam Smith Mobile

· Voice input by the user: "Adam Chayer"

· Voice output of assistant (1002): "I am calling Adam Chayer"

· I get a phone call.

Example 4: Calling a double ambiguous name

· Enter your voice: "Call Paul"

Voice output of assistant (1002): "< &lt; Voice change &gt; &gt;

Visual output of assistant (1002): "< &lt; Voice change &gt; &gt;

Voice output of the assistant 1002: Read names

Visual output of the assistant 1002:

Clear ambiguity menu

Paul Holden House

Paul Marcos iPhone

Paul Marcos Mobile

Paul Marcos House

Pablo Picasso House

· Voice input by the user: "Paul Marcos"

· Voice output of assistant (1002): "What is Paul Mark course number: iPhone, Mobile, or Home?"

Visual output of the assistant (1002): "What is the number for Paul Marcos?"

Visual output of the assistant 1002:

Clear ambiguity menu

Paul Marcos iPhone

Paul Marcos Mobile

Paul Marcos House

· User's voice input: "Mobile"

· Voice output of assistant (1002): "Paul Marcos <Pause> I am calling on mobile"

· I get a phone call.

Example 5: Calling a business by name without ambiguity

· Enter your voice: "Call Sonny Metreon in San Francisco"

· Voice output of assistant (1002): "AMC rou mat metron 16 found on the 4th street"

· Visual output of assistant (1002): "I found AMC Row Mettreon 16 on Fifth Avenue"

Visual output of the assistant 1002:

Local Search Snippet

AMC Low Mat Metron 16

San Francisco, 4th Avenue 101

(888) 262-4386

<Cancel button> <Call button>

Voice output of assistant (1002): "Would you like to make a call?"

· Enter your voice: "Call me"

· Voice output of assistant (1002): "Calling AMC Low Metron 16"

· I get a phone call.

Example 6: Calling a business by name, with multiple matches

· Voice input by the user: "Call Wolgreen"

· Voice output of assistant (1002): "I found several matches for <month change> <voice change>

Visual output of the assistant (1002): "I found several matches for the <green>

Visual output of the assistant 1002:

List of local navigation excerpts

Wall Green Store Los Gatos

Los Gatos N Santa Cruz Avenue 423

Wall Green

Campbell Campbell Avenue 1570, California

Wall Green

Cupertino Bowling Lord, California 1 2001

Wall Green

San Jose, California Saratoga Avenue 1334

Wall Green

San Jose Union Avenue 2680, CA

Voice output of Assistant (1002): "Can I read these to you?"

· User's voice input: "Yes"

· Voice of Assistant (1002): "Wolgreen Store Los Gatos at North Santa Cruz Avenue 423 in Los Gatos" Pause "2. Wolly Green <Pause> at Campbell Avenue 1570 in Campbell. "

· User's voice input: "What's in Cupertino"

· Audio output of assistant (1002): "I am calling Walton Green in Cupertino"

· I get a phone call.

Example 6: Making a scheduled call from your car

· The user starts driving home

· The assistant (1002) tones in the car reminder.

Visual output of the assistant 1002:

Reminder alarm

Call Adam Smith

<Close button> <Call button>

· User's voice input: "Read"

· Voice output of assistant (1002): "You have a reminder to call Adam Smith"

· Enter your voice: "Call him now"

· Voice output of Assistant (1002): "I am calling Adam Smith's mobile"

· I get a phone call.

Example 7: New voice  Playing mail

· Voice input by the user: "Play voice mail"

Voice output of Assistant (1002): "You have three new voicemail messages"

Assistant 1002 has prompts and plays voice mails one at a time.

SMS text messaging use cases

Example 8: Reading a single text message alert

Audio output of assistant 1002: to read alerts or alarms, or to generate sound indicative of an incoming text message

Visual output of the assistant 1002:

Text message

Mary Richard

Do you have time to have dinner tonight?

<Close button> <Reply button>

· User's voice input: "Read"

Voice output from Assistant (1002): "The message from Mary Richards is: <Voice change> Do you have time to have dinner tonight?"

Example 9: Replying to a text message (from Example 8)

· Voice input by the user: "Reply" <Pause> How about 8:00? "

· Voice of Assistant (1002): "I am ready to answer Mary Richard with the message"

Visual output of the assistant 1002:

To: Mary Richard

Of course, what about 8:00?

<Cancel button> <Transfer button>

· User's voice input: "Send"

· Audio output of the assistant (1002): "Transmitting": or emits an outgoing text message sound

Example 10: Sending a text message to a single recipient

· Enter your voice: "Tell Lisa I'll be 15 minutes late"

· Voice of Assistant (1002): Ready to send text to Lisa Gonzalez with the message "<Voice Change> I will be 15 minutes late"

Visual output of the assistant 1002:

Text message

To: Lisa Gonzalez

I'll be 15 minutes late.

<Cancel button> <Transfer button>

· User's voice input: "Send"

· Audio output of Assistant (1002): "Transmitting"; Or send text message sound

Example 11: Sending a text message to an ambiguous recipient

· Enter your voice: "Send the text to Adam with the message" Where is the meeting place? "

· Voice output of assistant (1002): "<Voice change> We found many matches for Adam"

Voice output of assistant 1002: Read names.

Visual output of the assistant 1002:

Clear ambiguity menu

Adam Chayer House

Adam Sandler House

Adam Smith Mobile

· Voice input by the user: "Adam Chayer"

Voice output of the assistant 1002: "Adam Chayer"

Visual output of the assistant 1002:

Text message

To: Adam Chayer

Where is the meeting place?

<Cancel button> <Transfer button>

· Voice output of assistant (1002): "We are ready to send text to Adam Chayer with the message" Where is the venue change? "

· User's voice input: "Send"

· Audio output of Assistant (1002): "Transmitting"; Or an outgoing text message sound

Example 12: Reading an SMS message from a message app

Visual output of the assistant 1002:

Text message

Betty Jo Vialkovski

You can call me Nancy.

<Close button> <Reply button>

· User's voice input: "Read"

· Voice output of assistant (1002): "Message from Betty Joe Vialkovsky <Voice change> You can call me Nancy"

Example 13: Returning from the context of a message app (from Example 12)

· Input voice of user: "Reply" <Pause> Hello Nancy "

· Audio output of Assistant (1002): "<Voice Change> Hello Nancy, I am ready to answer Betty Joe Vialkovsky"

Visual output of the assistant 1002:

To: Betty Joe Vialkowski

Hello Nancy

<Cancel button> <Transfer button>

· User's voice input: "Send"

· Audio output of Assistant (1002): "Transmitting"; Or an outgoing text message sound.

Example 14: Reading All Unread SMS Messages

Visual output of the assistant 1002:

Melanie Harbor (1)

Betty Joe Vialkowski (3)

Audrey Faber (1)

· Enter your voice: "Read my new messages"

· Voice of Assistant (1002): "You have four new messages from Melanie Harbor. How about dinner?" Pause "From Betty Joe Vialkovsky, you have two messages: Pause> 1. <Change Voice> My name is McGill. <Pause> 2. <Change Voice> I call myself Lil. "<Pause> 3. <Voice change> You call me Nancy I can "

Example 15: Returning the name from the context of a message app

· Enter your voice: "Tell Audrey I'll be 15 minutes late"

· Voice output of Assistant (1002): "I am ready to send text to Audrey Faber in the message" I am going to be 15 minutes late "

Visual output of the assistant 1002:

Text message

To: Audrey Faber

I'll be 15 minutes late.

<Cancel button> <Transfer button>

Voice output of assistant (1002): "Do you want to transfer?"

· User's voice input: "Yes"

· Assistant (1002) Audio output: "Transferring"; Or an outgoing text message sound

Example 16: Sending text messages to multiple recipients

In one embodiment, this task is determined to be out of the scope for the hands-free context. Therefore, the assistant 1002 returns to tapping for resolution of ambiguity

· Enter your voice: "Tell Lisa, Paul and Nancy that I'll be 15 minutes late for our important meeting"

· Voice output of assistant (1002): "Please choose from these people"

Visual output of the assistant 1002: GUI for selecting a plurality of recipients

The user selects the recipients via the GUI

· Voice output from Assistant (1002): "I am ready to send text to Lisa Gonzalez, Paul Marcos and Audrey Faber in a message that" I'll be 15 minutes late for our important meeting "

Visual output of the assistant 1002:

Text message

Recipients: Lisa Gonzalez, Paul Marcos, Audrey Faber

I'll be 15 minutes late for our important meeting.

<Cancel button> <Transfer button>

· User's voice input: "Send"

· Audio output of Assistant (1002): "Transmitting"; Or an outgoing text message sound

Alarms and Reminders  Use cases

Example 17: Single Reminders  Read alerts

Assistant (1002) gives a reminder tone

Visual output of the assistant 1002:

Reminders

Call Mary Richard

<Close button> <Call button>

· User's voice input: "Read"

Voice output of Assistant (1002): "Reminder for calling Mary Richard"

Example 18: Executing the reminder alarm (following example 17)

· Enter your voice: "Call her"

· Voice output of assistant (1002): "Mary Richard, I'm calling on mobile"

· I get a phone call.

Example 19: Creating a new reminder with an alert

· Enter your voice: "Remind me to call Bob Jones at 5 pm"

Voice of Assistant (1002): "Please check: <Voice Change> Reminders at 5pm to call Bob Jones"

Visual output of the assistant 1002:

Please check your reminder.

Call Bob Jones

Reminder Time: 5:00 PM today

<Cancel> <Save>

· Pause

· Voice output of assistant (1002): "Is that correct?"

· Enter your voice: "Yes, save it"

· Voice output of Assistant (1002): "Your reminder has been saved"

Example 20: Create a new reminder without any alarm PIs

· Enter your voice: "Remind me to buy bread" or "Remember to buy bread"

· Voice output of assistant (1002): "Please confirm: Remember to buy bread"

Visual output of the assistant 1002:

Please check your reminder.

Buy bread

<Cancel> <Save>

· Pause

· Voice output of assistant (1002): "Is that correct?"

· User's voice input: "Yes"

Voice output of Assistant (1002): "Added to your to-do list"

Calendar Use Cases

Example 21: Creating a simple appointment (given description and date / time)

· Voice of the user: "Schedule a meeting at 2 o'clock tomorrow to plan a company picnic"

· Voice output from Assistant (1002): "I'll put it in your calendar for 2 o'clock tomorrow"

Visual output of the assistant 1002:

Description: Plan a company picnic

When: 2 pm tomorrow

Example 22: Creating a simple appointment (no description given)

· Enter your voice: "I'll meet Carol at 2 o'clock tomorrow"

· Voice output from Assistant (1002): "I'll put it in your calendar for 2 o'clock tomorrow"

Visual output of the assistant 1002:

Description: Meet

When: 2 pm tomorrow

Example 23: Creating a simple appointment (no time given)

· Voice of the user: "Dental appointment next Thursday"

· Audio output of Assistant (1002): "What time is next Thursday?"

· Voice input by the user: "3 o'clock"

· Audio output from Assistant (1002): "I'll put it on your calendar for 3pm next Thursday"

Visual output of the assistant 1002:

Description: Dental appointments

When: Next Thursday - 3:00 pm

Example 24: Create a simple appointment (no given date or time)

· Voice input by the user: "Schedule a meeting to plan a company picnic"

Voice output of assistant (1002): "When do you want to meet?"

Visual output of the assistant 1002: Editable appointment type

· Voice input by the user: "Tuesday at 3 o'clock"

· Audio output from Assistant (1002): "I'll put it on your calendar for about 3 pm Tuesday"

Visual output of the assistant 1002:

Description: Meeting to plan company picnic

When: Tuesday - 3:00 pm

Those skilled in the art will recognize that the above examples merely illustrate the use of a hands-free context in certain situations. Additional uses include, for example, playback of media such as maps, music, and the like.

 The present invention has been described in detail with respect to possible embodiments. Those skilled in the art will appreciate that the present invention may be practiced in other embodiments. First, the naming of the components, capitalization of terms, attributes, data structures, or any other programming or architectural aspect is not mandatory or critical, and the mechanisms or features implementing the invention may be different names Formats, or protocols. The system may also be implemented as a software component, either as a whole, as a whole, or as a combination of hardware and software, as described. It should also be appreciated that a particular division of functionality between the various system components described herein is merely illustrative and that the functions performed by a single system component may instead be performed by multiple components, The functions performed by the components of the system 100 may instead be performed by a single component.

In various embodiments, the present invention may be implemented as a system or method for performing the techniques described above, either individually or in any combination. In another embodiment, the present invention provides a computer program product, comprising computer readable storage medium and computer program code encoded on the medium, for causing a processor in a computing device or other electronic device to perform the techniques described above, Product. &Lt; / RTI &gt;

Reference in the specification to "one embodiment" or " an embodiment "means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the invention. The appearances of the phrases ("in one embodiment") in various places in the specification are not necessarily all referring to the same embodiment.

Some of the above portions are provided with respect to algorithms and symbol representations of operations on data bits in a memory of a computing device. These algorithmic descriptions and representations are used by those skilled in the data processing arts to most effectively convey the essence of their work to others skilled in the art. An algorithm is considered herein to be a consistent sequence of steps (instructions) leading to a desired result and in general. These steps are those that require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals that can be stored, transferred, combined, compared, and otherwise manipulated. Sometimes it is convenient to refer to these signals as bits, values, elements, symbols, letters, terms, numbers, etc., mainly for reasons of common usage. Moreover, it is sometimes also convenient to mention certain arrangements of steps that require physical manipulations of physical quantities as modules or code devices, without loss of generality.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Quot; or " computing "or" displaying "or" determining ", etc. throughout the description, unless explicitly stated otherwise, as apparent from the following discussion Quot; are intended to encompass all types of computer systems and / or devices that operate on and modify data represented as physical (electronic) quantities within computer system memories or registers or other such information storage, transmission or display devices, or similar electronic computing modules and / And processes.

Certain aspects of the invention include process steps and instructions described herein in the form of algorithms. The process steps and instructions of the present invention may be embodied in software, firmware, and / or hardware and, when embodied in software, are downloaded to be on different platforms used by various operating systems, Lt; / RTI &gt;

The present invention also relates to an apparatus for performing operations in the present invention. Such a device may be specially configured for the required purposes, or it may comprise a general purpose computing device selectively activated or reconfigured by a computer program stored on the computing device. Such computer programs may include but are not limited to any type of disk, including floppy disks, optical disks, CD-ROMs, magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, Cards, ASICs, or any type of medium suitable for storing electronic instructions and coupled to a computer system bus, respectively. In addition, the computing devices referred to herein may include a single processor or architectures that utilize multiple processor designs for increased computing power.

The algorithms and displays provided herein are not inherently related to any particular computing device, virtualization system, or other device. Various general purpose systems may also be used with the programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the required method steps. The architecture required for a variety of these systems will be apparent from the description provided herein. In addition, the present invention is not described with respect to any particular programming language. It will be appreciated that various programming languages may be used to implement the teachings of the invention as described herein and that any reference to particular languages is provided for the implementation of the invention and for the initiation of the best mode .

Thus, in various embodiments, the invention may be implemented as software, hardware, and / or other components for controlling a computer system, a computing device, or other electronic device, or any combination of these or a plurality thereof . Such an electronic device may be, for example, a processor, an input device (e.g., a keyboard, a mouse, a touchpad, a trackpad, a joystick, a trackball, a microphone, and / A storage device (e.g., a magnetic storage device, an optical storage device, and / or the like), and / or a storage device Connectivity can be included. Such an electronic device may not be portable or portable. Examples of electronic devices that can be used to implement the invention include mobile phones, personal digital assistants, smart phones, kiosks, desktop computers, laptop computers, tablet computers, consumer electronic devices, consumer entertainment devices; Music player; camera; television; Set top box; An electronic game unit; And the like. An electronic device for implementing the invention may be any operating system, such as iOS or MacOS, available from Apple Inc. of Cupertino, California, USA, or any other operating system adapted for use on a device The system can be used.

While the present invention has been described with respect to a limited number of embodiments, those skilled in the art having the benefit of the foregoing description will appreciate that other embodiments may be devised which do not depart from the scope of the invention as described herein. In addition, it should be noted that the language used herein is primarily selected for readability and illustrative purposes, and may not be selected to describe or limit the gist of the invention in detail. Accordingly, the disclosure of the present invention is intended to be illustrative rather than limiting the scope of the invention as set forth in the claims.

Claims (67)

A computer-implemented method for interpreting user input to perform a task on a computing device having at least one processor,
Detecting, at the processor, whether a hands-free context is active;
At the output device, prompting the user for input;
At the input device, receiving user input;
In the processor, interpreting the received user input to derive a representation of the user's intention;
Identifying at least one task and at least one parameter for the task based at least in part on the derived representation of the user intention;
In the processor, executing the at least one task using the at least one parameter to derive a result;
Generating, in the processor, a dialog response based on the derived result; And
In the output device, outputting the generated dialog response
/ RTI &gt;
In response to detecting that the device is in a hands-free context, prompting the user for the input, receiving the user input, interpreting the received user input, Wherein at least one of the steps of: identifying at least one parameter for the task, and generating the dialog response is performed in a manner that complies with the constraints associated with the hands-free context. . 2. The method of claim 1, wherein at least two interaction modes are available for user interaction with the computing device,
In response to detecting that the device is in a hands-free context, prompting the user for the input, receiving the user input, interpreting the received user input, Wherein at least one of the steps of identifying at least one parameter for the task and generating the dialog response is performed using a first interaction mode adapted for a hands-
In response to detecting that the device is not in a hands-free context, prompting the user for the input, receiving the user input, interpreting the received user input, And identifying at least one parameter for the task, and wherein generating at least one of the steps of generating the dialog response is performed using a second interaction mode that is not adapted to the hands-free operation. A computer implemented method for. 3. The method of claim 1 or 2, wherein detecting whether the hands-free context is active comprises:
A condition that indicates whether the computing device is located in a geographic location;
A condition indicating whether the current time is within a time of day;
A condition that indicates whether the computing device is connected to another device;
A condition that indicates whether the computing device is moving beyond a critical speed;
A condition that indicates whether the computing device is near a user; And
A condition indicating the presence of a specific word in the received user input
And detecting a condition selected from the group consisting of: &lt; RTI ID = 0.0 &gt; a &lt; / RTI &gt; 3. The method of claim 1 or claim 2, wherein prompting the user for the input comprises:
In response to detecting that the device is not in a hands-free context, prompting the user through a first output mode that is not adapted to the hands-free context; And
In response to detecting that the device is in a hands-free context, prompting the user through a second output mode adapted to the hands-free context
Wherein the computer-readable medium comprises computer-executable instructions for: 5. The method of claim 4,
Wherein prompting the user through the first output mode comprises prompting the user through a visual output mode,
Wherein prompting the user through the second output mode comprises prompting the user through an audible output mode. 6. The method of claim 5,
Wherein prompting the user through the visual output mode comprises displaying a prompt on a display screen,
Wherein prompting the user through the audible output mode comprises outputting a spoken prompt. &Lt; Desc / Clms Page number 21 &gt; 3. The method of claim 1 or 2, wherein interpreting the received user input comprises: in response to detecting that the device is in a hands-free context, using the vocabulary associated with the hands- And interpreting the user input. The method of claim 1 or 2, wherein identifying at least one task and at least one parameter for the task comprises: in response to detecting that the device is in a hands-free context, And performing at least one task flow identification step. delete delete delete 3. The method of claim 1 or claim 2, wherein generating the dialog response comprises generating a dialog response in a speech-based output mode in response to detecting that the device is in a hands- A computer implemented method for interpreting an input. delete delete delete 2. The method of claim 1, wherein detecting whether the hands-free context is active comprises:
Receiving a user input specifying a hands-free context;
Receiving data indicative of environmental conditions associated with the hands-free context from at least one sensor;
Detecting a connection of a peripheral device associated with the hands-free context;
Detecting a connection release of a peripheral device not associated with the hands-free context;
Detecting communication with an onboard system of the vehicle;
Detecting a current position; And
The step of detecting the current speed
And at least one step selected from the group consisting of: &lt; RTI ID = 0.0 &gt; a &lt; / RTI &gt; delete delete delete A computer program product for storing a computer program for interpreting user input to perform a task on a computing device having at least one processor,
The computer program comprising:
At least one processor,
Detecting whether the hands-free context is active,
Causing the output device to prompt the user for input,
Receiving user input via an input device,
Interpreting the received user input to derive a representation of the user's intent;
Identifying at least one task and at least one parameter for the task based at least in part on an expression of the derived user intention,
Executing the at least one task using the at least one parameter to derive a result,
Generating a dialog response based on the derived result, and
Causing the output device to output the generated dialog response
, &Lt; / RTI &gt;
Responsive to detecting that the device is in a hands-free context, the computer program further comprises at least one processor prompting the user for the input, receiving the user input, interpreting the received user input Identifying at least one task and at least one parameter for the task, and generating the dialog response in a manner that complies with the constraints associated with the hands-free context The computer program product comprising: a computer readable medium; 21. The method of claim 20, wherein at least two interaction modes are available for user interaction with the computing device,
Responsive to detecting that the device is in a hands-free context, the computer program further comprises at least one processor prompting the user for the input, receiving the user input, interpreting the received user input Identifying at least one task and at least one parameter for the task, and generating the dialog response using a first interaction mode adapted for a hands-free operation Lt; / RTI &gt;
In response to detecting that the device is not in a hands-free context, the computer program causes the at least one processor to prompt the user for the input, receive the user input, Identifying at least one task and at least one parameter for the task, and generating the dialog response using a second interaction mode that is not adapted to the hands-free operation The computer program product comprising: a computer readable medium; 22. The computer program according to claim 20 or 21, wherein the at least one processor is configured to detect whether a hands-free context is active,
At least one processor,
A condition that indicates whether the computing device is located in a geographic location;
A condition indicating whether the current time is within a day's time;
A condition that indicates whether the computing device is connected to another device;
A condition that indicates whether the computing device is moving beyond a critical speed;
A condition that indicates whether the computing device is near a user; And
A condition indicating the presence of a specific word in the received user input
And a computer program product configured to detect a condition selected from the group consisting of: &lt; RTI ID = 0.0 &gt; a &lt; / RTI &gt; 22. The computer program according to claim 20 or 21, wherein the output device is configured to prompt the user for input,
A computer program configured to cause the output device to prompt the user through a first output mode that is not adapted to the hands-free context, in response to detecting that the device is not in a hands-free context; And
Wherein the output device is configured to prompt the user via a second output mode adapted to the hands-free context in response to detecting that the device is in a hands-free context.
Readable medium. 22. The computer program according to claim 20 or 21, wherein the at least one processor is configured to cause the received user input to be interpreted by the at least one processor in response to detecting that the device is in a hands- And a computer program configured to interpret the received user input using a vocabulary associated with a hands-free operation.  22. The computer program according to claim 20 or 21, wherein the at least one processor is configured to identify at least one task and at least one parameter for the task, wherein the at least one processor comprises: Readable medium having computer-executable instructions configured to perform at least one task flow identification step associated with a hands-free operation, in response to detecting a presence in a context. 22. The computer program product of claim 20 or claim 21, wherein the at least one processor is configured to cause a dialog response to be generated, wherein the at least one processor is configured to: A computer program configured to generate a dialog response in an output mode. 21. The computer program product of claim 20, wherein the at least one processor is configured to detect whether a hands-free context is active,
At least one processor,
Receiving user input specifying a hands-free context;
Receiving data indicative of environmental conditions associated with the hands-free context from at least one sensor;
Detecting a connection of a peripheral device associated with the hands-free context;
Detecting disconnection of peripheral devices not associated with the hands-free context;
Detecting communication with the onboard system of the vehicle;
Detecting the current position; And
Detecting the current speed
&Lt; / RTI &gt; wherein the computer program is adapted to perform at least one selected from the group consisting of: &lt; RTI ID = 0.0 &gt; 28. The method according to any one of claims 20, 21 and 27,
The computer program configured to cause an output device to prompt the user includes a computer program configured to cause the output device to prompt the user via a dialog interface,
The computer program configured to cause at least one processor to receive user input, the computer program comprising a computer program configured to cause at least one processor to receive user input via the interactive interface. delete delete A system for interpreting user input to perform a task on a computing device,
An output device configured to prompt the user for input;
An input device configured to receive user input;
Determining whether a hands-free context is active, communicatively coupled to the output device and to the input device, interpreting the received user input to derive a representation of the user's intent, Identifying at least one task and at least one parameter for the task based at least in part on an expression of an intention, executing the at least one task using the at least one parameter to derive a result And at least one processor configured to perform a step of generating a dialog response based on the derived result
/ RTI &gt;
Wherein the output device is further configured to output the generated dialog response,
In response to detecting that the device is in a hands-free context, prompting the user for the input, receiving the user input, interpreting the received user input, Wherein at least one of identifying at least one parameter for the task and generating the dialog response is performed in a manner that complies with the constraints associated with the hands-free context. 32. The method of claim 31, wherein at least two interaction modes are available for user interaction,
In response to detecting that the device is in a hands-free context, prompting the user for the input, receiving the user input, interpreting the received user input, Wherein at least one of identifying at least one parameter for the task and generating the dialog response is performed using a first interaction mode adapted for a hands-
In response to detecting that the device is not in a hands-free context, prompting the user for the input, receiving the user input, interpreting the received user input, And interpreting user input, wherein at least one of identifying at least one parameter for the task and generating the dialog response is performed using a second interaction mode that is not adapted to a hands-free operation For the system. 33. The system of claim 31 or 32, wherein the at least one processor comprises:
A condition that indicates whether the computing device is located in a geographic location;
A condition indicating whether the current time is within a day's time;
A condition that indicates whether the computing device is connected to another device;
A condition that indicates whether the computing device is moving beyond a critical speed;
A condition that indicates whether the computing device is near a user; And
A condition indicating the presence of a specific word in the received user input
To determine whether a hands-free context is active by detecting a condition selected from the group comprising: &lt; RTI ID = 0.0 &gt; a &lt; / RTI &gt; 33. The apparatus as claimed in claim 31 or 32,
In response to detecting that the device is not in a hands-free context, by prompting the user through a first output mode that is not adapted to the hands-free context, and
In response to detecting that the device is in a hands-free context, by prompting the user through a second output mode adapted to the hands-free context,
A system for interpreting user input, the system configured to prompt a user for input. 35. The method of claim 34,
Wherein the first output mode includes a visual output mode,
Wherein the second output mode comprises an audible output mode,
Wherein the output device comprises:
A display screen configured to prompt the user via the visual output mode, and
A speaker configured to prompt the user through the audible output mode;
The system comprising: means for receiving a user input; 36. The method of claim 35,
Wherein the display screen is configured to display a visual prompt,
Wherein the speaker is configured to output a voice prompt. 33. The method of claim 31 or 32, wherein in response to detecting that the device is in a hands-free context, the at least one processor is configured to interpret the received user input using a vocabulary associated with a hands- , A system for interpreting user input. 33. The method of claim 31 or 32, wherein in response to detecting that the device is in a hands-free context, the at least one processor performs at least one task flow identification step associated with a hands- And to identify at least one parameter for the task. 33. The method of claim 31 or 32, wherein in response to detecting that the device is in a hands-free context, the at least one processor is configured to generate the dialog response in a speech- For the system. 33. The system of claim 31 or 32, wherein the at least one processor comprises:
User input specifying a hands-free context;
Data from at least one sensor indicative of an environmental condition associated with the hands-free context;
Connection of peripheral devices associated with the hands-free context;
Disconnection of peripheral devices not associated with the hands-free context;
Communication with the vehicle's onboard system;
Current location; And
Current speed
Free context is active based on at least one selected from the group consisting of: &lt; RTI ID = 0.0 &gt; a &lt; / RTI &gt;
delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images